I am so sorry you are going through this. I am white, and my daughter is African American, and I can see a variation of this scenario in the near future. She is only 15 months now, so it's not yet such an issue. We live in an area that is very welcoming to all sorts of families, so I don't worry so much about the adults influencing the kids, but more the kids feeling more comfortable with others who look like them. Already, other African American kids are much more likely to come running up to her and tell me how cute she is. White kids are polite, but not as ebullient.
So I guess my advice would be, you can't change people you barely know, so reach out to everyone who is positive. If you are bold, you could organize a block party and let people get to know you (but be mentally prepared for possible disappointment). Does your husband work out of the home? Does he have any Indian co-workers? Maybe he can find some more diverse social avenues for your family to head down.
Is it really impossible to move? I think that is what I would do if things didn't get better, if at all possible.