Here in NYC people tend to mistake you for a lot of stuff. Believe me, I have been mistaken for Hispanic and Middle Eastern quite a lot. I have heard people make curry jokes right in front of my face, a person who knows me points to me when they are making those jokes and then they realize I am Indian. In NYC you can't look at a person and just say who they are, I can't even go up to a White guy and start speaking in English without knowing him because he could be a White Latino for all I know. What stores have you exactly been going to and what Indians have you been associating with? If it is lower class then heck, I have seen White women walk on the other side of the street whenever they see a Black guy walking in their direction.
EiCib is right. There's a stereotype that Indians smell bad. Maybe it's unfair, maybe it's not. Maybe it's based on your cuisine being gassy, or your skin color, or your people having a tendency of being seen as dirty, or a combination of all of them... but "Indians smell like shit" is a very common perception, true or false. When the subject of an Indian person comes up, and there's a long enough discussion, something to do with "curry farts" is likely to be interjected into the conversation.