I think we have a natural fear of the unknown.
Like fear of the dark, it really is a fear of what we can't see in the dark more than the darkness itself. More what is hidden and we can't see that scares us. I think this is the natural tendency with all things we do not know and have totally no control over. I currently live in a predominantly French area and I get about a fourth of the conversations that go on around me. I don't find it bothersome until I go to an event where I am supposed to be included and can not join in the conversation. But then I think, well shame on me for not just learning the language seeings they can speak at least one of mine.
After several years of living in that type of environment I know what it is like to be on both sides of the fence because when I speak Spanish they don't get me either but the funny thing is, the French folks around my area are so open and willing to translate for you or speak your language if you were supposed to be included that you A) feel worse cause you become a burden to the topic or B) really know when you weren't invited to the convo. It's also really interesting to speak a language everyone is so into... instant entertainment, let's get Becky to talk Spanish. hehehe