Funny, I never thought much about this until my son's integrated preschool sent home an article in their monthly bulletin that was all about talking with others when you have a SN child. The first thing it said was that there was no reason to tell someone unless they had a NEED to know... and suddenly, a lot of my problems were solved. It further went on to say that diagnosis was not something anyone needed to know and that when talking about your child, you should limit it to the particular deficit that you or they are currently dealing with when it comes to your child.
I've found that since I followed these rules, conversations are MUCH easier. People want to dig for dirt and diagnoses for countless reasons and I find myself toying with them--withholding that info and looking at them like they're crazy, saying "What? He just has a little confusion about X..."
The term "special needs" doesn't bother me in any way, shape or form. What bothers me is when I point out to a parent of a gifted child that their child's needs are addressed by the special education dept., too (in fact, SpEd teachers can get a certification in gifted teaching). Special Ed/Special Needs seems to carry a connotation of disability vs. an accommodation that needs to be made (which is the education law's view of it).
If not "special needs", I'm not sure what they'd call it. I'd rather that than "learning disabled". He's not disabled. He needs accommodations to function in a classroom that requires everyone to be the same.