I don't think it's your fault at all or that you're obsessed...the problem is in how our society works. Parents (generally) will talk about nothing but their kids. It's natural for people to want to be with others who share their interests, if a CF hangs out with a parent who talks about nothing except the latest wall art from junior, the CF will feel alienated and distanced.
I think we also long to have others with similar views because it reaffirms the idea that we're "Normal" (I don't believe there is such a thing as normal--except on a dryer!) But, we still want to be part of the "group" and feel that to be a part we have to behave similarly.
I think we all wish we had more CF friends, but considering how baby-crazy our culture is, it just isn't realistic. I think we all eventually find ways to adapt. I've made friends with mostly single men, who don't care at all that I don't want kids because they don't really want them (at least right now) either. I know, though, that this might, and probably will, change if/when they get married.
I guess I didn't give you much advice...but, just keep in mind you have a team of supporters on this forum!