Why do people hate gnomes so much?

Fairy tale gnomes, garden gnomes, or travel commercial gnomes? In Harry Potter they had to be cleaned out of the garden like weeds (only in the book; that detail didn't make it into the movie).

Garden gnomes. My husband is always teasing me about my garden gnome. I love them, but others I've talked to think they're evil. Seriously.