Christo has partially hit it on the head with the comment about "posers".
Some of it is a hatred for trend hopping sheep.
Some of it probably carries over from many folks who have had run in's with psychotic cycle couriers who think that they own the road, the footpath and everywhere else and that the entire planet owes them a right of way.
The combination of reckless cyclists and talentless poser cyclists with a bit of "harder to control than the average bike for an underskilled poser" is a dangerous combination.
Some folks hate emo's, but emos aren't thinking they own all the paved surfaces while simultaneously putting your life in jeapordy with a bike they can't control properly.
Otherwise, fixies are cool. I have a friend who is a track cyclist. She has an Anthem now too, and we're gradually teaching her to change gears, use the brakes, coast/freewheel and even to turn right...
Like any group of cyclists, it's usually a minority that make it hard for the rest and result in a stereotype. (Not all singlespeeders are beer swilling, bearded, fat and tattoed for instance)
People say "fixie rider" and the instant mental image that pops into the head is a cloth capped cycle courier riding like a maniac, or a trendoid cafe wanna-be. Obviously not all fixie riders are like that.