Honestly, I've never understood the disdain for the hipster that most people tend to evince. Looking at how hipsters define themselves, it's a group of people with an interest in fashion, vintage items, and independent culture (art/music/film). That sounds like a pretty cool group!
I think the problem comes in with the attitude that you suggest in your first paragraph: smugness and/or elitism. It seems to me that when people use "hipster" as an insult, they are referring to that attitude of looking down your nose at anything mainstream, regardless of the inherent value in that item. Or they are suggesting that you're willing to pay crazy prices for mundane items. If you walk around with a smile on your face and genuine interest in people around you, I don't think there is a chance that you will ever be mistaken for a "hipster", even if you share a lot of the same fashion sense.