I totally sympathise. Since getting more serious about style, I actually look forward to A/W, sick as that sounds. I find when it's colder, I can layer, and when I can layer, my outfits are so much more interesting.
Layering in the summer is a nightmare, as I get really hot very easily. Got a jacket on today, but have spent most of the day with it on the back of the chair.
I think in summer, the key is to nail the fit. That helps set you apart. Work is the hardest for me, as I can't really embrace those other touches which I would like to, e.g. going sockless.