I think any occasion for which you have some measure of respect (Wedding, awards ceremony, parent's birthday, anniversary etc.) means that a change in your normal, everyday dressing patterns is needed.
As for the "doesn't affect the business" argument, If one person thinks you look like a lazy clown because you didn't bother to tidy yourself up for a special event, that's one person too many (and hiding behind "I won't be judged by my appearance"--while certainly a good example of High-School level iconoclasm--is in this day and age, in this culture, incorrect. You are judged by your apperance.)
I'm not going to wear a tux, but come on! Baggy shorts, sandals w/socks and a crappy t-shirt is just a bad Idea in general. It (along with ponytail/goatee/trenchcoat/anime shirt in colder weather) is what people are talking about when they are talking about "Awful Gamer Stereotype".
You know the stereotype: the one everyone decries as having no basis in fact? That one.
Anyway. It's a free country, dress as you will.