I have an old school D&D t-shirt that one of my players gave me as a gift - and if it is clean (which it usually is, as I very rarely wear it) I will sometimes wear that, as I want him to know I appreciate the thought.
Since my game is on Tuesday nights, after work, I wear whatever I wore to work that day, which is usually blue jeans and a t-shirt (my work has a pretty lax dress code). If I'm formal, it might be khakis and a polo with our company logo on it.
I wear whatever I happen to put on that day. Often jeans and a t-shirt. Or sweatshirt in the winter. Sometimes I'll wear a gaming-related t-shirt, sometimes not. It depends on what I'm doing that day.
Jeans, t-shirt, and some sort of button-down overshirt which occasionally is a dress shirt, occasionally flannel, but sometimes just casual or one with designs.