I attend one convention a year. That convention is CONvergence. It is not a gaming convention, but there is gaming that is done there, among many other things. I have found that I do not like to play RPGs at cons, but I am happy to play boardgames or card games.
This con has the advantage of being local, and being respectable in size without being too huge. This gives it many bonus points when it comes to selecting a con to attend.
I am leary of going to Gencon, or Origins, or other large conventions for a couple of reasons. First, I am a bit claustophobic. Tight crowds with no easy exit would easily set me off. It's no fun when the 40-year-old, 6'2', 230 lb guy freaks out and has a panic attack in the middle of a crowd. Someone is going to get hurt.
Secondly, I have some health issues that deal with my lungs. This is made worse by that nasty thing we all know as "gamer funk." I've been to some pretty big conventions in the middle of summer with people wearing upholstery fabric and/or latex. Those conventions are a walk in the flower garden compared to gaming cons.
Oh, and it's not just the large gamers who have this problem. Large, small, old, young, the "rank and vile" come in all shapes and sizes. I wonder if it has to do with obsessive compulsive behavior? I know that if I went to Gencon or Origins I would have to be blunt and hand out hygiene packets that include a map to a shower. I like the game company that was putting a d20 in a bar of soap. Now if I could get my hands on some cool limited edition figure to put in them I could sell a ton of soap!
So, if we disregard the expense and travel time, there are still things that would cause me to hesitate before going to gaming conventions.