Short answer: yes.
Most of my players work with me on the same shift (weekend evenings), and although I don't mind telling people I play games, go to gaming conventions, play in life-sized Mechwarrior cockpits to blow up my friends, and like to paint minis, what I DO mind is when my players want to have a discussion about our game at work, on work time. [BTW, where I work, it is all about production numbers, if you stop to talk, it means you aren't doing work. I realize many jobs are more flexible than that, but I am competing directly with a technician in Malaysia who can do the same job for half the cost.]
I don't like that kind of distraction at work, and I don't want my boss to get the impression I would rather be gaming than at work. I often compare it to the people who always talk about football, or other sports, but even so, I feel that talk about gaming in general is seen as juvenile for grown-ups, and for that reason, my boss and others may look down on either gaming or myself as a result. During breaks, however, I'm happy to talk about gaming with my players whenever they bring it up. I've actually gathered a lot of first-time players from work.