I explicitly assume characters are wearing light gloves unless they say otherwise.
There is nothing wrong with that. If you were up front from the start that I was always wearing gloves unless I stated otherwise, then it would be my responsibility to remember that I am gloved and inform you if I wanted to act without gloves, much as in my game it would be the responsibility of a player wearing heavy armor to tell me that he wanted to remove his gauntlets to help him better search through some pockets in a backpack or thread a sewing needle or do any other task that might benefit from less bulk, more sensitive touch, and improved dexterity.
Arguing over whether the gloves are realistic misses the point. The point is the GM already gave a ruling that the gloves weren't on and based on all the information available to that GM that's an entirely plausible ruling.
You saying, "I just assume the player has gloves on" doesn't solve the actual problem here.
The problem is that in any actually complex gameplay there will be situations where gloves are helpful and gloves are not, which is why as you say in reality players would be taking their gloves on and off through the day.
Given that you always assume gloves, what happens when something bad happens because the player is wearing gloves and then the player retroactively, after discovering the threat or consequences in this situation was the result of insensitive hands and not contact poison to bare skin, demands that you retcon the situation and replay it as if he was not wearing gloves?
And does it change your answer if you know, as the player now doesn't, that he has contact poison on the gloves that he was protected from?
You see at some point accepting the gloves are on or off on the basis of circumstances becomes a really nasty form of railroading where you are deciding to play the character for them. And at some point all the questions about "Are you sure you would do that?" or "Do you have gloves on?" becomes metagame directing, which is one of the most insidious and corrosive sorts of railroading a GM can engage in.
At some point you have to not only allow the player to play his character, but also learn how to do so.