I understand your reasoning but I don't completely agree with this. Heavy armor does limit more than just things requiring manual dexterity. For one, Heavy armor is "Heavy". A Wizard wearing heavy armor (whether proficient or not) isn't going to suddenly put it on just before or just as combat begins. He'll have to wear it all of the time. If he's not used to wearing armor (i.e.: not proficient), he is going to be in a constant state of fatigue. Not only would it be hard to perform "gestures", but even breathing becomes difficult after a time.
Even the best made, fitted, master work armor, weighs in at about 45 to 50 pounds. By comparison, a modern Interceptor Kevlar Vest weighs in at about 17 lbs., and brother, let me tell you - after a day (or more) of wearing it, your neck and shoulders will be sore and even breathing can start to be a chore. Now granted, I was in aircraft maintenance in the Air Force, so other than exercises, or when outside at deployed locations, I really didn't have to wear it that often, so I could be considered non-proficient. Someone in a combat position would be used to it (proficient). Without being used to wearing it, it would make a big difference in everything you do, not just hand gestures.