Eh, I get it. Really. I totally understand your perspective. if it makes you feel better, I would sign a petition to re-create the scene from Game of Thrones just for this rule...
...not because I feel that strongly, but just because it would be really funny.
However, let me provide you my perspective as to why I am indifferent:
1. In practice it doesn't matter. People get it. They understand the rule.
2. If they are going to have one "throwback" rule, this might as well be it. It's pretty inconsequential.
3. It would be much more difficult to express the rule correctly using alternate methods. Which is weird, but think about it- if you do it just using the proficiency system and the "normal" rules, it gets weird. If you do it in a positive manner (by expressing that, for example, Druids can wear hide and not stating the other medium armors) then that doesn't answer questions about multiclassing or alternate materials. If you attach a penalty (Druids are 'non-proficient' with metal armor) you'd have the same arguments that you currently have- "It doesn't make sense. How can you be proficient in Dragon Scale, but not proficient in Metal Scale." Etc. While you find the rule inelegant, and others argue about player agency, it's a quick way of expressing the concept that most people understand.
...but again, IMO the reason it's a poor rule isn't really about the rule
qua rule, it's because it reflects a design choice that no longer exists that much in 5e.