IMO it's just a question of game design, simulationist vs gameist- I think I straddle the line, trying to balance what the system accounts for vs. verisimilitude and the world's "rules."
The best way I can rationalize such cognitive dissonance is to come at it from the angle of: your character sheet and the game system itself are the methods through which you interact with the world- they don't guide the mechanisms of the world itself; then it becomes easier to justify such limitations, while maintaining verisimilitude. It also helps answer questions like "why do NPCs have things that the characters can't have?"
After all, you can (I do) run different systems in the same setting- so the "these rules are how your character/this particular campaign is being guided" thing makes sense.
Well afaik you can't learn skills in downtime, that's another feature restricted by class/xp; I have to reason that armor and skill proficiencies are judged to be more valuable from a game design standpoint by the designers than the rest; a wizard with a sword still only gets one attack with it, whereas a wizard with plate armor now has ratcheted up their AC drastically which will come into play frequently. I've had the "artificer1/wizard## with 10 strength riding around with plate armor" before and I wouldn't like to encourage that further
That's the best reason I've seen
Learning a weapon (or armor if you allow it) takes months, spellcrafting takes weeks.
Although learning a weapon (or armor) is
much cheaper