The whole caster/martial discussion also ignores 1 VERY important thing. The cost to a caster to gain new options is trivial. For a fighter to gain a bonus feat/ASI, the only way that fighter can do it is gain levels. There is nothing else the fighter can do to gain that new option. Even magic items are dependent upon the DM making them available. And, generally, magic items for sale are a major expense. That 20th level fighter most likely did not buy +3 armor and a +3 shield. Not that that's unreasonable for that 20th level fighter to have, but, it's certainly not guaranteed.
OTOH, the only thing a wizard has to do to gain new options is find/buy spells. And those are cheap. And at every spell level, there are 30 (ish) options to choose from. Never minding that the wizard is getting 2 of them every level. A single enemy wizard could easily net a dozen new options for the wizard.
Imagine if killing a troll netted a fighter a dozen new options. Laughable right? But, an enemy wizard? You score that spellbook, which every enemy wizard has, and poof, you've got a dozen or more new options for your character. Forever.
The idea that we cannot give non-casters new options seems a bit strange since we have zero problems granting casters dozens of new options every two levels for the non-book casters, and potentially dozens of options simply for killing the right monster.