Well, if 1e rule you posted said
"The character may mix functions freely and still gain experience, although restrictions regarding armor, shield, and/or weapon apply with regard to operations particular to one or both classes." (Emhpasis added.)
Which would mean that if a wizard can't cast spells in armor, then he would be no more abile to with a level in fighter.
Anyhow, there are mechanics to address your arguments.
1. Why does brestplate have ASF while it leaves my arms relatively uncovered? Presumably because there are bracers, armlets, or other things that impare the somatic processies of arcane magics. Breastplate has a lesser arcane spell failure than chainmail or heavier armors, also.
2. Can't I have armor custom made to lessen the penalty?
Yes, you can have it constructed with lighter, more flexible materials, like mythral.
3. What about proficieny? Why doesn't training in armor's use grant me a lesser ASF?
Because the feat / class level grants you knowledge and training necessary to use the armor as it was intended, in combat. Why? Because that's what fighters use armor for. Fighters don't use it to cast spells.
4. Can't I undergo training to make myself more capable of casting spells in armor, reducing ASF?
Yeah, that would be a prestige class. Spellsword, Bladesinger, and I think Eldrich Knight reduce ASF as features of the class.
5. Why don't Clerics incur ASF? Why can bards cast in light armor?
A cleric's somatic component involves holding their holy symbol out and chanting a prayer. It's more than one can do when bound or held, but less than a wizards arcane gestures involved. As for bards, their magics are based more are on their oral performances than on their arcane gestures. A bard's suggestion and a wizard's suggestion are essentially different spells, cast in different manners. (The same can be said of a cleric's hold person and a wizard's hold person. It's more of a cop-out than the Cleric / Wizard difference, but never the less, that's the explination I've come up with.
6. Why is using a rope or being in a storm demanding of concentration of clerics and bards, but armor is not?
Because outside conditions may affect your ability to focus on casting a spell as opposed to the more simplistic somatic compontents. If you don't concentrate enough, the jostling of rope or a nearby strike of lightning might interupt your spell.
Ultimately, it should be a caster level check or a concentration check, because ASF stands nearly alone with concealment as one of the few d% checks.