Some repetition, but just running with what's in my head...ya know, like how we did when it was "brain-storming"!
1) Casting in Armor is tied to the "nature" of magic in the game world.
Divine Magic is the granted/borrowed/bestowed/syphoned energies given off by the deities. Maybe they know/notice/care, maybe they don't. Maybe it is beneath their interest/involvement or simply a "[super]natural" way the cosmos works that doesn't require their interest/involvement..until, say 4th level spells. Then they start/take interest/get miffed[?] to notice when someone's using that much energy at once? ANYwho, no matter how you fluff your deities and their relationship to clerics, the "nature" of the energy/source that fuels divine magic is not inhibited by armor. That "flowing through" the person, someone mentioned upthread. Divine casting in armor of any kind is fine. Period.
Arcane Magic is the grabbing/weaving/harnessing/"playing like a piano" and then directing preternatural/supernatural/para-natural/occult/unknown energies of the multiverse that may or may not ebb/flow, replenish itself, be drawn from other planes of existence...or is it a "finite" -on a cosmic scale, but still ultimately finite- resource? No matter HOW you fluff the "nature" of the energy/source that fuels arcane magic, it IS inhibited by armor. It is NOT divine in nature...but rather "natural" in its origin...a normal existing part of the game world/universe/cosmos. Gaining access to it, either because of elaborate gestures and gesticulations, "iron/steel/metal interferes with the flow", or a hundred other...well, arcane reasons no one really has ever figured out, you an not access arcane magic while wearing armor.
2) Casting [Arcane] in Armor is tied to the race of the caster.
Depending on how you fluff your races and their innate relation to magic/magical energies/other planes/or a hundred other variables, you allow casting in armor for certain archetypes and not for others. Off the type of my head, the way I likes my own classes, I'd go with something like this:
Elves: casting in any armor. No problem. They are the top of the "magic being" food chain.
Gnomes & Half-elves: casting in up to medium armor/chain mail. Gnomes for their innate magickyness. Half-elves as a boon from their parent/bloodline.
or possible Half-elves [and Tieflings if you use such things] in light only.
Humans, Halflings & Others: just can't quite get the magic to "work right" if they are in armor. The best casters of these races simply can not become as adept in working with sorcery as innately magical beings.
Or to take it a different step/direction/level of detail: Elves can cast any spells in up to Chain...but can only use Enchantments in heavy/plate armor. Gnomes can use illusion spells in any armor, illusion or enchantment in medium, and any spell in light or no armor only. That kind of thing. Just depends on how/what you want to the fluff to be for your world's races.
3) Casting in Armor is tied to some "effect" of using [arcane] magic in the game world.
a la the OP's thing about whatever world/game that was that had a magical "heat after-effect" resulting from spellcasting. Like one would sweat after physical exertion...one gives off some kind of magic sweat-radiation after working out with magic. Casting in armor could be anywhere from uncomfortable to debilitating to deadly for the caster to constrain themselves in those finicky buckled leather and metal suits.
This can be taken in a hundred directions depending on one's fluff for Arcane Magic in the world.
For example
...
a) Arcane Magic is thousands upon thousands of energy-threads that connects/maintains/holds together [?] reality, unseen by normal mortals/beings. During spellcasting the mage grabs/tugs on/strums like a harp/weaves into ponchos/whatEVer...The grabbing/pulling/coalescing of too many threads into a single effect/area (say, perhaps, like once one gets over 3rd level spells...maybe with increasing chances of issues for each spell level) produces a sort of magical [seen or unseen] focal point - called "wizard's lightning" by some for the low rumble of thunder that follows any big effect spell- which erupts around the caster after their spell is cast [with or without observable effect in the game world. Wearing armor while doing so would result in a "mystic feedback" that would fry the mage's brain.
b) All Arcane magic [known or unknown to most people?] is actually exacting calculations that open rifts and/or "invoke" beings from other planes: enchantments occur by opening a "screen" [of sorts] to the Land of Faerie, illusions could be the same (though some use the Plane of Shadows instead
; a Fireball is a very specifically shaped, and utterly momentary, "bubble" in the Veil between the material and Elemental Plane of Fire; Wall of Stone is a "slab of open space" created between the material and Elemental Earth, so the stone -just sitting where it is on its home plane- seems to appear where the mage specifies; Featherfall/Levitate/Fly is just varying degrees and control over Elemental Air allowed to enter the material plane or Nature/Fey Spirits/minor Air elementals themselves summoned to "carry" the caster....things like Magic Missile or Shield are controlled "rips" with Radiant/Positive Energy, necromancy spells play between the negative and positive energy planes (with some shadow plane thrown in for good luck), conjurations are self-explanatory, etc. etc. etc.
Wearing armor while working this world's arcane magic could offer a [high or ever increasing with spell level] percentage chance that the "beacon of lifeforce" of the caster/the tear itself/maybe the "shiny" [the caster in armor]/[whatEVer fluff reason] when these rifts are created will draw the attention of something more powerful than the mage can deal with...letting something "in[to the game world]" along with the spell effect or pulling the caster into the other plane (almost invariably certain and immediate death). So casters are taught before they can even DETECT magic that you DO NOT, under ANY circumstance, EVER want to try to cast arcane spells wearing any kind of armor!..and dark/dull or muted colors for robes and cloaks are the best and most unassuming.
c) Arcane Magic is, for lack of a better term, a "radiation" that exists throughout the cosmos (perhaps it is the wakes/waves of this radiation flowing over/past deities as they move through the multiverse that actually constitutes divine magic?!?). It is, for normal beings, harmless. Knights and soldiers wear armor to no detriment. Thieves and rangers traipse about in their leathers unaware. Barbarian warlords brandish their battle axes and great two-handed swords into battle. But for the man or woman who wants to work with and wield this radiation, armor is simply intolerable. The focusing and harnessing and directing of the radiation into specific effects, their "spells", either simply will not work or begins/gives the caste an ever-building "radiation poisoning" [made worse by being contained within their armor?]. Now, their tolerance for this radiation increases, with exposure and time (i.e. gaining levels). But the serious practitioner of the Arcane Arts is not going to risk or limit themselves by donning armor (making a case for multi-classed characters if a player so chooses, understanding that their magic might be more limited or risky).
By the time the mage is casting 4th or 5th level spells, they can only cast a few per day without risking this "magic-overdose" and ending up in a coma or killing themselves. "Magicians" in courts and carnivals across the kingdom use their "simple and safe" spells (cantrips, 1st and 2nd level spells) for entertainment and minor utility (levitate, floating disc and the like). Only "true" Mages, like the grand wizards of old, dare to harness the real power!
4) Casting in Armor is tied to The Star Trek Defense:
Donning armor depolarizes the occult defaults of the caster. Due to the juxtaposition of the quantum surge, the wearing of any kind of armor produces a breach in the cosmystum, which as every mage knows, contains the whole of their personal supply of arcane energy. The cosmystum would then seep anti-magic! Infecting the space-time continuum and causing a magical feedback loop that would not only consume the caster, but produce a supernatural ripple in the fabric of the multiverse, destroying reality as we know it!!!
-------------------
That's about it for the moment. Big day ahead. I might have time for more later.