This is part of what makes tight fiction-mechanics marriage especially good for newbies. Giving a strong character concept that is then reinforced with mechanics makes the wall o' rules mastery much less intimidating, and gives them a strong character type to anchor to.
My point, exactly. Classes, like pre-generated characters, make a good set of 'training wheels' while new players get used to the concept of RPGs. Through much of the hobby's history, new players started with D&D, which did have convenient class concepts ready to go, then when the limitations of that approach palled, they'd move on to other games. But, over the years more and more options were added, and, by 3e/d20 classes could be used either as complete character packages by new players, or as modular blocks to build to a concept by more advanced ones. It wasn't, and still isn't that clean - there are many options some available/necessary enough that they still represent a barrier to entry, the building-blocks when you start using them are, well, blocky, giving you low-res customization, and can also be meta-game optimized - but it's a side benefit of combining the disparate approaches of past editions that there something there for both new players and long-time ones, even if it does, ironically, assume a little familiarity or system-mastery to point you at the right newbie option. ;P
In the hypothetical world where I DM'd your setting, I would want my Guild Mages and my Thaumic Knights to be classes or subclasses or maybe feats or a faction I could get rewards from or some sort of mechanical heft that I can use to express the archetype.
Backgrounds and PrCs seem like the ideal mechanics for that sort of thing, depending on whether the organization is something you get into before adventuring, or after proving yourself in some way. Themes would be a nice addition, in that sense, a customization option that starts & stays with the character through his career, rather than being backstory or kicking in later.
This is part of why I say that more abstract/dissociated mechanics tend toward classless systems (and would actually include 4e in that bucket by some definitions, though that's a bit of a tangent).
Abstraction's a matter of degree. 'Dissociation' was coined for the edition war and had little meaning even in that context. I know it may /seem/ like a convenient synonym for abstraction, but we can already use abstraction when we mean abstract. If you're going to use us it as something distinct from abstract, at least pick & articulate a definition for that purpose.
Classless systems go pretty far back in the hobby's history, they're not a recent trend pioneered by 3e multi-classing or 4e class-balance/rules-clarity/mechanical-constitency. RuneQuest is a great example of a classless system and how much more versatile it is than a class-based one, and it's downright simulationist in retrospect, not very abstract at all on the scale of RPG abstraction - which, BTW ranges from highly abstract to overwhelmingly utterly abstract, and never remotely concrete, since we are sitting around a table rolling dice, not crawling through dungeons.
If the mechanics really aren't telling a particular narrative, then there's no real reason for them to be locked into a particular class (even one as broad as "magic-user" and "fighting-man"), and you should just be able to get the numbers right and then apply whatever fluff you want to it.
That's the logical extreme, yes. And it has been working extremely well for skill-based and point-buy systems for as long as they've existed, which is all but the first few years of the hobby's history.
But whether you have classes or not can be quite independent of how focused mechanics are on modeling just one thing. In Hero, for instance, you have an entirely classless point-build system, where fluff is all but entirely independent of mechanics, a 2d RKA is an attack, but it might be a laser, firearm, thrown blade, arc of lightning, divine flame, or a host of other things (with various advantages/limitations to customize it to exactly the net effect the player is going for to model the concept). In GURPS, OTOH, you still have an entirely classless point-build system, but here fluff and mechanics are linked. A fireball spell has it's own write-up and is different mechanically from a .45 or a laser or whatever, and that difference is married to the different fluff. Similarly, in 1e, the only least slightest hint you had that the same mechanic could be re-used for something even slightly different was a handful of weapon equivalencies (scimitar also includes saber or tulwar, kinda things) or the odd monster, like a young hobgoblin fighting 'as a goblin' or the like. For the most part, choice of fluff dictated choice of mechanics with no wiggle room. But, starting with 3e, you could re-fluff how things and characters looked, and thus, in a sense, what they 'were,' without changing mechanics at all. Your samurai could wear banded-mail re-skinned as O-Yori, and wield a katana (bastard sword) and hankyu (composite bow). What 3e did for the appearance of characters and gear, 4e did with descriptions of powers, though some mechanics, like Source keywords, were still inextricably coupled to certain types of fluff.
5e is as strongly class-based as ever, with things like Backgrounds layered on top of class rather than providing alternatives.
Like, it takes 4 hits to kill a monster equal to your level, you hit on a 9+ on a d20, and PC's are KO'd after 10 hits between "rests." Every 2 levels' difference adds +/- 1 to the d20, every 5 levels' difference adds +/- 1 to the hits someone dishes out. Lets add some variety: three times per rest, you can "spike" your hit to make it count double. Everything else is description. That's a viable system! A bit bland, but well balanced
Not so much 'balanced,' though. Balance requires more than just everyone being on a level playing field (that's fairness), it also requires that choices, in addition to being viable, be meaningful, and, well, exist in the first place. What you're describing is a fair system, but it isn't balanced at all, as there are no meaningful choices.
and easily customizable (and not a bad core to hang a lot of more involved mechanics on).
Once you hang more involved mechanics on it, though, it can graduate to balanced.
The more abstract the power, generally, the better fit it is for multiple concepts. If, instead of Counterspell, any character could take a "Negate" ability that simply cancelled out an ability of equal or lower level, that could be a counterspell, or it could be a kung-fu block, a shield bash, a brief stun from a lazer, the power of a Time Lord to turn back time a few moments and all sorts of things!
If you're rolling d20 vs a DC to counter/negate successfully, they're all equally abstract - and all the same mechanic, whether, the mechanic is coupled to only one narrative option (memorizing and casting one specific spell) or can be re-used for several (prepping & casting a counterspell from a book, relying on Faith to stand against hostile magic, or calling upon innate magical talent to cancel out another magical effect directly) or many (as you describe above).
The mechanic is innately very abstract, regardless. It's just a matter of the latitude given the player in deciding what available mechanic best fits his concept - or how to describe the character when using that mechanic - that's different.
There's plenty of variety, of course - lots of folks have specific roles for classes "except for fighters and rogues," it seems! We have a clear vision of what we want a Wizard to do, or a Monk to accomplish.
We've had a fairly consistent vision of the Wizard & Monk in D&D. But the vision of a Wizard, particularly, never had much to do with memorizing spells or managing 'spell slots' - those are D&Disms. For that matter the D&D Monk's odd lack of affinity for weapons & armor doesn't exactly match a lot of martial-arts traditions, either, just the 1970s American perception of martial arts.
As an aside, I kind of wonder if this is one of the keys to some of the perennial "non-magical is underpowered" complains that crop up - We don't necessarily know what we want a "Fighter" to do, or we all want different things. But a Knight! Or a Monster Slayer! Or a Commander! Or a Gladiator! THOSE are archetypes we'd see filled more robustly, with more agreement on how well it's doing one of those things. One class that does all those things can feel stretched thin - a class that did one of those things and ignored the others might be ultimately more satisfying.
When you come down to it, there's a D&D spell for just about anything any magic-using character in genre might have done, and a few more as foibles of the early game or mechanical artifacts on top of that, and, while there's a lot of caster classes, each has relatively few unique spells. The fighter, OTOH, doesn't begin to do everything characters in genre might do, and whenever the possibility of filling in the gaps comes up, it's in the form of a new class (Barbarian, Cavalier, Knight, Warlord, Swashbuckler, whatever), that's at least as limited in it's ability to emulate characters from genre, just with a different focus - and, generally, taking something away from the base fighter, as well.
So, no, I don't think any particular clarity of vision of the respective classes has anything much to do with casters being consistently overpowered through most of D&D's history. Unless it's a tautologically clear vision of casters as needing to be overpowered to be 'really magical.'