Basically, 2e had something along these lines. I seem to remember a table where with XP you could "buy" certain aspects of other classes, like spell casting ability, fighter THACO, etc.
Oh that wonderful table in the DMG. Less said about that...
More to the point, isn't this the EXACT rationale behind Player's Options: Skills & Powers?
A character gained Character Points (CP) by race and class to spend on class abilities. While still slightly less flexible than true classlessness (IE you were still a wizard or a thief) you could customize your PC the way you wanted to. It came at the expense of other class (or race) abilities though.
Wanted a ranger who couldn't dual-wield but could use a bow awesomely? Done.
Want a fighter who fights in light armor well? Give up your followers.
Want a mage who casts in armor? Okay, but your going to have less spell school access.
Same with races. Want an elf who isn't a bow-master but can identify magic items? Or a dwarf who is not adept at dodging giants but deadly with an axe? Both are possible.
As a bonus, humans got 10 CP but had no starting racial features. That's 10 free points! Spend them on racial traits (that +5% XP looks niiccee...) or save it for classes!
In addition, you could get CPs (up to 25) for flaws (such as nearsighted or powerful foes hunting you) and CPs could be spent on weapon and non-weapon proficiencies (and thus you gained some every level, but they couldn't be used to buy new race/class features.)
It was a great idea that suffered some, ah, balancing problems. It was perfectly possible to min/max to hell with S&P (such as an elf ranger with a 14 THAC0 with his bow at 1st level) and some class features were basically "free" CPs (really, who kept followers the minute you could trade them for some other cool feature?)
Still, a better balanced version of S&P would allow some basic class model (warrior, wizard, priest, rogue) but the ability to detail (or min/max) the character you want. In fact, I'm sure Mr. M. Morris (aka Spoonybard) might still have his greatly revised DUSK versions of S&P classes kicking around somewhere.
Might be a good compromise of classless flexibility and D&D traditional "roles" (whatever they may be).