1. Extreme complexity of high-level/epic-level play. 3.x is truly wonderful at low to mid levels, but at high levels it bogs down into tedious accounting.
2. Multiclassing mathematics. You could get hideously stacked saving throws by taking single levels in various classes.
3. Lack of a spontaneous divine caster in the core rules.
4. The fact that the later 3.x splatbooks came out with so many radically changing mechanics and concepts that a late 3.5 character would only be superficially like a core 3.5 character (Tome of Magic & Book of 9 Swords, looking right at you)
I am NOT going to complain about the existence of things like Hit Points, or Vancian Magic, or Healing Magic, or any other perennial D&Disms, because we are talking about a D&D edition here, not about designing your personal ideal fantasy RPG.
If you remove classes & levels, the six ability scores, hit points, Vancian magic, healing magic, and other staples of the genre you have nothing left that can identify the game beyond a trademark stamped on the book.
If you want a game which is so radically different from OD&D, Basic D&D, AD&D 1e, AD&D 2e, and D&D 3.xe (maybe even D&D 4e) that it drops common threads that appear in all or even most of those games, you might well be happier going to another game entirely.
D&D is a quasi-medieval fantasy game. It has the superficial trappings of the middle ages and renaissance, but with magical elements of high fantasy. You are going to have castles, and peasant farmers, and vast armies, and monsters, and wizards, and clerics, all coexisting. Trying to completely, totally re-work the implied setting away from medieval fantasy would be something else that cuts the heart out of what it means to be D&D in any edition.
If nothing else, understand that many games/settings imply that the high-level spellcasters that are powerful enough to shift the tide of a battle, or singlehandedly bring down a castle, or create food for an entire town, are very, very rare.
2e sourcebooks like The Castle Guide implied that a 14th or 15th level Cleric would be a religious leader on the scale of the Pope: the head of a major faith across the known world, and a 9th level Cleric would be something on the scale of a Bishop of a major city, with a local Parish priest being around 5th level, assisted by lesser priests. The vast majority of characters were implied to be 5th level or below, and 20th level characters being rarer than 1 in a million (2nd edition sourcebook Player's Option: High Level Campaigns explicitly said that a 20th level character would be a little rarer than 1 in a million in the population)
1e D&D, a monk couldn't even go over 17th level, and high-level monks were so rare that there were only 3 8th level monks, and only a single 9th through 17th level monk in the world (Oriental Adventures implied it was per religion, the PHB implied it was global).
In 1e AD&D, the leader of the entire Assassin class across the world was 15th level.
In 2e AD&D, the leader of the entire Druid class across the world was 15th level, with rules for higher level Druids reserved for retired former leaders of the entire Druidic order.
In the D&D 3.5e setting of the Order of the Stick comic, the PC's aren't even sure if there are ANY 17th level Clerics on the planet, and the High Priest of an entire pantheon, based out of one of the more powerful nations on the planet, was 11th or 12th level (the High Priest of the 12 Gods vs. Redcloak in the Battle of Azure City, the highest-level spell he used was Hold Monster).
A large town with a single 5th level Cleric isn't going to be able to feed the entire town with Create Food and Water, and a 5th level wizard (that might be the highest level wizard with a small army) might drop a single Fireball or Lightning Bolt on the battlefield and do a lot of damage, but that's his single big shot and he's then unarmored, poorly armed, and hopefully not unescorted.
In Dragon Magazine #5, there was the famous article "Gandalf was a Fifth-level Magic User" where it showed that every spell Gandalf cast in the LotR novels was 3rd level or lower in D&D magic.
I say all this to point out that high level characters in earlier editions were implied to be extremely rare, so that the medieval fantasy was kept by making high-level magic (and other powers of high-level characters) extremely rare, with most characters being low level, and spellcasters being less common than other classes.