I can see how you'd reach that conclusion, but no, I would never eliminate classes. They do constrain your options, but they offer major benefits to make up for it. Ability scores (IMO) do not.
The class system allows the designers to custom-tailor a suite of mechanics to fit a particular concept; ensure those mechanics work in harmony; balance them against other classes across a range of power levels; and present the entire package as a single choice to be made at chargen. This is incredibly helpful to the novice gamer, who can just pick an option off the menu and be done. And for the veteran, these customized suites greatly extend the replay value of the game. Bored with playing wizards? Try a warlock, or a fighter, or a rogue--it's a whole different way to D&D*.
This kind of thing, where you get a packet of mechanics bundled together in service to a concept, is one of D&D's strongest features. But I want the packets to be as independent as possible, allowing them to be mixed and matched. So you pick your class, and your race, and some skills; and ideally any combination of these things should result in an effective character. That ideal may be impossible to achieve in practice (and feats are a whole other can of worms), but it's possible to get tolerably close.