To my way of thinking, you've got a whole bunch of different issues that you're tossing into the whole "we don't need classes" bag...
Basically I think that level based games are starting to shed their need for classes. They have been together for so long that most people don't see how that could be but we have Mutants and Masterminds and D20 Call of Cthulhu already that don't have classes and they work fine without them.
There's never been a "need" for them. It's been a preference in some cases and a deliberate design choice in order to focus on certain things in others. Calling M&M a "level based" game is going to annoy some folks, as they consider level as being used to limit the overall power/capability of a character, not as something to pursue (unlike D&D).
You've explicitly called this out as D&D instead of a general rpg thing, so in the context of D&D, classes are a core part of what makes D&D "D&D". If you removed classes (as some versions of d20-based games have done), plenty of folks might say it's a good game, but very few of them will say "Oh yeah, it's totally D&D". Whether or not you could play "D&D styled" games of dungeoncrawling and mass killing is completely beside the point; it's such a fundamental part of the D&D identity, you're not going to get rid of it.
In a broader sense (rpgs in general) there's been a pretty consistent split of non-classed and classed rpgs, so I'm not quite sure the point you're making.
If certain traits (feats, class features, talents, etc) are available in a certain way so you can't get them in stupid ways (ie they require prerequisites of level or each other) then which way you place them in order to create a class is largely not only unnecessarily restrictive on the player but on the whole game system. You are railroading the player down your view of a Fighter or Rogue, etc. Then you give them options to Multiclass and you negate your own railroading and the point of having Classes again.
Errrrr.... what? I'm sorry but this just doesn't make sense to me. Can you break it down further? From what I can figure out...
You might like the character building "mini game" of how to make a character, but not everyone does. Note, I'm not talking about making an _optimised_ character, that's a whole other issue. Some people love screwing around with putting points into this or that skill and tweaking every little aspect of a character. Others don't.
The funny thing is you'll find both of those people on either side of the class vs classless debate. Additionally, you've got a whole other group of person that wants "realism" of a greater or lesser degree. They frequently like classes because they can make a class that reflects their idea of what's "reasonable" or "realistic" for a person with that sort of background to have. Of course on the other side of the realistic fence, you've got people that complain how a class (and level-based for that matter) system forces you to have characters that don't necessarily make sense (high level scientists or other skill monkeys having high combat capability).
I know that certain settings need Archetypes that are laid out. However if we define them as Archetypes and just say that a particular player is close to or is deviating from a traditional Archetype is all we really need to do. If there is a social consequence to creating a weird character type then let that occur. Maybe require a background explaining how that character learned those strange combination of skills.
First, why is it that "archetype" gets a free pass and you don't hate it, but you hate "class" instead? There really isn't a whole lot stopping you from changing the name from "class" to "archetype" if it makes you feel better.
Both options are constraining choices. The one of the biggest differences is that if they're calling it an "archetype" instead of a "class" they game system author explicitly gives you permission to change things, usually by saying something along the lines of "Feel free to add [skill or power] from another archetype if it matches the player's vision of the character better."
D&D doesn't explicitly contain that instruction, but it's an option for any GM that feels like it. The problem isn't that the rules don't explicitly tell the GM they can do that, the problem is that too many GMs and players refuse to deviate from "Rules as Written"; that's a completely seperate problem and a failure on the part of the game group, not the game.
If you have some Archetypes laid out then the players will have an idea of your setting but most of the time the players are going to either want to create something traditional or not.
Ummm.... this doesn't make sense to me. To me it reads like you just said "Archetypes help the players have a feel for your game world, but most of the time people are going to want to play anything."
Which is sort of a "Yes, and...?" statement to me. Of course some players are going to want to play a stereotyped (another term for "class" or "archetype") character. "Dude, I wanna be a badass fighter like Conan!"
Classes provide just as much of a background for the setting as "archetypes" do. And again, not everyone is really going to feel like spending hours making some character. Honestly, if I'm expected to learn a huge amount of background information just so I can figure out what kind of a character to make? I won't play the game. Period. I'll find another group or hang out with my friends and just do something else.
The problem is again an issue of flexability. For people that want to invest minimal effort into the game, classes/archetypes/stereotypes are ideal. Same thing for people that have minimal experience with the game system. You know, early in the days of 3.0 D&D I saw an awful lot of folks that showed up and were curious about D&D but were nervous about all the rules they had to learn. The advice I saw frequently given? Find someone that knows the game and they'll teach you over the course of a few months. Regardless of whether you find that advice appalling or not, the complexity level of the game goes up a fair chunk when it's completely open/class-less.
Now, presenting a method for class-less play? That's something that could certainly be done and should probably be given as an option. However, that of course does make the issue of balancing the system much more difficult.
Why restrict the players or yourself as a GM and more importantly why go through the work of creating the classes in the first place?
Because some people don't see it as a restriction. You're confusing your subjective preferences with some sort of objective truth that doesn't exist.
Understand, I say this as a dude who's go-to system is PDQ, which relies on player-defined traits (in other words, a players stats/skills/etc are entirely up to the player); I'm a strong supporter of the class-less (and even level-less) approach to rpgs. But systems like the 3.x d20 system definitely have their place; I'm even working on several d20-based projects.
The class system provides a clear example of the sort skills and characters one can expect to find within a setting.
The class system provides a much lower barrier to entry for certain types of players. It's important to realise that the whole "some kinds of players" thing doesn't mean "For those players that have played rpgs for a while and their tastes have changed" or something like that. While it can be a factor in some cases, plenty of folks from the moment they try rpgs are better suited to one style of game system as opposed to another.
The class system allows for an attempt to have a niche for each player within the group, allowing the players to feel as if their character has a value.
In the specific case of D&D/d20, the class system is part of the process of trying to ensure a "balanced" game; balance being defined as a character being able to meaningfully contribute to an encounter, as well as a GM having sufficient tools to be able to design an encounter with the goal of it being challenging but not unintentionally insurrmountable.
Strange as it might sound to you, designing classes is where at least some folks dip their feet into the general arena of design. It's a starting point where people that are interested can actually sit down and think about the underlying assumptions of the game system, in terms of character competence, encounter difficulty, player expectation, etc. There's an awful lot of potentially complex stuff that can go into class design. The results of this may or may not be obvious in the final design of the class, but a well designed class has some thought behind it. It's also where folks play around some with mechanical subsystems, looking to tweak the game in "small" ways. Of course, certain gameplay changes for a class have far-reaching implications, but that's part of what a designer should be thinking about as well.
The class system provides a base frame of reference. Sure, you can do a game without classes, but it's harder to judge mechanical interactions in a number of respects. Starting from a fixed point with mostly consistent assumptions means it's easier for a designer (either company or homebrewer) to develop their creations.
In other words, d20 can work with a non-defined class/archtype structure. However, there's been a decade of work done within the context of the class structure which means that it's significantly easier to look at something and say "that'll probably work" and "That combines with that to produce gameplay we don't want; we'll need to limit one of these mechanisms in some way."
For better or worse, it's a part of the balancing system of d20. While folks bitch about how useless the CR system is (and it does have problems), the CR system is much friendlier for people that are inexperienced with either GMing or the game system in general. Lots of rpg systems basically shrug and say "do whatever sounds good... you'll figure it out eventually" when it comes to the actual question of designing encounters of some sort. The class system is a part of how that balance is set up in d20. Doesn't mean it's perfect, doesn't mean there aren't better ways, just sayin' it's a part of the assumptions and things would be much more difficult if it hadn't been set up that way.
Some people also like the minigame of character building, as I've mentioned before.
I think I've offered plenty of reasons for why D&D has classes and why it should continue to do so. Understand that in the context of whether or not "d20" should have classes, it's a bit of a moot point. The d20 system you're playing around with (released under the OGL) is out in the wild for folks to play around with and do whatever they want. Make it class-less or not as you want. But at the time it was put out there, it was released with the expectation that it would be supporting D&D. This is important because when you look at 4E, WotC have closed off the ability to system hack the rules the same way. Which means that as far as _WotC_ is concerned, if you're doing it class-less, you're playing it wrong.
The joy of the class/level system is that it allows characters to build characters that they know will have a certain effectiveness, and that it allows the DM to create adventures that will challenge those characters without needing in-detail knowledge of those characters.
I'd strongly disagree with you. It takes no effort whatsoever to find people optimising characters far in excess of what you'd expect a character to perform as. A rather vocal chunk of folks that dislike 4E explicitly dislike it because the characters "are all basically the same" and there isn't the ability to blow the character competency assumptions out of the water.
No, not everyone plays that way and not everyone has to play that way, but it's a strong component. What the class system does provide is an idea of what the designers of the game expected in terms of overall competency.
The class system can aid a GM in creating adventures, but at least in the d20 system a GM that doesn't actually have knowledge of the characters and just relies on a class name and a level? That's a GM that's going to have serious problems.
My own perspective and all, so it doesn't mean I'm "right" just what I've seen.