(Disclaimer: I think classes are part of what makes D&D, D&D. So classless would be better in a different RPG. That said, this sounds like a fun thought experiment so let's move forward.)
Okay, the chief word (well, words) here are D&D. So the goals must include to make a game that still feels like D&D. So we need to understand what classes have been used for in D&D are replicate that feel.
- Over several editions classes have been used for niche protection to stricter or looser degrees. A corrolary of this is that classes are a bit of a shorthand to communicate to other players, though this is not hard and fast.
- Classes gatekeep certain powerful abilities that most everyone would take if they had access to them. This includes not just powerful features but also things like d12 HD, or CON saves on a pure caster.
- Classes to a degree help provide variation - a sorcerer and a warlock play different. We don't want to end up like weapon wielders where some weapons are used frequently depending on style and others like the trident would barely change the game if it didn't exist.
- Classes prioritize different ability scores to create additional variation in characters.
I know there has been talk about feats, but I don't think they would be a good solo vehicle. To meet these goals there would either need to be long feat chains, which would be like classes under the hood, or other requisites to taking them that could bring us back to a 3.x situation where working out a build on how you will level ahead of time to make sure you can pick up what you want is more effective than just picking, and we don't want to force a specific type of gameplay on the players. (All of this is about using
just feats - using feats in addition to other aspects I think is a quite good idea.)
So how to do this? Everything below this part is separate thoughts then the goals - based off them but there are many ways to satisfy the goals. (And maybe the goals are incomplete.)
I'm thinking something like a priority grid like (early?) Shadowrun. Where you can pick what is most important, next more important, and so on by putting them in priority box A, B, C or D. (Or however many make sense.) I'd have more boxes than by default we would fill, with the assumption that some of the options will unlock some new categories - for example picking casting can unlock nature of spell lists.
That's good place to start - casting (full, 3/4, 1/2, 1/3, who knows) would definitely be a category. Full casting plus full martial would be, well, something everyone would go for, so we need to separate them. It would default to a small selection of very limited spell lists, with options put access to better spell lists in one or more categories. As a point of interest I would rate the spell list categories based on how far below the Casting priority they are, not based on absolute priority picked. So for a 1/2 caster it takes a lesser priority box to open up a spell list.
Basics like basic resilience and access to martial proficiencies is also something that can be put into a priority box.
Now the question is do we want to group things like paladin auras or ranger nature abilities as something we need to put into a priority box or have access via feats. I think we need to go back to the goals and determine if they are things we want niche protection on. My gut feel is that leaving those as feats is probably best, but some playing around to see if they become feat tax that everyone wants will help define it. Basically, if specific feats show up multiple times in most parties, they should be priorities instead. But let's tend towards more flexibility and reign it in as needed to meet goals - classless does imply more freedom in creating your character.
The priority boxes will preclude 5e or 3.x style "multiclassing", closer to AD&D/2nd multiclassing where you pick at character creation. But I'd add some of that back in with feats. But not all - if you want more then minimal casting, you need to put it in a priority. And while there may be a feat to increase your HD/HPs or to have more access to armor, the big selection for that is in priority.
