I think it's difficult to extricate tradition from fundamental game design that makes the system work for so many people. We could have a "better" system for defenses, but AC and HP are simple and easy to grasp.
It's quite rare that you see someone make a perfectly reasonable statement and then just BLOW OFF THEIR ENTIRE FOOT with the end of the statement.
AC and HP are no question, the biggest, hardest things to get people new to RPGs to really understand about D&D (aside from Vancian casting), in my experience anyway. HP, like, so many people, even when it's explained well, just sort of see it as meat-points. Even people raised in the age of computer games, where every character as a red bar, can get confused by HP. And the number of times I've seen newer players get hung up AC and how it doesn't make sense (and indeed, virtually no video game uses a system anything like it, almost all use damage reduction primarily), is just huge.
I do think there's some real truth in your "fundamental game design" being hard to separate from "tradition". And I don't see HP/AC going away soon because sacred cows and the fact that they differentiate D&D and its derivatives from other RPGs, but they are not a good example for your case.
They cause problems which often cause hang-ups for new players too, like HP gets in the way of KO'ing enemies from stealth and the like, something virtually every new player I've encountered post-2000 expects to be able to do (and there are D&D-derivative RPGs which have systems which work for it, but the existence of HP as a thing you must deplete complicates the issue).
I'd like to look at the rest of your points, because I think they're interesting:
Flexibility and no core setting - Yeah this is "D&D thing", it's unusual to have an RPG with such specific rules but no core setting, and yeah it's an asset to D&D.
Complexity and simplicity - I just can't agree, post-2000, that D&D is particularly easy-to-learn or that it's necessarily complex and simple in the right places. I think I understand why people would believe that, but I've introduced quite a few people new to RPGs, to various RPGs, including 4E and 5E, and 5E has not been one of the ones which people were really easily grasping in terms of rules and how the game flowed. It doesn't perform great here, in my experience. It's far, far better than a lot of '90s RPGs, or stuff like Exalted, but it's nowhere near as good as things like PtbA and Resistance (which ironically new-to-RPGs players may adapt to more easily than dyed-in-the-wool players).
No rules for things we don't need rules for - This is an incredibly subjective claim, and I think it is actually a problematic tradition, not so much in that we need "more rules for social stuff", which I agree we don't, but in that, D&D wastes huge amounts of time on stuff we don't need rules for, and doesn't even have much in the way of guidelines/suggestions for stuff that would be really helpful. You see this hampering new DMs all the time in my experience. For example, no rules for knocking someone out from surprise or the like is an actual problem. Not having rules for that is bad. Spending tons of time on rules about overland travel? That's pointless wibble that 90% of groups will barely engage with. I think 5E particularly shows a real confusion about what we "need rules for", and Pathfinder was far, far worse, (3.XE was also terrible here), with huge amounts of rules for stuff we didn't need rules for. This is a place where things could be improved and where traditions about having rules for X but not Y are actively harmful imho.
Monsters - I don't think anyone thinks having monsters is actually a problem. I think the issue is much more that the idea that
everyone of X intelligent species is evil is the problem. Superhero movies almost never do that. They'll often have the villains be X species, but there will likely be some positive characters from X species too. And most villains in superhero movies are either ex-humans, the same species as good guys (c.f. Zod etc.) or unique beings.
Easy to comprehend magic system - I think I largely agree with you here, but I'd say D&D is very much in the middle re: easy to comprehend magic. I think there's quite a lot of complexity and even though 5E improved the situation, the way spells work in 5E is still fundamentally alien to pretty much all literary magic, which virtually all works on the basis of drawing from a well of mana, or exhausting the caster (or both), or is just limitless. I think we could improve whilst keeping tradition here by ditching the Vancian system entirely for every class except Wizards. Everyone else gets spellpoints or similar. It'd be easier to understand for 99% of new players and I think even an awful lot of old players would be happier - not like this is a shocking novelty either given there were people doing this in 1975.
Thank you for giving me some good jumping-off points to think about things!
I think the main "traditions" for D&D which I'd see as being worth maintaining are the race/lineage and class (inc. subclass/kit whatever) basic way of building a PC, combined with gaining levels to gain power. I'd also probably keep HP despite the problems they cause, just I think add a couple of simple systems to reduce those problems.
It seems to me that when you started doesn't determine what you like about the game. I, and others around here, also started in the early 80s, and like the modern game just fine as D&D.
Strongly agree. I've seen plenty of people who started later, even as late as 3E, who liked more "old-fashioned" approaches more than I did starting in 1989.
