That's something of a chicken-and-egg situation, I think. There's many tweaks that can be made to the design with the specific intent and result of making campaigns last longer provided the players and DM are willing.
My take is the game should be designed for the long haul, not to throw short-haulers under the bus but to allow both types of play to thrive.
I disagree that this would actually "allow both types of play to thrive." I'm quite confident that it would simply throw those who play short campaigns under the bus, as you put it. Note that I don't think there's any
preference for shorter-term play.
Instead, the game needs to be designed in such a way that both things are supported. Some things, you are correct that supporting one side really does actually allow both. Well-structured game balance is one example there, where it's a one-way function (you can break existing balance easily, just ignore all guidelines; it is very hard to
create interesting, well-structured balance from a jumbled mess). I don't think actively biasing the rules exclusively in favor of long-run campaigns is such a one-way function. Instead, I think a basic core can exist which supports both minimally, and then opt-in rules can help refine that on either end. Something vaguely like the original proposal of "rules modules" from the D&D Next playtest, a proposal which actual 5e completely and horrendously failed to achieve.
That's a mindset I'd like to change somewhat, not to force players to run whatever's handed to them but to be more flexible about what they're willing to play and be more ready-willing-able to quickly pivot from one character or concept to another.
Honestly? I don't think you're ever going to achieve what you want. People can be remarkably glued to specific preferences. And when we're talking about very deep, fundamental preferences--like how much or how little they get attached to specific characters--it's going to be a much more monumental task. I just don't see it happening,
especially not if you're trying to do it through the medium of writing different rules for the game. Because a significant chunk of players absolutely will see it as trying "to force players to run whatever's handed to them".
It ties in with the philosophical difference in char-gen between deciding what to play after you've seen the initial rolls and going in with a preset notion of what you'll play and then using the mechanics to build it. Me, most of the time I'm all about the former, and even if-when I come in with a pre-set notion I'm ready to pivot if the dice don't co-operate.
It's cool you can do that. I can't, and I'm not interested in becoming like that. I don't think I would actually enjoy TTRPGs if I
had to play them that way. I would just disengage from the hobby entirely. I very much doubt I'm alone in that. Whether or not it is a reasonable or warranted desire, a lot of people view TTRPG characters very similarly to beloved characters in fiction (books, TV, films, etc.), and become VERY attached to them. Telling them, "No, stop doing that, do
this instead" isn't going to make them
like doing "this instead" very much.
That's the world I - and most of the people I know who are older than college age - do live in. And good on you for staying committed to your campaign through the player turnover.
Indeed, WotC-era D&D characters take far too long to roll up! Tolkein fanboys aren't the problem, though; as 0e-1e D&D was quite Tolkein-based and one could bang out a new character pretty fast.
My experience is that 1e was much, much more Howard and Leiber influenced. Gygax's initial law-and-chaos setup isn't anything like the clear baseline morality of the LotR universe, the whole delving-into-murder-holes-for-riches thing is VERY much not in keeping with Tolkien's interests, and the fact that it, from Tolkien's perspective,
devalues dragons into being mere treasure-guardians rather than being both structural and thematic expressions of the work they're part of would have annoyed him to no end. (He had
very strong ideas about what a true, proper dragon should be, and explicitly said that he felt there were only two in all of literature up to that point: Fafnir and the dragon from
Beowulf. His two named dragons, Smaug and Ancalagon, both fit into this, as each is structurally important to the story being told, and each both advances and exemplifies the themes Tolkien wished to explore through his work, though obviously Ancalagon is rather less developed since he only appears in the
Silmarillion.)
There are certainly Tolkien influences (elves and dwarves as beings who are roughly human-sized was
definitely not typical of fiction before Tolkien, amongst other innovations directly due to him), but it's not for nothing that Gygax kinda rejected Tolkienesque influences, and why they only really took off after he was no longer in charge.
Another way of saying this is that the things you mention below (which get their own response) only occurred
because the Tolkien influences had grown. Those things didn't
cause people to view the story like their own table's LotR, but rather those things got added
because people had started to view D&D as a way to produce their own table's LotR, and wanted that feeling to actually be backed up by mechanics. (The fact that this also meant the game had bigger design space and could both ask and answer more complicated questions was certainly also a factor.)
The main cuplrits are a) chooseable feats and b) assignable skills. Skills came online in mid-late 2e, feats with 3e. Get rid of those, along with at least 3/4 of today's PC-playable species, and make everything hard-coded class abilities that come online at set levels and both char-gen and level-ups become (relatively) a breeze.
Problem: People like being able to customize their characters and express themselves through character-building. This is an extremely strong appeal of role-playing games in general, not just the tabletop versions. It's
the reason why "roleplaying mechanics" (meaning, choose-able character advancement based on levels, skills, etc.) have so egregiously proliferated through the video game space. Hell, these days even
basic shooters tend to have elements of "RPG" to them, again meaning "experience" and "levels" and advancement and picks etc., etc., because that's a huge way to increase the number of people who are interested in playing your game.
So....how can we square this circle? Your position is that the only way to get things back to the format that you believe would work is one that, demonstrably, removes one of the most popular elements of play in RPGs-in-general, a thing which video game RPGs derived
directly from D&D. If we do in fact do as you say, it's going to make a LOT of fans, including a lot of the ones picked up by 5e, very unhappy. It may even drive them away from D&D entirely, into other games' hands, which is a pretty significant threat now as a result of the SRD debacle.
How do we simplify character creation
without taking away the customizability that many fans demonstrably love?