Some observations and thoughts.
The game has shifted progressively toward player enablement for decades. In OD&D and AD&D, the game was presented as one where the the DM is in charge, and the players are coming to play in their game. The DM, especially before the publication of settings like Greyhawk, Dragonlance, and the Realms, was expected to design the setting, assuming there was one. Many just played the published adventures.
Dragon magazine had new races, classes, and such, but only those noted or the ones written by Gary Gygax were "official" and even when actually published, were usually treated as optional. It was always up to the DM as to what options you had available beyond the PHB.
I think Dragonlance was the first setting to include a different mix of races and class options, with kenders and gully dwarves among other things. At the time, they differentiated the setting. The Realms had smaller tweaks initially (elves were taller than humans, which was different than Greyhawk and Krynn). But these were limited (in general) to their settings.
At some point, TSR figured out that there were far more players that DMs. The majority of the books (monster manuals, etc.) and adventures were published for DMs, though. They could greatly increase sales if they started targeting the players. I see this as a dramatic shift in design and presentation of the game. Now, there were books of a great many options for the players, and despite them being "optional," they wanted to play them. They spent the money, they liked the ideas, and that seems reasonable enough.
By 3e, instead of publishing player and DM books, they combined them altogether. So a new FR release, for example, contained setting information (primarily copied from the 2e sourcebooks), and new races, classes, feats, spells, and magic items. This pattern was very consistent (at least in the Forgotten Realms). This was an even bigger shift. Because now all of those options were canonical. The DM couldn't say as easily that "I don't like that race, and they are optional, they aren't part of the Realms." No, now they were officially part of the Realms, and limiting options was now seen as the DM taking something away. Things were sometimes still labeled optional, but in practice, anything published in a core or canon book is generally treated as not optional. I think feats tend to be the default for 5e for example.
The second major shift that this caused was an ever growing list of options. Initially, the options were presented with few mechanical differences. The earliest "Complete" books had very little mechanical impact, they were almost entirely fluff. It was focused on the role-playing aspect (pillar?) of the game. But that soon shifted, and mechanical aspects became a bigger and bigger part.
For example, in 2e, there are no mechanical benefits to joining the Harpers as described in Code of the Harpers. Master Harpers gain some benefits, but they are reminiscent of the Jedi Council, and you have to be selected by them to be a Master Harper. It seems pretty clear that the abilities are for high level NPCs, not PCs. The only thing that is gained when fully becoming a Harper is the Harper Pin, which is a magic item of some considerable defensive value.
But in 3e, everything must be crunch. So now a Harper is a one of a couple of prestige classes, gaining all sorts of mechanical benefits.
There have always been gamers that approach D&D as a game of rules and numbers to be optimized. Munchkinizers, min/maxers, optimizers, whatever you want to call them (often derivatively). This became a problem for many in 2e, simply because there was an enormous amount of power creep in the releases over the years. Things were often rushed, it seems that being a publisher took priority over game designers. So the quality of the game design suffered. All of these new rules provided lots of opportunity for optimizing, and even potentially "breaking" the game.
I recall for quite some time how most players I knew (and I'm sure I participated) disapproving of this new play style. Their focus was entirely on rules, and not on role-playing. But in 1e/2e, it was also relatively easy to avoid, because as the DM you were expected to modify the rules for your campaign, or just not allow it at all. I almost always allowed everything that was published at the time (including from 3rd party and magazines). I didn't have an issue dealing with "overpowered" characters, primarily because most mechanical benefits are combat oriented, and combat is typically only about a third of the campaign and not the focus for us. Other players were always more concerned about the story and narrative than who did what in combat.
In addition, we were caught up in the power creep as much as anybody at the time. The fact that the Realms of the novels was one of high magic and high level characters helped along with that. So we didn't really care if somebody was optimizing a bit more than the rest of the players. Nobody was well balanced in the late 2e era with options. We just went with it (and also largely got it out of our systems by the 3e crunch days - I didn't have many optimizers by then).
Another shift that happened was the introduction of the standard array as the default option for character creation. With that, every optimizer would know exactly what they are starting with. By studying the rules, you could map out "optimal" paths of abilities. I think this was the missing link for the introduction of the "character build." You could still discuss options before, but with a set starting point, you could design the "best" rogue in a way that everybody else could copy it. This was a key step for the player enablement - that a player can play the exact character of their choice.
Again, from a D&D player standpoint at the time, what did it most resemble? Magic: The Gathering.
The fact is, these differing playstyles always existed. I'm sure my perception of how things evolved isn't the same as everybody elses. But there are some key features that I think changed the landscape of D&D in a big way:
Player enablement is the big one for me. For many, it's not the DMs game anymore. The DM doesn't have the authority they used to have, for good and bad. Many, like me, still run the game that way. The setting, rules, and options for you to choose are in my domain. But the game itself is the players, and they are always welcome to challenge or question those decisions. Most of the time we find what works best for that group. I'm not sure if I have any non-negotiables. I can't think of any. This helps ensure
The standard array is another. The approach I use most often is to roll your stats, in order. That's what you're born with. You don't have any control over that. Mathematically, it's almost the same as the standard array (roll 3d6, reroll 1s once). You can roll up to six sets and select the one you like best (if rolling a single character). Usually we make 3 at a time, roll six, pick three.
Over the years, game balance has been all over the place. When optimizers weren't a huge part of the gaming community and/or the rules had so few options that there was little to optimize, being unbalanced wasn't a big problem. Wizards were horrible at low levels, everybody knew that, and people who played wizards really wanted to play wizards. Perhaps it was because they knew they'd be the most powerful characters by the time they reached 7th level or above, but you really had to stick through a lot to get there.
When the game has more options, there's more to optimize. Then balance becomes a bigger problem for most games. The late 2e/2.5e years were notoriously bad in terms of game balance. I think this is the time where optimizers really started to find their niche. Note that from a game design standpoint, designing an awesome set of game rules is much easier when playing to the optimization approach. Sure, you have to work hard to have lots of options and get them to balance. But the 3e designers did a great job of redesigning the 2.5e mess into a coherent and workable system. The problem they ran into was the same as 2e, though. Supplements, and the inevitable power creep that comes with it.
I think that happened largely because of optimization - new abilities had to compete (balance) with the best options of earlier releases. Which means that the less optimal ones became even less optimal. The rules had become the bigger part of the game. It makes sense, it's easy to explain, and easy for people to understand since they can compare it to other games and rules. It's a logical exercise. The RPG side is hard to explain. "It's like playing make-believe when you're a kid." OK, except most haven't done that in ages, and never learned how to do it well. Role-playing exercises in business training are probably among the most hated kind of training. Heck, the gaming community has a tough time defining role playing or RPGs.
The 4e rules proved you could write a well balanced set of rules that would scale, allow optimizers to do what they do, but not break the game. It is a beautifully designed set of rules. But for many (most?) it went too far. It lost touch with the RPG side. It was too crunchy for many who didn't want to optimize. For me it felt too much like MtG. I wanted to show my then 7 year old daughter how to play, and went to sit down to make a character, and realized that it was going to take hours to accomplish that, and she didn't understand any of the options because there was no context yet. Fortunately, DnD Next with Keep on the Borderlands was available. This is what I wanted. I started with the Holmes Basic Set and the MM. This makes sense - roll some abilities, pick a class, and go on an adventure.
Incidentally, I never could get into MtG. I'm not an optimizer. And I've found that you really have to be, because a large part of winning consistently is building a good deck. On the flip side, a friend of mine is a killer optimizer. It comes naturally. Within weeks of learning how to play MtG he would consistently win any draft he entered (until PAX - he wasn't that good yet...). He could not play D&D. Literally could not. He could not imagine playing a character that didn't exist. In combat he could select a spell, and that sort of thing. But anything outside of combat he could not do. At all. I've never seen anything like it.
With players coming from video games, MtG, board games, it's very easy to do what I call "playing the rules." This also leads very easily into wanting to optimize to those rules. It makes perfect sense. How do you play almost any other game on the planet? You read the rules and follow them.
You know where you don't do that? Sports. In sports, the rules offer boundaries. Take baseball. You have to hit the ball. You have to swing, or you'll be out on strikes. The field has boundaries where the ball has to land. Other than that, you can hit the ball wherever you are capable of hitting the ball. You practice hitting, and might be a power hitter, but always hitting for the fences also tends to mean you miss a lot as well. Other players recognize they don't have the power (and aren't training for it), and are focused on connecting more frequently. Most fall somewhere in between. You can hit it left-handed, right-handed, and you don't even have to take a full swing.
The rules don't tell you what you can do, they tell you what you can't.
The D&D rules have moved the other direction. More and more rules that tell you what you can do. They grant you an ability above and beyond what others can do. And if you combine them, you can sometimes do more. And since combat is among the most common encounters, and is the crunchiest (combat engages the mechanical rules), then it's only natural for some folks to focus on the rules, and which options and combinations are the "best."
And that holds true, if you play the rules. I teach a lot of new players, and I love it. My approach is the same as it was in 1e. Build a character in broad strokes: Do you want to be sneaky, a good fighter, or a spellcaster? There are four basic classes, fighter, rogue, cleric, and wizard. We'll help build the character, and get to the adventure. Tell us what you want to do, and we'll tell you how to do it.
Once they've actually played, then we can tweak their character - so you don't have to be a wizard, sorcerers cast spells a little differently. Fighters have a couple of options, or you can be a ranger or a paladin. We'll modify the character for the first couple of levels. They'll get to see other people play other characters. And they aren't limited to what the rules tell them they can do. They tell us what they want to do, we tell them how likely it is to succeed, and how to try. In other words, we approach the game by playing the character, and let the rules set the boundaries.
Is one better than the other? Of course not. They also aren't exclusive, unless you choose to make them that way. The type of campaign and DM can make a huge difference as to what fits best. The 5e rules do a great job in my opinion of balancing the races, classes, and feats, and also a good balance of crunchiness vs. role-playing. While there will always be some amount of optimizing, 5e has worked hard to make every option viable. That is, even when you're playing with a group that has some hardcore optimizers, the other players don't have to. In the past, that could be difficult for some groups.
