The Elusive Shift is excellent, but it's more about the formation and evolution of the concept of a role playing game, and its discussion in the gaming community in the 70s and early 80s.
For more on Gary and what happened to him, I recommend Game Wizards and particularly the recently-completed 14 episode podcast When We Were Wizards.
Yeah, I'm not really interested in Gygax's story. I'm interested in the hobby and that transition.
He seemed to grow more and more obsessed with his income and status and efforts to turn D&D into a licensed media property, with movies and TV shows which could finance a lavish lifestyle for himself and his family and hangers-on.* And less and less interested in writing and creating game materials, or supporting, leading, and managing the creative folks back at TSR. Who were laboring long hours for little pay under the mismanagement of the Blumes, because Gary didn't want to be bothered managing and leading. Somewhat in his defense, he was clearly also belabored and demoralized a bit by, over the years, the death of Don Kaye, the lawsuits with Dave Arneson, his internal conflicts with the Blumes, the death of his mother in 1980, and the pressures of leading TSR when he clearly didn't know how to run a company but was determined to look like an autodidact genius and Great Man.
*(As well as his personal royalties for every book he could put his name on, and his sense that he owned D&D regardless of what any contracts said, and was entitled to every dollar he could possibly get out of it, no matter what).
Yeah, I've heard this part of the story before. By itself I don't feel like it explains Gary's apparent divergence from the rest of the roleplaying community. Or, I guess, his failure to diverge along with the rest of the community. I can accept that he was burdened by a lot that went on at TSR, but even before all that, by the late 70s Gary seemed to be both really knowledgeable about roleplaying and had some really mean-spirited ideas in the rules he wrote. I think he was clearly deeply conflicted in a lot of ways, but I don't really know why he didn't follow the transition like so much of the rest of the hobby did.
By comparison, looking at things Arneson did and wrote about, even in the early or mid 70s he was going through the Elusive Shift himself. Gary didn't. Gary kind of
never did in a lot of ways. That's weird.
I don't think Gygax is unique, though. Maybe it was just how they thought game rules should be designed. I think many of Zeb Cook's rules changes for 2e AD&D were aimed at curbing PC power, but they kind of come across as similarly mean-spirited. So many rules have consequences that punish the player. 2e also did weird things like make multiclassing and spellcasters both way better. I don't know.
I started with B/X or BECMI, but didn't play them long (like 5-10 sessions) before switching to 2e, and then went back to 1e and played a mix of 1e and 2e until 3e came along. But now when I go read OSE or B/X, that edition seems incredibly fair and well-wrought in ways that BECMI, 1e, and 2e don't. Like Thief still sucks, but every other edition of the game made them even
worse. The more I really look at the early design of the game, they had the best design and layout in Moldvay/Cook's B/X. Elsewhere they added a lot of options in AD&D and BECMI, but then they curbed power by
being mean about it. Like they have this presumption of curbing power gaming and punishing players that do it, instead of just not making designs rewarding you so much for power gaming! The whole of D&D after 1977 is at once enlightening and
baffling. Like why
wouldn't you always play an elven magic-user/thief in 2e, or an elven fighter/magic-user in 1e? It just seems correct.
I think it's Shadowdark to compare with that has brought it all into focus for me. That game took B/X, threw out the stupid early D&D attack roll, ditched the time-consuming initiative systems, kept the dungeon exploration system, and then bolted on modern d20 fantasy to fill the holes left by eliminating all the overwrought or cumbersome designs. It's not perfect, but it lets you see the early game through a different lens.