Is the idea of the big three PHB, DMG, and MM just sub-optimal these days (except for tradition)?
IMO, not really.
The base rules in the PHB are far less complete than are generally realized by most DMs and even apparently the writing team. So the DMG is adrift, but it really shouldn't be.
I run a D&D 3.25 game that forked off of D&D 3e prior to D&D 3.5 and which largely rejected most D&D 3.5 revisions. It has it's own version of the Player's Handbook that is 95% complete and available for players to use that is based off the 3e and 3.5e SRD.
But I still have a ton of ideas for a DMG that I will probably never get around to finishing. Briefly, aside from giving advice on how to run the game, a DMG needs to tell a novice DM how to run different environments and how to adjudicate the sort of hazards that will arise in those environments. So my DMG focuses on rules for falling, drowning, fatigue, roughing it, heat exhaustion, hypothermia, weather and weather effects, poison, diseases, addictive substances, avalanches, quicksand, lava, traps, and so on. It also has advanced rules for underwater and aerial movement and combat, and rules for running chases - situations where the normal round-based combat rules tend to break down and could use some modification. It also has the guidelines for awarding experience points, treasure tables, and all the other classic features
Just because the combat rules moved from the DMG to the PH, doesn't mean there isn't a ton of things left to cover. The thin content in modern DMGs is not a requirement. A well done DMG should feel essential to a DM. I still reference the 1e AD&D DMG a ton when running games just because I don't feel a modern DMG has surpassed it.
How would the classic B/X have been viewed if they were in three books instead of one each?
I think that B/X has the reverse problem. If B/X had come out not with the thin rules supplements for a partial game, but with something like the BECMI compendium from the start, I think it would have been more highly regarded. Or better yet, if there had actually been a unified game and the basic rules were actually an introduction to it.
I have mixed feelings about starter sets. You could do a beginner's box that had just the rules for running 1st-3rd level characters, maybe simplified class options, a set of dice, and a good introductory adventure as a budget way to get into the game and try it before investing in $160 worth of books. But it would have to be really well done. And the trouble is that I think a starter set made much more sense in 1980 than it does in 2020 when hardly anyone was familiar with RPGs.
My biggest problem with where D&D is at in terms of publication schedule is D&D doesn't ever seem to fill my needs as a DM. The books I want are never published. Instead WotC invariably publishes books that lack focus and are essentially bundles of supplemental rules touching every aspect of the game but which to me only add bloat and not value.
Fourth Book - Crafting Guide. I want a book devoted to how everything in the game universe gets made that has actually functional rules. And it should have a comprehensive price list that makes sense from an economic perspective and within the rules it lays out. This should include an actually balanced guide for pricing magic items, as well as extensive rules for alchemy and optional advanced rules for reagents that get rid of the handwave that in a major metropolis you can just buy blue dragon hearts or whatever else you need for your puissant creation. The rules should also discuss the cost of labor and different assumptions about currency to help new DMs understand how much a gold piece is worth or more to the point how much they want it to be worth and how to scale the prices on the basis of different assumptions about what is a daily wage. The rules should go on to describe typical businesses and how to stock a typical town with businesses and stock those businesses with goods depending on the wealth and technology level of the area. The rules should further contain rules for establishing and running small businesses and running games centered around that. Rules for abstracting out a business empire would just be bonus. There is just a massive amount of material to cover and it's rarely covered well and it's often scattered around a bunch of books. Note that all of this material is useful to both PC's and DM's.
Fifth Book - Tome of Magic. I want a book that covers all the magic that isn't part of the daily life of a spellcaster, but which touches on how magic works at a deep level in the game. This would include explanation and rules for familiars, holy sites, shrines, magical libraries, researching new spells, sacrifices, sacred worship, magical rituals, nodes of power, astrology, ley lines, artifact creation, magical pollution and taint and all the stuff that shows up in adventures as magical weirdness and all the things that show up in the story that NPCs do and why they are doing it. Also include discussion of what magical apprenticeships are like and different ways that magic might be treated in your campaign world depending on level of acceptance, ubiquity, and how mature magical knowledge is in the game world, and so forth. This is also a good place to dump all the narrow spells that you wouldn't want to bloat a Player's Handbook with, particularly spells NPCs might love but which PCs wouldn't prepare except during down time. As a bonus, variant magical systems could be discussed here.
Sixth Book - Tome of Battles. This is the mass combat supplement with rules for running battles at different scales, fortifications, naval combat and so forth. It's also something of an end game campaign guide, with discussion of reputation and rank, with rules for different ways of raising armies and the cost of maintaining armies in the field, and operational rules for armies degrading from poor supply, disease, bad weather, etc. Discussion of systems of government, taxation, and dissident action and revolts with an eye toward the players being on either end - either as the rulers or as the revolutionaries.
But no, instead we get a bunch of random supplements that have like 5 pages of material each I might use and as such aren't worth the cost.