I believe that there are plenty of tables that do not really want a challenge, they just want to kill some stuff, find some loot and have a fun time with their friends, so the fact that plenty of people enjoy the game is not necessarily an indicator of it having the challenge level / balance figured out
Well, it sounds like its figured out
for those table. And no game is going to be all things to all people. I mean, we don't expect that in any other field of gameplay - that's why there are so many games (traditional as well as contemporary) in those fields (different parlour games, different board games, different card games, different field sports, etc). Likewise for RPGing.
Of course WotC has a commercial interest in persuading as many people as possible to buy, and perhaps also to play, what they are selling. But just as in any other field, you wouldn't want to treat their marketing pitch as an objective description of what the game can and can't do - especially given that there is a long history, in the RPG hobby, of exaggerating the scope of play that is possible with a given ruleset (mostly by ignoring all the procedures of play that are necessary beyond the narrowest components of PC build and action resolution, and assuming that the table will "plug in" the needed additional procedures based on their past experience or folk wisdom).
I’d argue D&D was never good at higher levels, I’d stop at level 12 or so and throw out the rest (yes, will never happen).
As to most published campaigns ending around 12 to 14, most groups never even make it there, if anything you need more 1-5 and 1-8 adventures
I've only got experience of high level play in AD&D and 4e D&D. I would say that AD&D breaks down around about name level - maybe once 6th level spells come online. This is because, both mechanically and in fictional terms, the game doesn't really have the resources to establish or resolve conflicts at that level of play, other than maybe mass combat. This shows itself in practical terms in high level modules like Vault of the Drow (which relies on giving all NPCs ridiculous magic items that "self-destruct" when the PCs try and profit from them, in order to make the maths work) or Isle of the Ape (which I think is just poorly conceived all around).
4e D&D worked fine at my table all the way to 30th level. Both the mechanics and the fiction were there, and did what they had to do. The encounter-building guidelines needed a bit of recalibration (at Heroic Tier, a Level +4 combat encounter is going to be pretty challenging for the PCs; at Epic Tier, that becomes Level +6 or Level +8), but the recalibration was straightforward and all the maths still worked.
eh, no ruleset is so all-encompassing and clear that there is nothing that needs a ruling, so the only question is where to draw the line.
I think you are using the word "ruling" in an idiosyncratic way - you seem to use it to encompass all GM decision-making. But (just to give an example) I wouldn't normally call a GM decision about (say) which PC a given NPC/creatures attacks in a fight as a ruling; nor (say) a decision about how a NPC/creature responds to an unsuccessful attempt to influence them.
By "ruling" I'd normally understand a "meta"-level decision about how to resolve some declared action, where the rules don't themselves establish a procedure.
I’d say the number of pages has a direct correlation to how many scenarios are covered in what detail and make rulings unnecessary, the fewer pages, the more decisions/ rulings that need to be made or the less universal the game.
There is no particular correlation between rules length and rules comprehensiveness. Classic Traveller has a shorter set of rulebooks than AD&D, but is a more complete game (in terms of the scope of feasible action declarations that the rules give advice on). Just as one example, Classic Traveller in its 3 books tells you how to resolve not only a fight, and an interstellar space journey, but also (for instance) how to resolve an attempt to purchase contraband on-world, and how to resolve the sale of trade goods. Whereas, despite their many pages, the AD&D rulebooks do not set out any procedure for attempts to deal in goods in a settlement. All they have are price lists, and rules for percentages that will be charged for the sale of jewellery or the changing of money; but no actual process for working out whether attempts to buy or sell will be successful or not.
This is only a weak criticism of AD&D - it is a game aimed overwhelmingly at dungeon-crawling and wilderness exploration play, with the idea of settlements as somewhere where play might happen being mostly an afterthought. I make the point simply to illustrate one (of many possible) counter-examples to the assertion of a correlation between length of rules and comprehensiveness of rules.
DW lacks a ton of clear rules, for example take any of its monsters and compare them to their D&D counterparts.
Dungeon World has perfectly clear rules for how to resolve what happens when a player declares an action vis-a-vis a monster. They just happen to not be wargame-type rules.
If a specific DM abuses their position from your perspective then find another DM.
I don't think the notion of "abuse" is very applicable in this sort of discussion.
If a particular GM is not able to follow the rules and procedures of the game, that may mean they're hopeless or it may just be that they need more practice - some games (including, notoriously, D&D) have fairly complicated rules and procedures.
If a GM doesn't
want to play a game according to its rules, but rather according to some others - most often, in the context of RPGs, this means that they want to be free to ignore the action resolution rules if they have their own preferred view of how the fiction should unfold - then I think it makes sense for them to be upfront about that, so that other participants understand what sort of game play is on offer. (The idea that it is an inherent part of RPGing that the GM is free at any time to ignore the action resolution rules is an unhelpful one, that probably causes more confusion and unhappiness in the context of RPGing than any other single confusion about rules and procedures of play.)
But for GMing to count as
abusive I think would be at a whole other level, and not really about the way the GM approaches the rules and procedures of play.