See below for explication, but I think the primary concern is scaling with some affordance made for consistency with worldbuilding. Given that, scaling DCs make some sense. If you want a number that fits the current context, it’s a lot easier to provide a scale and let the DM make up something appropriate than it is to provide a menu of options wirth the required numbers. (I do think numbers should be knowable by the PCs even in a scaled DC systems. Otherwise, you can’t reason about the approaches available to you, and it makes resolution less immediate, and that contribute to the fun.)
I take a fairly hardline stance personally that scaling is a design side tool that should be used to set the tone/structure of the world and system, but should not be an encounter design facing tool. Page 42 should have been buried in the DMG2, as a "if you want to change the underlying nature of reality, observe how this matches up to different skill DC" section, not referenced by standard DM or worse, player side mechanics.
My assumption is the DM will be picking obstacles the same way they do monsters, which is by evaluating what’s available at the intended level of difficulty. The party is Xth level, which requires a set of CR Yth level monsters for some particular amount of challenge, and I want those to be undead, so I can pick from Z options. For that same Xth level party, if the DM wanted to include a trap or barrier, then the DM is going to look at things of the intended difficulty and pick from that list. In a sense, these things are helping to reinforce the game’s intended milieu by ensuring the challenges match what the characters are capable of doing.
As you intuit, I’m not a fan of the progression treadmill. I prefer the approaches used in old-school games and static DCs like PbtA games. As you get better, you get better. If done well, your journey in competence goes from requiring a lot of help (and risk of consequences) to being self-sufficient (or nearly close to it). That’s how it worked in our Blades in the Dark game. In the beginning, we had to work together, push ourselves, and take devils’ bargains. By the end, we were doing solo scores and taking on big things by ourselves. I like that much better as a form of progression than what amounts to doing the same thing at higher levels but described more fantastically.
Yes, this is precisely what I mean by scaling in
kind instead of
magnitude. Players should be able to do more and more different things as they increase in capability. If you start with scaling or generic difficulty, you end up mapping a DC 5 Jumping challenge is a narrow pit to a DC 15 challenge is a broad pit to a DC 25 challenge is a chasm, but the play procedure hasn't changed and the player's decision making is completely unimpacted by the game. It's just the adjectives used to describe the problem that are different. If skills map not to "scale of problem" but instead to "specific player abilities" then the whole decision space is changed as they level up.
I think part of this requires a deeper and more nuanced understanding of challenge/obstacle design in general though. Even something like a skill challenge (my least favorite innovation in game design) is really starting numbers first. I'm not actually interested in, as
@DEFCON 1 put it earlier, creating an "Ability check minigame." Instead, I want players to solve puzzles and problems and express themselves through gameplay choices against puzzles and situations. It's fine and good if a player finds a way to fix an issue in 3 die rolls, but it's horrible to design an issue that's solved (or is meant to be solved) by 3 die rolls.
Non-combat specialities are a WIP in my homebrew system, but that’s how things work thanks to static DCs. If you want to make a map of an area you explored (thus allowing you to find it again without having to navigate back), you make a Skill Check using Cartography + Wisdom. If you want to cast from a scroll, that’s Scroll Use + Intellect. Spells work the same way as skills, but they use your Mage rank the method instead of a skill. That avoids the problem of spells being categorically better than skills. You’ll still want to invest in skills because there are trade-offs to using spells. They cost MP, which you can only recover normally with MP potions (that cost stress to consume) or by using a weekly downtime activity; and your Mage rank is derived from your level, but skills are bought with EXP and have a maximum rank of +5 that’s independent of level.
I don't actually think "roll to cast" is that fundamentally important in the skill vs. spell capacity question. They have different resource expenditures, generally time vs. limited charges, and it's pretty easy to modulate access to them, thus that having a lot of spell ability limits broad skills. If anything, I think low level utility spells often represent a better core gameplay loop than skill checks do. Skills are nearly always rolled reactively; a problem or obstacle happens, you roll a skill check to see if it affects you. Spells are proactive; the problem happens, you decide if you want to spend a resource to overcome it. That same thing expands to agency in general, skills rarely provide any ability to frame or alter a situation while spells have lots of declarative text that lets you do that.
There's a lot of hacks that would make sense to put more abilities in the skill system that steal some of that gameplay from spells. Imagine say, a rogue with a pool of "thievery points" they can allot to increase skills checks so they can hit specific DCs, or a specialist character that can assign a floating skill bonus each rest to change their capabilities, or a berserker that can overdrive some skills for X period of time and risk fatigue...there's a lot of different resource models you could layer on top of a skill system.
I like this approach. It’s similar to the argument that old-school skills were actually providing you with something more than the baseline. A thief who uses Move Silently is not just sneaking about but is in actuality completely silent upon success. I don’t now how common that view was though at the time, and I think the idea of using that approach in a modern D&D-like might be a tough sell for some. (See also: the complaints people have about skill feats in PF2.)
I blame a lot of my idiosyncrasies on a 3e entry into the hobby. It feels entirely normal and natural to me that skills should largely define their effects in player facing material, and the
real design question was what those abilities should be and at what rate they should be delivered to characters. I'm still baffled by how comfortable everyone is with just making up a DC. I would have once upon a time called an RPG without an attempt to spell out the applications of skills as an incomplete product.