That's an interesting point. I don't disagree per se, but I would contend that's actually the less important part of the problem, or maybe just the thing that you need to solve second before you succeeding at creating a challenge rich environment. The important thing for an engaging "challenge" heavy scenario, is first and foremost ensuring that your player's decisions matter. I think that requires three things: choice must be real (i.e. different mechanics must be invoked when a diplomatic or a violent approach is tried), decision quality must be evaluated (diplomacy is more likely to work than violence), and decisions must be informed (players know how the diplomacy and violence mechanics function and are different, so they are aware they are making a choice). You have to have all three before you can have agency and the enjoyment a challenge offers at all. I think 5e is generally falling down on this step, because it rarely provides enough information (often because the mechanic doesn't exist and the DM is making it up on the fly) and/or because it doesn't provide decision quality (you were always going to roll a similar skill check, no course of action was more effective).
The resource management game you're talking about is a next level concern, about how complicated and interesting the optimization case in any given challenge is. I think we've got more fundamental design problems in creating challenging gameplay, but it's definitely something that could be be improved. Assigning limited resources to solve an unknown number of problems is roughly the gameplay loop spells create, and if you aren't careful with limitations, the optimization case becomes trivial (use them all, rest, repeat). There's technically more than one way to resolve that, either by limiting the resource system, or by changing the nature of the problem (time limits, imminent threats etc.). I'm partial to week-long long rests that require a safe haven to stay in to enforce the passage of time as a threat, but if I'm spending design time trying to improve the challenge appeal of the game, I'd start with more clearly spelled out skill DCs.