But that would be a huge change for the game- making magic unreliable. Mord's Mansion is 100% effective for what it does. There are no "random encounters" when you use it. Same as most of the magic in 5e D&D. Yes, we can keep trying to add "challenges" when the party uses Leomund's Hut or Goodberries or this or that or whatever. There are two problems with that though.
1. The casters are now driving the challenges of the game. We're not adding in random encounters that make sense for the adventure, we're adding in random encounters or events to counter using these spells. The only reason that these events are occuring is in response to the players using these spells. Which, at the end of the day, runs counter to why we use these spells in the first place. The whole point of using these spells was to avoid encounters and make resting safe.
2. And, I'm not sure getting into some sorts of magical arms race with the players is healthy. Well, the players now have resource X, Y and Z, so, we must up the challenges for exploration to match. It's essentially just a treadmill. Get stronger, face stronger opposition, get stronger... Again, the reason we're doing this is in response to the casters.
It's an arms race that I'm not interested in because it also has the knock on effect of making players realize that if they want to engage in the game, they have to play casters. Your fighter isn't going to be of much use traversing the Elemental Plane of Water, where you need things like Water Breathing, some sort of magic that gives you a swim speed and so on. All of that comes from the casters.
The tension between the magic system and exploration is very difficult to resolve.
The problem is, Tiny Hut, Goodberry and Co. wouldn't be a problem, if the Rest System wouldn't be so bad.
Most problems with Tiny Hut and Co. would be solved, if players wouldn't get just 100% of their spells back after sleeping/resting 8 hours no matter where they do it.
If you are on a 10-day Wilderness Expedition and wouldn't get spell slots back, because long rests only can happen in town, for example, Tiny Hut and Goodberry are suddenly very expensive, because Spell Slots become scarce.
But right now, except for maybe the first 3 or 4 levels, a caster barley ever runs out of spellslots, so utility spells don't really have an opportunity cost, because you always have spell slots left and because you always reset to 100% spell slots after a long rest, not using leftover spell slots before a long rest is stupid, especially of they can make the long rest safe.
And without a real opportunity cost, using such utility spells is always the best way.
But now imagine a rest system, that only gives a fraction of spell slots back. Suddenly, using Tiny Hut means, that maybe the next day you can't fight efficiently or help the group pass the chasm.
That is the big problems with 5Es exploration - that PCs are always just one good nap away from being at 100% of their capabilities.
The removal of any long term impact of battles, of traps, trivialises most challenges that exploration otherwise would pose.