That's true with monster damage, too; I roll damage for boss monsters instead of taking the average. Way more fun for me.
Here's something you'll notice. Because environmental difficulty doesn't and shouldn't scale with level, your heroes will be really good at lots of things. That's a great thing! Let them run through forest fires and breeze through locked doors. It will make them feel awesome.
One thing that doesn't change between levels is politics and moral dilemmas. In my game right now, I have a NPC who stole the throne illegally, and the PCs have orders to remove her. She's a much better leader than the old guy. Her reign is being destabilized by starvation, so if the PCs do nothing she'll probably be toppled, but millions of people will die as a result -- so the PCs are taking steps to stop the starvation, even though it legitimizes the illegal ruler. And of course, they can't just kill her without putting someone else on the throne, or the entire country becomes unstable and starvation gets worse.
You get the idea. Set up difficult choices that can't be solved by stabbing, whether the heroes are 1st or 20th level, and you'll add some fun depth to the game. Solving these with high-level divinations is really fun.
Oh! That's a good point. Encourage divination, don't nerf it. The fun part happens as they try to tie clues together.