I would honestly love to hear what you've done. What all do you do? I've fiddled with monster damage and HP. To me, that's nowhere near enough to get OSR-style play from the 5E chassis. How do you encourage out-of-the-box thinking? All I get it blank stares.
The blank stares are a lot like the Gen Z stare in the classroom, and I feel that hardcore.
I've found the best way to do this is to create challenges that require 2-3 steps to resolve by a single character, and I reward use of abilities and spells in ways outside their description but that make sense with the spell. The big difference I've found between 5E and OSR is just the sheer about of abilities in play. If those abilities ONLY do what they say in the desciption, like fireball for instance, then you end up having something that feels more boxed-in. However, if you tell them something like "Yeah, I'll let you use shape water to create stairs to get out of the pit," suddenly people are like "Damn these magic abilities CAN be used in creative ways."
I also created simple rules for fleeing, implemented Haven rests, and yes, increased monster damage.
What I've mainly learned is that whereas in the OSR your "tools" are material things, in 5E, your "tools" are your "spell and class abilities." In OSR, you have people planning how they'll use physical objects to achieve that plan. In 5E, you switch the focus from objects to the magicat hand. For example, you can use a spell like "Grasping Vine" and set up a climbing challenge where the cliff is breaking apart. Grasping Vine (and other spells) allow the players to save each other or mitigate failures. Or someone can cast entangle at the top of the cliff, preventing the biggest rocks from falling or slowing the crumbling of the cliff.
If a player busts out something that ends the encounter, then you have two options: 1.) let it ride, which I do maybe 1-3/4 times ; 2.) up the challenge so the spell doesn't work. So casting spiderclimb while climbing a cliff may seem like to ends the encounter, but a cliff that's falling apart and oozing some supernatural fluid requires more thinking.
I think some OSR GMs get really in their head that people pay a lot of attention to the character sheet. But there isn't much difference between looking at a list of spells and a list of random dungeon knick-knacks you've picked up along the way.
Once the players have realized their tools have use outside of combat due to permissive-but-fair GMing, it frees up the DM to create more non-combat encounters that FEEL satisfying. These encounters then force the player to have to think more about resource management, and that makes combats more tense and has promoted more non-combative methods for overcoming monsters. For example, you have a bunch of level 3 players hunting a pack of werewolves. Let's say 10 -- too many for level three characters to overcome. So they think, how do we kill them anyhow? They lure them to a closed in area, say a barn, filled with alchemist fire vials in the rafters. Then, when the wolves charge in, cast shatter on the vials and douse them all in flames, and then cast something like Arcane Lock on the barn doors, killing, crippling, or weakening the werewolves.
Lastly, one thing the OSR does a lot of is it will drop 20-100 potentially hostile enemies, like an orc warband, on the party immediately. I do the same thing in 5E. Players are often unaware of their own power, and don't think they can supernaturally push through these problems through raw force. Thus, they are encouraged to think of new strategies -- sabotage, finding allies, territorial advantages, etc -- to overcome it.
Always remember that humans are pattern-seeking, problem-solving animals, and we have solved some very hard problems when the chips were down. I mimic that idea in my games. Give the player's an interesting problem, encourage use of their tools, see the sparks.
