TwoSix
Everyone's literal second-favorite poster
I think the key here would be identifying "exploration moments" and "clever solutions" and somehow codifying them into something recognizable for the players to plan for.I want to talk a little about XP, how to award it to encourage certain play styles and behaviors, and what potentially unintended consequences those choices might have.
I will say straight away that I do not like, use or believe in milestone leveling. It is the players' job to go get XP, not the DM's job to give it out. Milestone leveling is arbitrary and by definition tied to a style of play I do not enjoy. After all, you can't have milestones without a path.
All that said, rewarding XP primarily for defeating foes encourages a style of play that i don't think is particularly desirable or fun. Rather, i want an XP system that encourages exploration and clever solutions to problems. The go to answer here is usually "XP for treasure" but that too comes with its own troubles. Specifically, in 5e in particular, money is essentially useless and I am opposed to 3.x style magic item shops and crafting.
What to do?
If you're running a hexcrawl or dungeoncrawl, you could say exploring a new hex or room fully is what grants the XP. Some hexes or rooms are worth more than others, depending on the difficulty of the room. Obtaining an important clue from an unfriendly NPC without combat could be worth as much XP as a challenging combat. Maybe successful (or failed) skill checks that advance the exploration of the crawl give XP.
To reward good character thinking, maybe earning Inspiration (assuming you use it) gives XP instead if the character already has Inspiration.
I know some OSR systems have Carousing, where blowing your treasure on absolutely nothing that has utility gives you bonus XP.
Maybe a spin on some of the old training rules? Maybe spending money on advancement gives the character XP, rather than bottlenecking them until they train. Spending your money on a fancy new arcane focus and inks for your spellbook doesn't give a numerical bonus, but gives XP. Or hiring out some local militia to do some practice duels with you. Or learning a new meditative practice that gets you closer to your god.