Quick thought experiment. In the “shared story, not advanced board game” - can you imagine a different part of the story wherein the characters need FEWER capabilities? Would they be alright with de-leveling so their capabilities matched the needs of the shared story?
I can’t (and therefore won’t) answer for your players. But I wouldn’t be alright with that if I were playing and you said, “ok, Brad we’re dealing with goblins again this time, so it’s back down to level two.” I suspect I’m not alone there. Advancement is a fairly big part of the game. Getting new abilities, new spells, and new magic gear is part of the appeal. And you see this across platforms and games of all stripes, not just tabletop; leveling is a measure of advancement and XP is a measure of leveling. Gaining XP feels like accomplishing or working towards something.
I’m not knocking story, or any in-game motivations. I’m just saying there are other motivations, advancement being one of them, and there are some ways to handle advancement (engaging and less so) just like there are ways to tell engaging shared stories and duds.
IMO, ignoring XP because it’s too gamey or too meta or too hard risks overlooking it’s value. And, it’s been around SO LONG now that people are accustomed to it, anyway.