I mean, the fact that you call it a compromise rather belies your words that you think characters should suffer it because it's a thing that happens narratively. If you genuinely believe that, why is it a compromise? What's being compromised? The idea that only Paladins, Clerics and maybe Warlocks should suffer from this? Or did you misspeak?
I think it's a story beat that you have to agree with players beforehand (not always immediately before, you could agree it as a general principle, though you'd need strong buy-in if you did that), just like a number of others. The difference between stories - as in books, radio, movies, TV, etc. is that no-one is playing those, and if you're playing Superman surrounded by Kryptonite for three episodes of Smallville or whatever, you're still getting to act like normal, still getting paid like normal, and so on. Players though, are losing something, they're losing the fun of playing their character the way they enjoy, and they're not getting anything in exchange. That doesn't mean it's out, but it does mean you should agree it, especially if it's going to rely on anything remotely subjective, and all the examples I can think of do (except, ironically, stealing a Wizard's spellbook, which you could do fully RAW/RAI with no subjectivity).
I mean, there are games that recognise this, and have been since the 1990s, I note - games which reward the character when their weaknesses are preyed upon, giving you extra XP or hero points/bennies/whatever. 5E doesn't have great tools for that though - in 2E if I'd done something like this I'd have probably given them a whole heap of extra XP for playing through it well, but I bet you some mean-spirited 2E sourcebook would have said the opposite lol.