The "at best its a wash" thing clearly goes against what other posters in this thread have explicitly mentioned (making a friend in the guard captain if you turn him over, or something along those lines), and it does directly affect their goal of "kill the grell" by having one less guy. But okay.
Nothing wrong with this approach, but it's hardly universal. I'll get to an example, below.
Okay.
Okay, let's look at a random encounter in a very popular TV series: Game of Thrones. Tyrion is captured, and on his way to await trail for a murder. Along the road, they are attacked by some hill tribesman. This does a couple of things.
One, it gives Tyrion and Bronn a somewhat superficial but real bonding experience afterwards, which might directly lead to him offering to champion Tyrion later. And two, it shows that the hill tribes are dangerous, and in the area, which sets us up for when Tyrion and Bronn meet them on the road later, where they go on to play an important role for Tyrion.
As a player, I'd have no problem with this. You might; it's not directly related to your current goal. But, for me, this random encounter might lead to a very different campaign than one where I hadn't played through it. If Bronn never championed, Tyrion, how different would things be in the series?
So, that's why I like them. I get why you don't. You've said as much. But I'd only suggest skipping these scenes if they aren't fun for you, or if you don't want the results they can bring. I don't think it's a good idea to skip it as a general rule, because, well, that'd be a bad suggestion for my group. We find that kind of thing interesting.
Wait, you mention 50 pages of backstory on the desert at one point... I think we can reliably assume that something in there makes it relevant. But, not, now it's retroactive? And, it's okay to force players to interact with it if it's planned relevance, but not improvised relevance? Why does that make sense?
YMMV, and all. As always, play what you like