But you haven't provided an objectively good reason fore taking such a blanket stance... that's my problem with your statements, nothing in them shows me why this is a better way.
...snip...
But more variance does not equal... better...especially if that variance includes less fun/interesting/engaging options in it's set. That's what I feel like you are missing... you haven't shown why more is objectively better.
failure on my part to explain my point, doesn't invalidate the point itself.
the * point about route optimization is merely to establish credentials that I may know more about the topic of routing than the general population. It's possible that you too have done similar work. But if you haven't spent any time working on such algorithms, I ask you to consider that you as a DM may not actually correctly divine the true best path, therefore how could your NPC do so?
Let me try a different, and hopefully shorter tack. if your NPC takes the best path, then the PC has NO choice but to follow him. Any mini-games you put along the way, be it dodging carts, jumping buildings, etc does not represent a choice. The PC is forced to play your mini-games because there are no other viable solutions.
By planning it, you limit it and are more likely to eliminate equally valid and fun alternative solutions.
With what I propose, I don't see any reason you can't make a mini-game out of the shortcut or the building hopping (which is a shortcut). The difference being:
the Runner PC will just run after the NPC
the street-saavy PC will out-maneuver the PC by way of the backstreet knowledge
the jumpy PC will prefer racing across rooftops to finally jump down on the NPC
When the NPC takes the true shortest path, by definition, no other paths are better. At best, the PCs can only hope to navigate the one true path faster than the NPC.