There really isn't enough for me to go on here, but as you were there and had the extra information I'm missing, your interpretation rules.
It’s PF2, so there are some rule considerations in effect.
The previous turn, I had maneuvered next to the bandit leader so I could unleash a spell against him (we were low level, so it was a 1st or 2nd level spell). I used my last action to move away, and this is where I made my tactical error.
I had forgotten that the bandit leader had a reaction that allowed him to follow me, so at the end of my turn, he was next to me and away from the rogue. At the time, the rogue was in melee mode, so he couldn’t easily attack the bandit leader.
In PF2, movement costs an action (though you get 3 actions). So the net effect of the bandit leader using his special ability was that he lost an action getting back to where he started. The best call would probably have been to drop me and let the rogue waste an action getting to him.
But even if he cared about the other bandits, he probably should have attacked me once or twice before returning to the fray.
Overall, the effect was, I chase you down, decide you are not worth attacking, then waste an action getting back to where the action is.