Generally the desire to not risk one's life in a violent confrontation should supply a fair amount of reasonableness in "you want to leave our heavily armed group alone" uses of Suggestion, even if you don't say that.
But Suggestion is a tough spell that requires an impromptu precise phrasing by the caster and then adjudicating what "reasonable" means to the target, whose psychology they may not have really thought out extensively, on the fly, and how they would respond to the precise two sentences of magical suggestion. There's a complex totality of circumstances for the DM to weigh, and that's before getting to the more problematic element that the spell is often going to nullify some encounter the DM planned or screw up their plot entirely. So it is a lot of pressure on the DM all at once, and also a lot on the player casting it to find the perfect phrasing that will get what they want.
A DM who I sometimes frustrated with this spell hit upon the angle of exploiting the "if the suggested activity can be completed in a shorter time, the spell ends when the subject finishes what it was asked to do" element with legalistic precision. Thus in this case they would move aside and let you pass, but the spell would end as soon as you did so and the target returns to his normal thinking (and possibly realizes he was ensorcelled). I think this would be a fair ruling here.
My main guiding principle for a happy table when adjudicating Suggestion should be that if the spell succeeds the players get something, not necessarily what they want, but something, even if they have not crafted the most perfectly worded Suggestion they could. At the very least something interesting should happen, and the real failure here, I think is just that the resolution was boring. "Not reasonable enough" rulings are almost always just going to be boring downers.
If I were the DM, if your phrasing really was "step aside and let us pass unaccosted", I would probably let you pass, and then give you an Arcana check to realize that based on your phrasing the spell would end more or less immediately after you had been allowed to pass, so you had better break into a run once you are past them, or I would have the leader get in an argument with the other cultists and have you make an Insight check to realize that you could get in a surprise round if you immediately attacked.