It sounds like the major issue your having is suspension of disbelief issues. For this litmus test conditions need to accomodate the existance of magic. Let's think of any movie that involves the middle ages and the use of crossbows. Take First Knight for example. Now add in magic to the mix. Your group of mooks now will include probably a caster or two. Compare your crossbow wielder ambush to include a rogue or two with good skills in UMD, or maybe they have a ranger or druid in their employ.
Think of the difference it would make, if the PC's have an entangle cast on them and the crossbow wielders are far enough outside the AoE to not be threatened. Imagine that all 12 crossbow wielders, like the PC's would operate, focus all of their attacks on one target instead of divided amongst the group. Imagine the crossbow wielders all have even just 1 level of rogue. Add in an artificer or cleric to their mix to cast spells on their bolts to make them enhanced with either numerical or magical effect bonuses.
If I was running the encounter, -well first, my PC's would know not to take 12 opponents whom have ambushed them lightly, no matter what their level is - but back to the point, I would prepare as follows:
I would assume that whomever was trying to detain them would know something about them (unless this group was just out to test what their skill level was). Therefore, I would assess that the mastermind behind this ambush would know he/she was dealing with a 10th level party. Therefore he would send/hire at least 1 appropriately matched individual with a variety of underlings.
Let's assume a 10th level marshal was hired.
So the marshal gathers up the mooks (assume 1st through 3rd level). I'd give him special access and say that these warriors each had 1 level of rogue on them. Then I'd give him a maxed out UMD (with the appropriate feat), a spellcasting cohort, or a rogue cohort to make sure he could get spells like: blessed aim, recitation, chained magic weapon, etc.
It really would not be unreasonable to have mooks with a to hit bonus of +14 after stacking all the spells and marshal effects.
Combined with an entangle (or worse spiked growth) and a few slow spells, the PC's would be in a bit of a pickle - especially if everyone focused on one PC at a time. You're looking at a potential of 24d6+12d8 damage (12d6 from SA, and 12d6 from weapon enhancement) in round 1 to one target at an effective +16 to hit (vs. their normal surprised AC).
Even if your PCs were metagaming, I would have metagamed along this line of reasoning and realized that yes, even at 10th level, mooks with crossbows can still be quite dangerous. Its just a matter of you as the DM using the rules to your advantage. Suggesting just carte blanche to override the rules in this case will just piss off your players and for no good reason - you can easily add back reality to the threat that even low level characters offer.
The other thing you can do, unrelated to tailored ambushes is add in critical hit charts. This means that when opponents critical, it is far more devastating than just the occassional 2x damage. Now they have to worry about losing limbs, permanent hindrances like losing an eye, or being killed outright. That would certainly make me more cautious.