Right. The way the math works out makes it disproportionately likely that someone will flub it, even if they're all skilled.
If I'm running my anydice calculation right, it appears that if you have four PCs with a +5 bonus (say, 16 dex and +2 prof at first level), the odds that at least one of them will roll a 9 or less and fail to successfully Stealth is over 90%. Make that a group of 5 and it's almost 95%. So with a party of five, even if ALL of them are good at Stealth, it's about a 1/20 shot that no one will blow it.
Group checks (50%+) are just a simple mechanical kludge to offset the broken math.