I am a relatively new DM and I dm for a group of 7-10 players that just made it to level 5. It's been really tough for me to come up with adventures that appropriately threaten the group. The group just finished an adventure with a cr 6 boss when they were at level 4. I thought that the cr was good, however they pretty much just waltzed through every thing.
All the things others have mentioned are true: D&D 3.5 edition is not optimized for 10 players; pitting a big crowd of low-level PCs against one super-high-level enemy will probably result in the monster getting some one-hit kills in before going down; terrain & traps will help; a mass of enemies concentrating fire on a single PC will probably kill that PC.
Really high level monsters have resistances that low-level characters might not even be able to get around. If the monster has a spell resistance of 25, the odds of your 5th-level spellcasters getting
any of their spells to work is pretty much negligible. So high level monsters are not a solution, unless you're willing to nerf their abilities.
Having said that, let's talk about tools to handle your situation. I've been there -- I just wrapped up a campaign with 9 players.
First, you should be using the
encounter calculator. It will show you how hard a challenge is. The challenge ratings increase like this: trivial, easy, challenging, very difficult, deadly, obscene. If I enter in 10 level 4 PCs vs. 1 CR 6 enemy, it is rated as challenging -- a little more difficult than the "easy" rating, but not much. It's certainly not going to be the "big tough ending battle."
Generally, I keep normal encounters to challenging levels, and the BBEGs to "very difficult." Back when your 10 PCs were level 4, the calculator says you could have thrown at them 1 monster of CR
eleven and still it would have only been "very difficult." Of course, the calculator doesn't quite take into consideration that a CR 11 bad guy can one-shot kill a level 4 character. So even though the level 4 PCs would defeat the CR 11 bad guy, they'd take losses. So let's look at lower-level bad guys in bigger numbers.
I put in your 10 level 5 PCs on the left, and on the right I put in 4 CR 4 enemies, and
also 6 CR 1/2 enemies. This is listed as a "challenging" fight for the PCs -- it's
run of the mill. It's what they should normally see, and normally defeat, if they're smart. There is one enemy for each PC, but the weak enemies will probably be one-shot killed by your PCs, while the 4 CR 4 monsters hang back. Why hang back? Because they want the PCs to expend resources and actions while they buff up or attack from range. This makes the combat rise to a more challenging level for the PCs.
And that brings us right into the next suggestion. To challenge this big of a group, you have to start being decent at playing the monsters. If you think of the monsters as objects you drop in front of the players to hold them off for a moment, rethink that. Try thinking of them as
your PCs that
you are playing. You want them to live, right? Give them a fighting chance. Think about what you'd do as a player to keep your guy alive. Obviously you cannot give the monsters immortality or other freebies that ruin their challenge rating. But you can play smart.
In the Goodman Games module, Cage of Delirium, there is what
should be a pretty scary, deadly enemy on the 2nd floor: an allip that has been advanced to CR 8. Now, I was already doing something right -- that module is for PCs level 6 to 8, and yet I was running my level 4 PCs through it because I knew that a group of 9 was bigger than the module expected. So it felt right. But regarding the allip in particular, I saw that CR of 8 and assumed that the level 4 PCs were going to really struggle to beat him. Instead, they took him out after just 3 or 4 rounds, and none of the PCs were significantly injured. After the game, I went back and looked at the allip. I read & re-read the monster stat block, and tried to figure out what I did wrong. Finally I landed on the "incorporeal" type in the listing, and looked it up in the DMG, and saw all the advantages it confers. Then I read here about how to make incorporeal types operate well -- they should be ending each turn hidden in the walls or ground so that it's very difficult to retaliate against them. They should use flyby attacks if they have them. Etcetera. Suddenly I realized that not only was the monster a good CR 8, but that if I played him intelligently, he probably would have killed every PC in the game. But instead, I had played him like a dumb tank -- he engaged the PCs, stood his ground in the open, took a lot of hits, and died. Suboptimal.
Don't forget that D&D 3.5 edition is all about the action economy. And a group of 10 PCs can levy waaaaaay more actions than a module expects. Your NPCs & monsters need to deny the PCs some actions, if they are to survive for a few rounds. If you have 10 5th level PCs, they can swarm an enemy and get 8+ attempted hits in the first round alone. A single enemy can't withstand that, especially considering that the D&D game was built expecting only 4 players and thus only 4 hits maximum per round at lower levels. So that's the advantage that a party of 10 has -- they can "frontload" combat with lots of hits, thus ending it fast. Your job is to get rid of some of those actions.
I'll close with one suggestion about the actions. While I mostly hate 4th edition D&D, one of the things they did right (IMHO) is mooks (or whatever they call them). These are enemies that fall after a single hit, regardless of how much damage was delivered. They are important because they eat up a PC's actions, but they are easy to manage even if there are a hundred of them, because they're essentially binary -- up or down. Fighting or fallen. There are no hit points to manage. So it's something to consider very seriously with a group of 10 PCs. You
need to give them targets, but in large numbers it can be nightmarish to manage. So we put together a team of enemies that includes a
few powerful bad guys that we really care about, and then a
lot of 1/3 or 1/2 CR enemies that are basically meat shields.
Good luck! Have fun!