Correct. I hate running full combats with more than 5 or so monsters for this reason, and its a big reason why I developed Raid rules for my upcoming release, and also why I embrace minion rules. With minions and raiding parties, I can have swarms without any added time to the combat. Minions have the added benefit of becoming living traps.Again, in my experience, more monsters tends to extend fights, which increases the number of dice rolled, which increases the probability of non average results (critical, very high or very low damage, failed saves) but it doesn't actually increase difficulty-- at least not to the extent that the chart suggests.
On top of that, adding a lot of monsters doesn't do as much for a combat's difficult as a unique arena does. The threat of serious fall damage, explosives, roof collapses, poison, darkness, and so on can jack up difficulty and add a much more tactical element to the combat. And since most vanilla 5E monsters are really boringly designed, adding more monsters tends to result in adding more bite or claw attacks. Yippee.