Depending on your group, it wouldn't slow down so much as annoy players. The initial idea of PrCs might have been to allow for specialized organizations, but they've evolved (for good or ill is not the subject of debate) into being akin to d20 Modern's "Advanced Classes" concept where you mix and match them for the abilities granted in order to better fulfill your character's "vision". There's nothing wrong with either method.
I see that most of the arguments against PrC-dipping (any dipping in general, in fact) comes from two schools of thought: One that PrCs are tied to organizations/should be special groups (Elite soldiers, SWAT teams, Special Forces and the like), and Two that a class is a profession instead of a package. I disagree with both of these viewpoints and I find nothing wrong with dipping into PrCs provided the requirements are met; requiring anything more than that is needless overhead in my opinion.
It's also my belief that PrCs should be better than a base class since you: A) have to meet requirements (usually involving sub-par feat choices), and B) Can usually take a PrC no earlier than fifth or sixth level, which is roughly one-quarter of a normal game's lifespan; If you're going to waste 1/4 of the game before being able to realize your "true potential" (i.e. your character vision/concept), you had better get something good in return for it.