There are a couple of problem areas that can slow things down if you aren't prepared for them which are worth being aware of.
1) Summoned monsters - if your players are planning on using these ensure that they have their stat blocks available and understand them. Particularly if templated monsters are being summoned.
2) Cascading effects - casting some spells can cause changes that require several numbers to be recalculated. If these are being used by players as buffs ensure that they do the recalculation away from the table so they can apply the effects quickly. If you are GMing, you can avoid using these effects to "debuff" the party.
3) System familiarity. This really the biggest one - the more familiar you are with the capabilities of your character and the rules interactions the less numbercrunching and lookup you need to do and the smoother things can go. Similar to this is the problem of analysis paralysis - if a player is spending too long on worrying which target to attack or which of their spells to cast this can really slow down the game.
As for the differences in speed between 4th and Pathfinder, a lot of that will come down to the style of characters you have. At low levels and/or with more straightforward character types (blaster wizards, healing clerics and so on) Pathfinder will run faster than 4th. As you go up in levels and class complexity (summoner wizards and so on) Pathfinder will likely begin to slow down. Most 4th characters are of a similar level of compexity across characters and throughout levels so they tend to scale better.
Similarly, the simplest 4e monsters tend to have a few more options than the equivilent PF ones but the more complex PF monsters are several times more complex than any 4e monster. This along with the "Solo" monster design means that a single/small number of monsters fight will run faster in PF but a larger, more sprawling one will likely run faster in 4e, particularly when using Minions or more complex monsters.
One thing that comes with the 4e design philosophy is that each fight should last long enough for everyone to take several actions - fights tend to have a minimum length of 4-5 rounds while there are still enough "one shot" effects in PF that it may last only one round or less. This will tend to make PF combats a little shorter, but the general principle is that a 4e combat will have more rounds but each round will be shorter. Overall, you are likely to find that for similar characters and challenges, the games will likely run at similar speeds.
As an example of what can really slow your game down, watch out for things like this :
Mage's Disjunction - Pathfinder_OGC
I've experienced the single action of casting this spell taking nearly an hour to resolve. If you are GMing, avoid stuff like that.