Gark and Nark, orc brutes, are part of a raiding party that is assaulting a merchant's caravan that the PC's are defending. Hodo Bigguns, a halfling fighter, is standing between them and the wizard and ranger, who are creating all kind of havok among the other orcs.
Gark: let's cut that half-pint down, then kill off the squishies!
Nark: uh... okay, why not.
They both set upon Hodo, but in the first round, neither can lay a sword on him.
Gark: bah, can't hit the slippery little eel. He keeps getting underfoot! He's got me marked though, so you go on ahead, and I'll catch up if I don't start hitting him soon.
Nark: uh... okay.
Nark runs around and starts laying the smack-down on Mandarb, the half-elf ranger.
Gark continues to battle Hodo, and notices that he's much easier to hit now, so contents himself with trading blows with the little fellow.
So... there is no good reason for two brutes to focus on the halfling defender, because the defender isn't a damage dealer, while the wizard and ranger are. Most opponents aren't so stupid that they can't figure out that the defender is using their numbers against them. Since the fighter is only really good at tying up one, the other has no good reason to stay, and plenty of good reasons to leave.
At that point, the Hodo Bigguns has only a slight advantage in AC against a dwarven defender, and none over a human, who has probably put his stat bonus into an attribute that counts for fighters in heavy armor. Further, the human has more feats to play with. The dwarf has a whole host of defender-friendly abilities to make up for the +1 AC that the halfling enjoys.
The halfling will still make a good defender, but his advantages will only balance his natural weaknesses, rather than add to his strengths. His situational advantage will be big, but as my example demonstrates, it won't take long for opponents to figure it out, if they don't know about it going in.