Plots are harder to pull off. The PC's abilty to gather information skyrockets, and the NPC's ability to gather information (without using cheesy Scry) decreases.
Combat slows down; all those options aren't that great when combat takes four times as long.
Some classes, like the fighter, are a lot less fun to play at high levels.
Almost every group has someone who doesn't know the rules, which is fatal in a high level campaign.
Combat changes so much it's hard for many GMs to challenge the PCs (especially if they're all rules-competent). Combat can often get "swingy" as well, with lots of over-the-top PC-killing and lots of wimpy encounters with nothing in between
Making matters worse, the ridiculous 4 encounters per day system breaks down even further at high levels. Spellcasters aren't going to run out if they're doing something other than direct damage, and in some campaigns the PCs can just take a nap anytime they want.
High level NPCs take longer to design, and are (comparatively) weaker, especially if they're not a spellcaster.