It's not a game-breaker for me, but it plays havoc with my preconceived notions of the game. Like pawsplay and Hobo have suggested, the ability to fly is more of a superhero game element, not a fantasy one. It's perfectly fine to have superhero characters in the game; it's just not something I care for.
Gandalf, King Arthur, Bilbo, Merlin...none of them fly. That's more of a Superman or Magneto thing. But I digress.
The ability to fly doesn't necessarily have to break the game, but it certainly can...especially when made permanent, and when combined with powerful ranged attacks or other magic. It takes a bit of finesse and tact on the part of the DM, to restrict flight in a way that is fair to the player and fitting to your game. And it takes a bit of trust and understanding on the part of the player, to not exploit the rules in his/her favor. This is true for any rule, though.
For us, we got rid of the fly spell. Characters can still fly, but they have to use potions or winged mounts (only) in order to do it. This gives the players what they want, without letting the game drift too far into Superhero Land.