Not at all. Flying PCs make great targets.
What would bother me more about them is the fact that they aren't relatable to most NPCs in my campaign. It always annoys me when a player wants to play alien looking characters because I either have to treat them like they are a walking freak show or I have to change my story to make them fit.
If you want a limitation on aarakocra hovering that "makes sense" in a realism standpoint, how about this: if the creature uses its movement to remain still, then it is not able to use any dexterity bonus to its AC, because actively not moving is directly in opposition to being able to move quickly in response to attack.
Why on Oerth is it considered "balanced" for a race to have unlimited flight at level 1?
If an unconscious non-bird can still benefit from their dexterity modifier when calculating AC, then a conscious, hovering bird-person ought to be able to do so also.

(Dungeons & Dragons)
Rulebook featuring "high magic" options, including a host of new spells.