The issue is conservation of momentum. The momentum the moving air loses when it collides with a person standing on the ground can be transferred to the ground, leaving the person stationary. The momentum the moving air loses when it collides with a person floating in mid-air is necessarily transferred to the person. That momentum will give them a non-zero speed relative to the ground, even if it is a very small one.
Of course, momentum may not be conserved in your game world, in which case the above would not apply.