How can you pick up something off the ground 5 feet away from you without moving? Sure there is no rule about it, just like there is no rule saying you can't pull a torch off of a scone on the other side of the room without moving over there to get it, but it is not located in your space.Yeah, having it go flying is a house rule. But there is no rule saying you have to enter a space to pick up an object in it (nor any rule saying you don't have to enter the space).
That's why this is the territory of house rules or adjudication. Personally, I'd probably adjudicate that you can kick it away by default, and houserule that if you want to try to send it flying you can make the check with Disadvantage.
RAW the term for what I am calling a free action is "interact with an object" which has to be done "in tandem with your movement or action". I suppose the disarm attack would satisfy the action part of that, but it still is not located where you are at this time.
I agree you can kick it away, but that is an attack and you are using your reach to do it.