Ok so a fighter is getting about 8 attacks a fighter roughly (3 rounds, 4 attacks on the first with action surge, 2 on rounds 2 and 3).My rough experience is that it only applies to roughly half the attacks that get made with a toppling weapon. Often less.
So 4 extra saving throws in every fight....which is still a lot.
I would also argue you said that prone characters you don't apply the effect....except in terms of conversation time I do. Happens all the time.
Player: I hit them for X. Oh yeah and my topple activates, that's DC...um....what's the DC....its"
DM: "don't forget he's already prone".
Player: "oh yeeeeeah....nvm."
Sure a save wasn't rolled, but I still have to take half a minute to remind them of that part. And again not that big a deal in one fight, but when its over and over and over and over again it does get quite tedious.