The problem with this IMO is it leads to GWM-syndrome. AC's are generally too low and getting over the -5 penalty isn't that hard for the benefit. In a like manner, if you could accept disadvantage to get a rider effect, etc., people would be using it a lot because hitting, frankly, is not hard in 5E. Against many creatures, the "risk" for the reward simply isn't really there.
Ultimately, it depends on how good the riders are, but given most homebrew stuff is more towards the OP side than the conservative, odds are it would be too much. I could be wrong, shrug, I would have to see some ideas.