I looked up a horse length is according to Wikipedia it's 8ft or 2.4M from nose to tail. Centaurs would approximately be a foot shorter in length as they're nose and neck don't extend out the same way, so maybe about 7ft in length.
Being 7' (Athasian Elves) or 8' (Firbolgs) in height is still qualified as being medium for some humanoid races, so you could still sort of work in that Centaurs are medium sized even if they're probably about as tall as those 2 examples as well. Certain measurements such as length aren't all that determine what size medium is, as some snakes could be 10' or 15' and still be medium.
As you note, it's more than just length/height, though. It's bulk or weight as well. Not to mention being a quadruped vs. biped. An adult horse typically weighs 800 to over 2,000 lbs. Try using a shove attempt against that as a human in the midst of combat.
If the issue is that you don't want them to wield large weapons, that's simple. Despite the size of a centaur as a whole, their humanoid portion is human-sized and makes perfect sense that they'd only be able to wield medium-sized weapons.
I prefer the large creature wield medium-sized weapons better than medium creature that can carry more because it also addresses things like the bulk in terms of squeezing through narrow passages, grappling and shoving, and things like that.