It won't take scores of rounds in a row. Boxers fight for 3 minutes before needing to rest, and they have virtually no weight involved, and aren't fighting for their lives, which allows them to conserve energy and fart around with hugs and such to get some rest.
In deadly fight, with armor and weapons, the effects will be felt enough to impact fighting ability after a minute or so. And yes, I've seen larger fights go 10 rounds or more. So yes, I need rules.
That's only if your goal is to emulate real life. Up to this point, we've been talking about repetition-based exhaustion--whether from swinging a sword or casting a cantrip--in more general, abuse-frustrating "You can't keep this up forever" terms. Mimicking real life is a new criteria.
Sure, if you want to simulate real-world exhaustion, you need rules for it. But then you're also getting into the whole "how much of hit points are meat and how much are avoiding damage?" discussion, and applying two different mechanics--exhaustion and hit points--to the same issue. It's not anything the D&D system is, or has ever been, meant to handle, and it's a ripple effect of complexity that I honestly can't imagine being an improvement.
Plus, characters in D&D are already unlike real life people in most physical respects: movement speed, encumbrance, ability to take physical damage (even with hit points representing a combination of factors), etc. etc. And that's without even getting into the stuff that's
supposed to be unnatural. Why are we looking for realistic exhaustion rules with characters that aren't--and
cannot be--realistic?
Again, play how you like. I promise I won't call WotC to come to your house.

But I, and I think most others, would be quite content with "You can't do the same thing for many minutes or hours on end," and leaving it there, even if that's not really in the ballpark of how long someone in the real world could do it.