I like the idea for standard encounters but at the very least it would need tweaks for other encounters.
I had one session that involved a huge battle that lasted well over 100 rounds. Often there would be "spend 4 rounds to get across the field" or "everyone roll 20 times, tell me how many...