You keep saying that as if it were true. Grab an extra die. Same type as your regular damage die; you don't even need to think about which die to use. Roll them together and compare. No adding, no subtraction, no modifications. Which is lower? Use that one. No one thinks the same mechanic used for advantage and disadvantage is slowing things down. In fact, it is probably one of the best (not that there are many) innovations of 5e. And I have never played at a table where dice were in small supply, or designated the responsibility of a single person who thought one of each would be sufficient for the whole table.
By comparison, your method--which is reminscent of older and outdated ideas--requires more mental conversion and math. It may be simple and minor, but that kind of thinking is what adds up in the bigger picture. Which die is a step down from the normal? Don't forget -1 on your roll, and three other modifiers already in play! All the little things add up.
It was just a suggestion. It's not slower than what you're suggesting. That's why I brought it up. Do what you want with your project. Good luck with it.