This is why you have the option to take the average instead of rolling. It's an issue that also exists in the standard HP rules. Jeremy Crawford covers those here, if you want to check them out:
https://twitter.com/JeremyECrawford/status/938885709783384064?s=19
That said, I see no problem...