Here are a couple of quick ideas for reducing damage calculations without making it entirely static. (I'm shooting from the hip halfway through my morning cup of coffee, so I haven't really thought these through).
One alternative would be to scale damage based on how convincingly the attacker hits the target. Just making up some numbers for this example, but maybe minimum damage if the attack equals the target AC, average damage if the attack is 1-5 points higher than the target AC, and maximum damage beyond that. Some problems I see with this are that you'd always need to make the extra comparison of the attack relative to the target AC, and I think it would scale damage up in favour of hard hitters down in favour of high AC targets (and vice versa).
Another alternative could again go with the minimum-average-maximum idea, but roll a second universal damage die that indicates whether the damage is minimum, average, or maximum. Perhaps a d8 roll indicating minimum on a 1-2, average on a 3-6, and maximum on a 7-8. This would be faster in play than the first option and shouldn't have scaling issues.
For both alternatives, critical hits could just double the indicated damage. If you want to add a bit more randomness, they could be the only attacks you roll damage for as normal (doubling the result as usual).