I wouldn't like taking the average for weapon hits. There's a lot of variability in a single die (or two dice), and that variability is part of the fun.
However, variability decreases as the number of dice increases, so I've made it a regular practice to do "partial averages" on damage from fireballs and the like -- for a 10d6 fireball, I'll call it 4d6+21. I do this as both player and DM, and my DM has begun doing it as well. The speed increase to combat is significant at high levels.
At the other end of the spectrum, we have a guy who rolls the same d6 10 times for a fireball, and he doesn't seem to even grasp why we're annoyed.