I mean, I'm sure I could do whatever and the DM wouldn't notice, and if he did notice, he probably wouldn't care. As far as I can tell though, it would only add 1-3 damage, since I'm guaranteed to roll at least a 1 and can't roll higher than a 4.
I might max it out just for the sake of speeding...