And of course, what if there’s additional effects of the spell beyond damage? Like ice storm? Is a round of difficult terrain worth a damage adjustment of some kind?
Yes, but the penalties are sometimes too harsh.
Ice storm for example covers a huge area and one round of difficult terrain can mean one round of extra hurt before the enemies can approach.