P205 of the PHB. Combining magical effects. The effects of the same spell cast multiple times don't combine. Instead only the most powerful effect... yadi yada... etc...
You're not casting the same spell multiple times. And this rule doesn't apply to damage anyway. When you cast the same damage spell multiple times, you get the damage multiple times. But this is irrelevant, because, again, you're not doing that. There's nothing in this rule that brings it to bear on
eldritch blast or
scorching ray.
At the same time, it looks weird to me. It is only one casting of the spell. Not four castings.
Spiritual weapon is only one casting of a spell, but it can hit a bunch of times, and very definitely adds your Wisdom modifier to damage each time. One casting of a spell doesn't always have to imply simultaneity.
Heck, since it's a cantrip, you can even say that high-level warlocks are actually casting it two to four times a round. Like I said above, when casting the same damage spell multiple times, the damage
does stack.
Or, y'know, you can just say that they
are simultaneous, but that you add your Charisma modifier to each blast anyway because that's what the rules say and there's no particular reason why it's implausible that they work that way.