But that doesn't mean that one spell 'goes to sleep' and doesn't check it's triggers, only that the most effective result is applied when triggered. If there are two castings of booming blade, for instance, only the most potent effect is applied, but how do the spells tell which is most potent before they are triggered? How can the suppressed version know that they other is the most potent? Simply, they cannot. The effects are in existence at the same time but only the most powerful result is applied. One spell doesn't sleep and wait for the other to finish. Instead, both spells MUST trigger and MUST check their damage in order for the most potent to be applied, at which point the end of spell mechanism has already be achieved and both spells end. Further, you introduce the issue of how do you resolve the damage for the second booming blade? Do you reroll when it's applied, or take the roll to check against the first application, which causes issues of how did that happen, since it hasn't resolve. If you re-roll, is the first damage application a hard ceiling, or is this a new resolution and the old one was just for that trigger?
Both spells are active, both spells trigger at the same time, but only the most potent damage is applied, then both spells end. One spell cannot determine that the other will be more powerful and go to sleep until the first resolved. Otherwise, the 'most potent effect' rule has almost no cases where it applies, as you can use the logic of the sleeping spell from almost every case it would.
If we follow that logic, then
both spells have their damage rolled and a total amount of damage for each spell is generated. It's just that we only
apply the largest of these two totals.
It is do-able, but is this what 5E has us do? Specifically, do we note that the two spells are the same, choose the one that is the highest level, then roll only once and apply that? Or, as you suggest, we roll both sets of damage, but only apply the highest
result?
Let's see what 'Combining Magical Effects' on PHB p206 says:-
"For example, if two clerics cast bless on the same target, that character gains the spell’s benefit only once; he or she
doesn’t get to roll two bonus dice".
He does
not get to roll the bonus d4 twice and add them together; we all agree on that.
He
also does
not get to roll the bonus d4 twice and apply the highest result! He does not get to roll
two bonus dice
at all! Not according to the quote.
That means that the second spell simply does not even trigger, nor does it have any effect. If it did trigger then we would indeed need to roll a second bonus d4 for
bless, but only apply the highest. Since we know for a fact that the second d4 was never rolled, it follows that the spell did not even trigger. The only thing preventing it from triggering is the existence of the first spell.
That results in the conclusion that the second spell, while not being consciously 'aware' as such,
is prevented from having its effect by the existence of the already in-place effect of the first spell.
Since the 'trigger' and the 'end early' definitely are part of the effect (of
booming blade), then they, as part of that effect, are prevented from taking effect
at all by the presence of the effect of the first spell.