The rules do not make this distinction. There is not supposed to be a way to tell which is first or second. Now, adding that in, sure, you could make the argument. But, I thought house rules were off the table here in this discussion. I mean, I have an add on in my VTT that tells me which die is first and which one is second. It's not difficult. But, it's not part of the actual rules of the game either.
But, sure. For the sake of argument, let's say you're right. Ok, so, in cases of disadvantage, you might be given a bit of information about why you failed - you failed due to disadvantage. Fair enough. That's satisfying my definition. It's providing any information about how the result was achieved.
So, in any roll where you don't have disadvantage or advantage, you would say that you can still tell? That you are given just as much information? In a normal d20 Test, how do you tell what caused the failure or the success when the die roll is only called for when it is narratively interesting?