Attacks while unseen (by the target) giving advantage doesn't seem one bit silly to me.
Don't forget attacks while effectively blind give disadvantage.
This is why attacks made in darkness you can't see in ends up with neither advantage nor disadvantage. Advantage from being unseen is cancelled by making attacks without seeing the target (or much of anything).
If your target sees you, you end up with disadvantage.
If you see the target, you end up with advantage.
If you both see each other (two Dwarves brawling in natural darkness, for instance) neither of you gets any 'vantage at all and again, the fight proceeds normally.
Instituting a house rule saying blind fighting defaults to disadvantage rather than "novantage" is fine.
But the rule granting advantage to people that attack you from an unseen position (from hiding, while invisible etc) seems perfectly normal and uncontroversial to me.
The context on that comment was in conjunction with also not being able to see. It makes sense when one can see and the other cannot. It doesn't make sense when neither can see.