To the point of "oh darkvision has the big drawback of disadvantage on vision checks, no color, etc etc." honestly in practice it's such a PITA to make sure that you're including consequences for relying on darkvision that I'd rather not have it... But I'm not the kind of GM that enjoys punishing players.
It feels like the "gotcha! You didn't specifically say you were being quiet so the ogre heard you stomping around!" style of play, which is not for me. "Ha! You didn't carry a light and were just using darkvision so that puddle you stepped in is actually a black pudding/ochre jelly, surprise!"
Again, id rather just not have darkvision in the first place than have to add in those extra punishments for relying on it. You could look at it in a positive way, that it adds more challenge and depth, but in practice to me it's juice that's not worth the squeeze.