No need to remove darkvision, but penalties can be broaden a little, and do NOT FORGET to implement DISADVANTAGE.
So you have disadvantage on Perception as normal,
add disadvantage on Investigation also, and thief tools when dealing with traps/locks as you cannot see fine details, this goes for most usage of any tools
disadvantage on Survival, tracks are easy to miss in dim light,
disadvantage on Insight(this will come rare) as you miss fine visual "tells" when someone is lying.
So, more or less, anything that can rely on sight in some degree suffers disadvantage