I agree with you that Devil's sight doesn't make sense from a logical point of view ... but magic isn't necessarily logical.
Some recent Sage Advice from Jeremy Crawford the game designed regarding Devil's Sight
https://twitter.com/jeremyecrawford/status/957031642979221504
"If you have Devil's Sight, darkness within 120 feet of you effectively isn't there. For purposes of sight, you can treat that area as if it's brightly lit"
"Devil's Sight has no interaction with dim light. It alters only how you experience darkness."
What this means for Devil's Sight is the following:
1) If it is completely dark all around you - you see normally as if brightly lit.
2) If it is completely lit all around you (e.g. daytime) you see normally as if brightly lit.
3) If you are in a large area that is dimly lit, Devil's Sight does nothing. If you have darkvision then the dimly lit area is perceived as brightly lit. If you don't have darkvision you can't see very well.
4) If you are carrying a torch that sheds bright light in a 20' radius and dim light out to a 40' radius then it gets even more interesting. Within 20' you see normally as if well lit, between 20' and 40' the region is dimly lit so if you have darkvision you see it as brightly lit and if you don't it is dimly lit and you have disadvantage on perception checks. However, from 40' to 120', the area is in darkness and Devil's Sight allows you to see that area as brightly lit again.
Based on the designer comments that seems to be how it is intended to work. Personally, I don't really like that approach but if going by the stated interpretation and RAW, Devil's Sight only affects the character's ability to see in darkness (either normal or magical).