I'm good with thinking like this. Why? Because that physical effect would include getting tired and fatigued. Which means that if fighting causes you to get tired and fatigued... then that absolutely can still occur even when you are missed with an attack. He swings, you dodge, you suffer the physical injury of fatigue, you lose hit points.
Easy peasy.
I think for me the distinction of suffering the damage
from the attack is an important point in the fiction. Not that the injury came as a result or consequence of the attack, but that the attack caused the injury directly. Which is to say, a hit PC saying "I sprained my ankle during a dodge!" doesn't jive as well with me. It is the HIT that did something to you. If you dodged, you'd be fine. And if you could take damage from a successful dodge that "made you tired," there's no reason you'd take more from ducking a red dragon's bite than you would from a peasant's drunken punch. Also no reason the former would also deal fire damage.
I mean, if damage from an attack can be narrated as a result of you avoiding the attack, then it can get kind of absurd: "The snake's fangs didn't touch me, I took damage and got poisoned because I ate some bad meat at breakfast, the snake missed me." Or "Oh, I dodged the orc king's fierce axe blow, and died because I had a heart attack coincidentally at that very moment."
A hit is a chance for someone
else to do something to your character. It robs you as a player of agency. That's part of its power, part of why it's something we want to avoid: it makes this story about how your enemy came out ahead this time.
So I'm not a big fan of "I'm just tired after all this amazing dodging I just did!" as an explanation for damage. Come on, man. If you dodged, you'd be uninjured. Your dodge was not good enough to avoid the blow. It might have been good enough to avoid it opening up your ribcage, but those fangs broke skin, that axe hurt.
That's me personally, though.