Sure. And some people are less inclined to trust someone they find attractive. Or to dismiss attractive people as vapid and not worth listening to. And none of that usually comes into play at all if a human is trying to trick a goblin or cast a spell.I would argue that appearance can affect any or all Charisma skills. Many people are legitimately more inclined to trust someone they find attractive, for example.
I'd leave the effect of appearance to be defined by personality traits. I can easily see a character with a flaw that provides disadvantage vs. CHA checks from an attractive person. And if the DM and players have not defined those traits, then I'd leave it to the dice. 1d20 provides plenty of swing to account for varied reactions to any particular appearance without tying that appearance to a CHA score.