I guess this is where I am getting stuck on this argument. The general position seems to be that somehow charisma based skills are not magic or special enough to be considered an exception in shaping character's emotions, but they are somehow more impactful than other skills, and therefore cannot follow the regular rules for skills. That doesn't make any sense to me. Either it has the same effect as magic or a class ability, and therefore would be something that I would consider to be a reasonable exception based on the effect, regardless of whether or not the source is technically magic. Or, it has the same level of impact as every other skill, with the check providing a basis to work from without absolutely forcing any particular result. I really don't get giving it the effect of magic, but because it's a non-magical source, it can't be used against the players since only magic can effect players that way. Isireth's examples, to me at least, seems to put these skills in a category all their own, creating a lot more work than simply accepting them as either similar to magic or similar to other skills, and simply using the appropriate usage and restrictions for which ever category is chosen. Trying to treat them as a little of both just seems weird to me; not bad, just weird.
If it seems like it's Magic vs Skills, well... it kind of is, but maybe not for the reasons you think.
Magic spells are discrete bundles of rules. They have clear effects. A character casts a spell and you just follow the rules: Roll a saving throw vs a DC (or attack vs AC), apply effects. They are pretty straightforward.
Also, IMO based on the wording chosen in the game, mind-affecting spells in 5e are more like hindrances or obstacles than game-enders or robbers of agency. A Fear spell used to make you run away from the caster; now you can't willingly approach it and get disadvantage to attacks while it's still in sight. The difference may be subtle, but they're night and day game-play-wise.
Skills, on the other hand, operate mostly by fiat. The rules aren't as hard-coded as spells. The game offers no explicit effect to using certain skills in specific ways, only guidelines.
Also unlike spells, skills are not a limited-use resource, so it's something I have to consider when I make my ruling; I can't have something used at-will be stronger than something else which is limited, it wouldn't be a fair game.
There is also the issue of setting a DC to succeed against the PCs. Spells have a clear rule for it, skills don't. Whatever is considered fair will vary too much from one person to the next to be of use, in my opinion.
So there. One of the reasons I won't use skills to influence my party's behaviour is because they don't have clear rules or limitations. Even if I had the party's buy-in, the game doesn't provide me with tools to adjudicate these rolls in a consistent manner. That's why I treat them differently from other skills.
But, FWIW, I'd probably rather get rid of mind-affecting magic altogether. I very rarely use it against my party, and when I do, it gets a point across.