I am one of these people who believe that magic is powerful should be powerful and trumps non magic because that is its nature. So yes a magical spell like charm person is more powerful than a nonmagical skill.
"If you want to be a hero, play a spellcaster. Everyone else gets to be a henchman".
That's fine as long as you're up front with your players about it.
In my example the bard has killed many of the BBEG minions and his family he has ruined so many of his plans that BBEG just wants him dead well not just dead but hurting then dead.
... and then meets him and he seems kind of nice. Or seems like he'd be worth more as an ally than an enemy (obviously he's powerful and ruthless...).
Like I said - the guy isn't going to just roll over and give up, but he might offer that traditional "Join me, and we rule the world as equals, or stand against me and be the target of my regretful wrath" speech.
With that kind of mindset there is no way a diplomacy skill no matter how high should ever work.
Why, just because you say so?
Now if the two used to be friends like Professor X and Magneto then yes I can see using diplomacy to try and resolve the issue.
To me its not just the roll that should matter the attitudes of the NPCs should also play a part.
I totally agree. And so do the rules. Helpful creatures don't just hand over their loot and retire from evil. They just help you while remaining true to their nature.
Its like say jump I don't care how many ranks you have in it without magical aid you are not jumping safely across the Grand Canyon. I don't care how well you roll.
If they actually had the ranks to do that, they'd be at the sort of level where colossal dragons are gnats they crush with their bare hands on a whim. And that's just the wizards.
Now, I seriously doubt that you're playing at that sort of level, and if you were, I'd be highly surprised to find that it was the moment when barry the fighter jumped across the grand canyon that was when your sense of verisimilitude broke.
So lets just put this one down to a strawman.
Once my suspicion of disbelief has been ruined the game loses the fun for me. Both as a player and as a DM.
You are right friends don't just blindly obey but in the skill description they can become friendly and helpful. That last part is what I had an issue with in my game. Which was the bard expected the cleric to become friendly towards him and not to stay hostile and suspicious. The player was upset because under the rules the cleric should not have been that way.
I think you did the right thing in that scenario. Not being a supremely gifted negotiator, it's hard for me to imagine how the conversation that leads up to someone befriending the target of their hatred goes (however apply the same thing to, say, casting magic missile), but I don't see an issue with the result of that friendship being a grudging tolerance rather than skipping down the street holding hands.