The basic problem with charisma is that if all it does is enhance social interaction, only one person in the party needs it. Everyone else can pull dark cloaks over their heads, stand back and let the party negotiator handle in of the tricky stuff while they play the part of the ugly nasty looking stupid looking henchmen (and in some cases, looks need not be decieving).
Depends on the setting and scenario, though. If the adventure is playing out at the Imperial Palace, and you're informed to leave your ugly nasty-looking stupid-looking henchmen at the door, then the rest of the party will spend a lot longer doing absolutely nothing, and probably the one person in the party with Charisma can't catch all the clues in the Palace gala because he can't be everywhere at once. And of course, many characters have secondary goals that require at least a modicum of social skills: lead an army, found a thieves' guild, win the heart of the efreeti princess, and so on. You are not going to charm the Princess Tira'Qabal of the Thousand Molten Swords by standing in the back of the party with a black cloak on while your buddy says "You should hook up with my ugly nasty-looking stupid-looking henchman."
Of course, that feeds right into your next point:
The secondary problem with charisma is that it depends on the GM to make charisma relevant. Too many DMs take the tact that what you are supposed to do is fight. If you don't fight, you aren't rewarded with XP. If you try diplomacy when you aren't supposed to, it inevitably fails to yield a meaningful result.
Exactly. And it also depends on the players as well. You get much better returns on charisma if you're interested in goals that aren't best achieved with fighting; if all you want is gold and XP, yeah, that's also going to result in fighting at all times even if the DM is open to other avenues.
When I offered to run a 4e game for some of my co-workers, they voted for a Renaissance Italy-inspired fantasy game. Without much prompting from me, they gravitated more toward Charisma-notable builds: the rogue took a build that used Charisma as a rider for many of his powers, the "necromancer" used a warlock as the basis, and the newcomer who offered to play a leader settled on a warlord. All of these players have demonstrated strong interest in making allies, wooing romantic interests, getting involved in intrigues, that sort of thing. The swashbuckling milieu made it understood from the beginning that rewards like that would be common, if you were up to claiming them.
Rules tweaks like adding riders to certain builds that add given bonuses if you have a high Charisma help. But the real meat and drink of a game where above-average Charisma is common is incentives to impress, charm and inspire NPCs. And that probably relies on table dynamic more than anything.