Ballpark calculation... if the monster is smart enough to target uncharmed PCs before charm victims, then each time someone fails a save, someone else will have to divert some of their firepower to damaging the person who failed the save, in order to break the charm. I'm going to estimate this has a net cost to the party of 1 PC-turn for each failed save*.
Bards, clerics, paladins, sorcerers, and warlocks are proficient in Cha saves. Three of those classes are also Cha specialists, and will likely have maxed-out Cha saves. The cleric and paladin have some incentive to invest in Cha, but rarely specialize. Everyone else is making do with a flat Cha bonus. After some back-of-the-envelope calculations, I estimate around a 60% chance to make a DC 12 Cha save (will be slightly higher for a high-level party, lower for lowbies, but that's the general ballpark).
So: If the party's offensive firepower is reduced to 60% of normal, then the monster's effective hit points are multiplied by (1 / 0.6) = 167%. So, factor that into its defensive CR. Typically it's going to be +2 to +3 CR based on the DMG guidelines, but there's a fair bit of flex in that.
This does, however, assume that the party understands how the monster's ability works, and adopts a policy of whacking each other to break the charm, and have the means to do so without causing too much pain. If they just keep focusing fire on the monster itself, then the effect is far more dramatic. Very quickly, the entire party will be charmed except for the monster's current target, and the combat will turn into a series of duels as the monster takes each PC one-on-one.
[SIZE=-2]*There are a lot of variables here which could cause it to be more or less than that. A dual-wielding fighter with 3 attacks could break the charm by diverting just one attack. But that attack might miss. And if the charmed PC's turn comes right after the monster's, that's another turn of not attacking. And the charmed PC is taking damage, boosting the monster's firepower. I really don't know how it all shakes out, so I'm gonna assume it's a wash and call it 1 PC-turn.[/SIZE]