Felon, again you rail against setting up false scenarios then do it yourself.
Using Charm in a City can have potentially hugely negative consquences. I DM in a version of Ptolus, an urban game that has been played for 2 years now. In Ptolus it is illegal to use Charm magics, or magics that control behavior. Makes logical sense, cause which one of us has not used Charm Person to make great deals with the shop keep.
In most games I have played, cities frown on charm magics.. If it is not illegal it is usually a "burn him as a witch" type response. So your city scenario is exactly the point I made before, better hope you do not have tocharm again, and are either away or safely hidden when the charm wears off because someone is, as Henry said, going to raise the alarm.
Secondly, Charm magics are never that great for groups of monsters, as once you charm one, the others get bonuses to saving throws and combat usually starts. A Sorcerer still has the option though of using enough Charm Person spells to control a band of orcs, not the Warlock.
Moreover, Charm makes the subject Friendly. It considers you one of its dearest friends, it takes your words seriously, but will not act in a self destructive manner. It will follow reasonable courses of action, but as your friend might ask you to help move

The spell says nothing about forgetting its actions while charmed, or not being aware it was charmed. If the greedy gem merchant, that never gives a discount, just sold you his latest consignment of diamonds for 1/3 the price, you think he wont think something is odd in that, when he has never met you before.
Also what happens if he had promised that consignment to the Local Big Cheese Wizard, the one that turns people to stone, (it is rumored) for displeasing him. Would that not count as a possibly sucicidal action and allow a saving throw. Cities often feature NPC that have more motivational depth than Grimlock Barbarian #6.