Speaking more broadly here--about the "indirect harm" argument--I find this to be some pretty hinky-sounding logic. If "indirect" harm qualifies, then it's effectively impossible to use charm person in a significant swathe of situations. Further, given the text explicitly speaks of harm to the target itself, it seems at the very least a stretch to start talking about "indirect" harm based on harm caused to others.
Like, I get that these spells are very powerful when they're allowed to be, and that many DMs are keen on trying to prevent that power from becoming abusive. But this sounds like trying to finagle a way to break the spell no matter what, while maintaining a veneer of "playing fair." I don't know whether this is a player's logic trying to break a charm cast on them, or a DM's logic trying to wriggle out of a failed save without any actual consequences. Regardless of the motivation, it strikes me as trying to invalidate the resources expended. I don't care for that.
5e is a "natural language" game. It is supposed to speak clearly, using straightforward meanings, without goofy gotchas or weird jargon. When I read, "until you or your companions do anything harmful to it," I don't think, "Ah, so if I do something harmful to someone it cares about, that counts as doing 'anything harmful to it,' and the spell will break." The thought wouldn't even cross my mind without it having been brought up here. By far the most natural, straightforward reading is that you have to actually attack the target of charm person in order to trigger that effect, though "attack" could be indirect (e.g. dropping a wall of fire on top of them isn't technically attacking them, but is still an offensive action directly affecting them.)
Given this is by far the most natural reading of the text in question, why should one consider any other reading? It seems to me that this "indirect harm" concept requires defense. Why should it be that indirect harm would break a charm?