I like your self-analysis to see if your thinking is consistent here, but please let me ask, would you be okay with a DM who "improv'd" that because X spell was cast on Y artifact, that artifact radiated pulses of fear for three rounds? Assume for the sake of the question that this sorta-basically made sense given the known properties of the artifact, but absolutely was not codified in the rules whatsoever? There isn't a "wrong" answer to this, I'm just trying to get a sense - I'm guessing you wouldn't like that either anyway.
Or what about a DM who rules that the roof instantly collapses because of a player casting Shatter right by a structural pillar (which the had no idea nor warning was structural, and indeed, we don't know if it was until the DM said it was), and then damaging and prone'ing and incapacitating PCs in the area (with no save for those except the ones on the edge)? I don't see a meaningful difference between a DM insisting my PC is incapacitated and prone, even though it's complete improv and I may find it entirely implausible, and insisting my PC is prone and incapacitated
I think from my perspective, if it's unreasonable for an improv'd non-magical effect to take away control, it unreasonable for a magical one too, if improv'd, and all improv'd infliction of serious conditions is probably crossing a similar line.
There's definitely a "degree of trust", issue, and a DM who is constantly improv'ing that his NPCs cause Frighten or similar effects non-magically is probably not going to be very popular with his players, but I suspect that is also true of a DM constantly improv'ing poison, paralysis, incapacitate and so on, on his PCs.
Either way I agree codification helps.