A character (doesn’t really matter if it’s a PC or NPC) wants to frighten a monster, and doesn’t have any spells that can do so or doesn’t want to spend the spell slots. So they declare “I try to frighten the monster by acting all scary.” Since there is not a specific rule for how to resolve this action, the DM falls back on the general action resolution mechanics. Can this action succeed? Assuming the monster isn’t immune to fear or anything and the approach of “acting all scary” is something that could indeed frighten it. Can it fail? Sure, maybe the character acting all scary isn’t enough to scare the monster. Are there meaningful stakes? We’d need more context to determine that, but for the sake of argument let’s assume there are. So, the DM ought to call for an ability check to resolve this action, probably Charisma (Intimidation), and in this thought experiment, that would mean imposing the frightened condition on the monster.
I'll elaborate on this example: the DM could also decide that, no, there's absolutely no way this monster would be afraid of this character's antics. No roll, no intimidation. It's right there in the rules.
Now, the rules don't actually provide any guidance on how to handle the inverse situation. It may not be provable, but it's certainly reasonable....again, given the text on 185....that in the inverse situation the player is the one who adjudicates the effect on their own character. The DM gets to do that for the entire rest of the world, but the Player has authority over their own character. So if they say, "Nope, no chance," there's no chance.
Again, not RAW, but in my opinion (and it should be @Lanefan's!) that seems more logical than saying, "No, the DM gets to both declare NPC actions and adjudicate their outcome, even if it involves PCs."