I greatly enjoy playing monstrous games, so this is right up my alley.
There are generally three different underlying world concepts for playing a monstrous campaign:
1. The monster races are good and the pretty races are evil. This creates a game that is very similar to normal D&D play, only with racial alignments shifted.
2. The monsters are evil, and they do evil things to achieve evil ends.
3. The monsters aren't really evil, it's just that their morality is very different from our own.
As an example of #3, a monster who can only breed by turning the children of other races into its own children would likely not consider taking and turning someone else's child an evil act. They may even consider that the original parents didn't deserve the child because they didn't protect it well enough.
Regardless of which underlying world concept that you choose, you have to choose an adventure mode. The OP has already mentioned what those are.
Firstly: Playing opposite the players, creating lairs, setting traps and generally luring unwitting adventurers to their doom in order to increase your treasure hoard. Think DS9 v. Voyager, instead of going to the challenge, the challenge comes to you.
Secondly: Destroy all humans. Now this may sound a lot like a chaotic evil game, and you'd be right, but on that note I'm going to posit that chaotic evil games are actually monstrous games due to the inherently insane, violent and potentially NC-17 content of such a game.
Regarding the first adventure mode, there is a bit of a complication. Unless the PCs have a way of monitoring what's going on in the dungeon, you are probably dealing with a situation where the PC monster party is split up and waiting for the adventurers, or where the PC monster party repeatedly fights and retreats from the invading adventurers.
I've been tossing around a monster campaign for several years now but I have been attempting to stay away from the "destroy all humans" scenario, but realistically when expanding groups of humanoids of various types seek to rid the land of anything that might challenge their rule, monsters are target #1. It's not an unreasonable stance to take for creatures facing hatred and extermination. However, due to the potentially NC-17 content of such a game, I really just don't want to go there.
So I'm trying to figure out a way to do something distinctly monstrous whose end-game is perhaps to destroy all humans but in a more humorous and less psychopathic manner. I like the idea of luring adventurers into lairs designed by the party, I want to keep some of the classic "adventure" in there but I am trying to really get a good grasp on how I can make "destroy all humans" less...murderous.
Making "destroy all humans" less murderous is a tall order; I mean, it is genocide.
Maybe the monsters could use the dead bodies as part of a ritual to create living monsters; at least that balances the death with life.
Alternately, the monsters could attack with the intent to K.O. the humans, or they could have special abilities that charm humans or render them unconscious; once tied up, the humans could be hauled back and converted into monsters. You could even modify monsters a bit to allow this. I created a medusa demon race for the Buffy the Vampire Slayer RPG that could remove its own petrification effect; they eat people, but they turn people to stone as a food preservation technique as much as they do as a combat technique. So, a medusa could turn a human to stone, haul the statue back, de-petrify the victim, and then the monsters could turn the human into one of them.
Magical poisons/diseases that convert humans into monsters would also work. The monsters would then attempt to poison as many victims as possible, and then it would have to retreat with its life so the poison could do its job.
There's also the face-hugger solution. it's deadly, but it creates life from death.
Here's some thoughts:
-Monsters are facing hate and extermination, so why not make more monsters? Breeding may take a long time, but humans are certainly plentiful and malleable too magic. Perhaps the party starts off with some sort of "monster elixir" and the purpose of luring humans to their dungeons is to get the adventurers to find the potion and take it home. This has some limitations: what's the point in making a dangerous dungeon if you want the adventurers to find the magic potion? Further, how do you distribute it widely among populated areas?
I feel there are a couple different things to explore with regard to the monster potion.
First: Does the target retain its non-monstrous self? If the people who are turned into monsters remain mentally human (or whatever race they were), then letting the potion fall into the hands of hardened adventurers is a bad idea (you're just giving them incentive to hunt you down and more power to hurt you with if they find you).
Second: If the target does not retain its non-monstrous self, then setting up dangerous dungeons could be a sort of test so that only the strongest adventurers are turned to your side. If the potion is especially costly to make, this would make sense. Given the choice between a 10th level adventurer and a 1st level adventurer, you'd definitely want the 10th level adventurer converted to your side.
Third: If the potion can be made in such quantities that it can be disseminated among a population and the targets do not retain their non-monstrous selves, then monsters would definitely seek to do this. However, they might wish to do so across multiple settlements, and in a coordinated manner. This would require building dungeons beneath the city, probably co-opting part of any existing sewer systems as well, and waiting until all the dungeon-building "cells" are ready to go. Doing this in a non-coordinated manner is going to result in provoking people into raiding the newly-turned settlement and coming after the creatures responsible for turning them all into monsters. It also opens up a possibility for "cure" research.
Fourth: This isn't really any less disturbing than "kill all humans." It's less murderous, sure. However, the body-horror element of the change is likely to be at least equally disturbing as a monster attempting to slay you outright.
-Not all races are equally inclined to massacre monsters. Perhaps only a couple races or civilizations in the world hate monsters. This would allow me to present a little more grey area in things as monster-hate could be a product of the specific circumstances of the rise of these kingdoms and we can avoid baby-killing monster players because humans aren't naturally inclined to hate monsters, they're raised that way. This however means a lot of moralizing at the table and I know not everyone is interested in such things being a major game element.
I don't really mind moralizing, but you're right that it's a table preference. You know your players, so only you can say if this makes sense for you.
The upside of this approach though is that you can have monsters and other humans competing for the alliance of the civilizations and organizations that do not automatically hate monsters, especially if those civilizations are strategically located. Relating specifically to 5e, this would be an opportunity to use the renown rules in the DMG.
-Subversion: The characters are respectively members of low-population monster types (and arguably more powerful ones), this would resemble most closely a normal campaign in reverse, instead of raiding monster lairs and caves, you're raiding castles and military bases. This would play well with the fact that the players would be higher CR monsters since castles and military bases would be more heavily fortified than your average goblin-infested ancient ruin.
That sounds a lot like the normal game where the PCs are monsters (albeit, high CR monsters), that you appeared to wish to avoid. Although, it does sound like fun.
As someone who really enjoys playing monsters, I think I would really like playing a "turn others monstrous" campaign.