I´m not sure this counts, but my first DM had a tendency to play "by the RAW" in incredibly annoying ways...
Like using the random encounter tables, and not fudging the rolls..
"But Gary, we are 1st level! We can´t fight 12 trolls yet!"
"Too bad, thats what the dice say you encounter!"
Yeah, I gamed with a guy like that. We spent 4 gaming sessions being killed by Random encounters. By the end, only my first character was left--cleric / thief halfing who only survived by his stealth skills. Finally, in a miracle I still can't believe, we managed to kill a hill giant with most of the Pcs at first level!!!! (1E game)
In general, I've had lots of bad gaming experiences but most of them didn't actually relate to homebrew rules/settings, per se.
However....
My friend Lee Garvin cooked up what started as a really cool homebrew setting--our PCs were basically the Suicide Squad--we were doing missions for the government that had imprisoned us to reduce our prison time.
I lost my character shortly after the Party went renegade and escaped our masters' control; I got killed by a trap I missed (Halfling Rogue)
My new character was a half-elven female cleric, created to be the half-sister of another PC (who was an Elven Cleric). Cathy and I put a bunch of work into building up the background and stuff. So I finally get introduced just in time for when our Wild Mage decides to go out in the woods and summon a familiar while the Dwarven Paladin is busy getting his full plate repaired. (This = 2E game)
Most of the party goes with him in case wandering monsters show up. The reason he went out in the woods is that he was afraid a wild surge might set the town on fire or something.
Instead, he gets a wildsurge, and the DM's homebrew wild surge table indicates...gateway to the Abyss, Type VI demon comes out.
Round 1, the Demon TKs me high into the air and I fall to the ground and die. Splat.
Then the Demon slaughters the party like wheat. However, they manage to knock him down to about 10 HP. At this point, Cathy's turned into a tree, most of the party is dead, and only the Halfling Juggler (Bard kit, IIRC), who has only ONE ARM LEFT, is left facing Mr. Demon. The Dwarf Paladin is alive...3 miles away.
What follows is a running battle in which the Juggler keeps popping the demon just enough to prevent him from regenerating using a magical sling and magical slingstones, while the demon keeps trying to kill him and he keeps making saving throws.
Eventually, the Demon chases him within crossbow range of the city. The Dwarf takes his +3 blessed crossbow and blessed bolts, goes up on the city walls and kills the demon with a critical long-range shot.
They become heroes of the city, while wild animals gnaw on our corpses.