Ok, just to clarify things a bit.
Why the Madness (edit to add - AND SANITY) Mechanics didn't Work For Me.
Context: I was not running a Cosmic Horror game. It was a Primeval Thule game, so there were pulp horror elements but, it wasn't the main thrust of the game. Think Conan, not Dunwich Horror.
I introduced the mechanics to the players. The players immediately realized that the mechanics were very bolted on and did not interact with any other part of the system. The mechanics are basically, "See something really scary, make your Sanity check. Fail enough of those, you get a temporary insanity. Fail enough times after that, and you go permanently insane".
The players and I realized that there was absolutely no context with these mechanics. 1st level or 15th, you had the same chances of failure (more or less). Didn't matter your class, background, level or anything else. Everyone had the same (again, more or less, certainly within about 10-15% either way) chances of failure. So, sure, the first time you meet that horrible tentacled thing, you make your saving throw. The third time you meet yet another tentacled thing, you are still making the same saving throw. By 15th level, you've literally met dozens of tentacled things, but, you make the same saving throw at the same odds of failure.
There was no way for the players to mitigate any of this. You can't get "better" at these checks. Which makes it stand out very starkly from the rest of the game. A 15th level barbarian who has faced dragons without flinching suddenly goes insane from seeing a gibbering mouther, even though his grandmother is more scary.
The players, immediately realizing they had zero input into the system, became totally passive/agressive about it. "Oh noes, another tentacled horror. Whatever shall I so. Oh, look, I failed my saving throw. Oh noes, I'm insane now. Whoopee!"
There's nothing for the players at all in the system. It's just a thing that sits there, like a lump. It doesn't impact any role play (having a lower or higher Sanity score doesn't mean anything, other than it's slightly easier or harder to make a saving throw). It only interacts with this single element - seeing something scary/experiencing something scary. There's no integration. There's not even the slightest attempt to explain what anything actually means. If I have a 15 strength and you have an 8 strength, we can roleplay that. Anyone can roleplay that. But, what does a 15 sanity score mean? What does an 8 mean? It's just numbers on the sheet.
So, yes, that's why I say the mechanics are a failure. They are bolted on lump of meaninglessness that just serve to annoy the players and add nothing to the table.