Yeah, CoC is a funny story. So, WAY back in the day I was always smitten with the genre, and immediately picked up the game when it came out. For any who haven't played it, rules/play concept wise it is not super different from any other RPG of the time, the rules just describe concrete attributes of the character strength, etc. and skills they have, and you wander around making checks whenever the GM says you need to resolve some action. State of the Art ca. 1978 RPG design.I think there are two broad approaches that go beyond the limits of "classic" D&D - and both are canvassed in the posts above.
One is a player-side "move" or ability that obliges the GM to establish and share some fiction within parameters that are useful or meaningful for the player given the player's goals for his/her PC. Apocalypse World (eg Read a Charged Situation) and Dungeon World (Discern Realities, Spout Lore) provide canonical examples of this.
These moves don't ignore fictional positioning - both the context which enables their use, and the way the GM establishes responses, have to respect fictional positioning. But if the player makes the check then these moves have the effect of giving the players access, here and now, to highly salient and useful fiction that will enable him/her to drive things forward (via the play of his/her PC).
Having recently re-read some old CoC scenarios (looking for a good murder mystery to run for a kid's birthday party), I'm struck by how weak they are in this respect. All the information is presented with its meaningfulness being GM-sided, and then there are various clumsy devices hinted at to help secure salience to the players - eg the NPC draws the PCs attentions to it or the even worse hopefully the PCs will notice this! It's GM-driven all the way through.
The other approach is the one adopted in @AbdulAlhazred's 4e spin-off HML. It's also prominent in Burning Wheel, and Cortex+ Heroic: a successful check enables the player to directly establish some element of the shared fiction.
Again, this doesn't circumvent fictional positioning. It builds on it.
So we had fun with it, clunky as it is, and then now and then dusted it off when we wanted a shorter campaign (PCs don't tend to survive long) and it tickled our fancy. Then there was a pretty long hiatus. I found myself playing RPGs with a few close friends in the late 00s and we dusted off CoC and came up with a sort of clever idea for a mini-campaign where we would each run part of it (the PCs being reincarnations of themselves in different eras). It was just unplayable. In the intervening 15 years or so, we just moved on. The system was SO CLUNKY and got so much in the way we finally just ditched it entirely and went back to using PACE, which we'd used in a previous mini-campaign I'd run before that (PACE is a diceless system where all the participants exchange tokens to 'buy' narrative control).
I will always have a soft spot for CoC, but it is a great illustration of a game which did not age well. The idea of using the core 'engine' of RQ was one of those decisions that was perfectly in sync with the game design trends of the early 80s, but it just doesn't work. Not without more help. The addition of SAN and tweaking of the way POW works HELPED some. The way the core rules become this deadly meat-grinder when you remove things like armor and add in guns, that works. The rest? As you say, the PCs have to stumble through each adventure hoping they make the right check at the right time, and only GM force keeps the story "on the rails", and each story is VERY MUCH a railroad!
If I were NOW to design a game in that genre, I would design it such that NARRATIVELY the outcome was essentially a foregone conclusion, with maybe the alternatives being "die with your boots on", "become totally corrupted, THEN die", and "just go mad, after leaving a puzzling typed manuscript for your heirs to puzzle over." The story would then simply be coloring in the details and basking in the oeuvre of it. You could have choices of which monstrosity devours you, and how and when, etc. Clues and such wouldn't really be an issue. Deciding which ones to follow up on, and exactly how, would be within the PCs wheelhouse, but their real choices might revolve more around things like "do I let the policeman bite it, or do I cast myself to my fate now and go out with a bit of nobility?" (all the while knowing that the poor slob is still probably going mad even if I saved his butt).
Well, that and you have to really avoid all the icky racist nastiness. I must say I admire the way Charles Stross has handled it in his books. They are a good model.