If the DM is not considering the threats he is placing in the PCs path, relative to their level/capability, then the DM is being thoughtless about the game. This is not the same as saying that every thing placed in front of the PCs needs to be something that they can defeat. It simply means that if the DM is responsible for the elements of the game world, then he should give some thought to how they will play, not just the fictional justification for their existence.
And they need not be mutually exclusive. Yes, things should make sense in the fiction. They should also make sense as a game. Sure, it makes sense that the lord would place his most capable knight at the front gate. It also makes sense that the knight has been sent on a quest by the lord, and so is unavailable to guard the gate. The fiction can be anything the DM wants......so whatever the fiction is, is the DM's choice.
If no thought is ever given to challenges, etc, the game can be considered thoughtless on the part of the DM. If thought is given in other areas, thought, then what I am describing is not thoughtless.
So if the DM decides that the super high level knight is guarding the front gate, it's because he wants to deter the PCs from attacking. Which in and of itself is fine. I've absolutely done this in my game.....sometimes, it's interesting to remove one of the most obvious options, or perhaps the option that the PCs most often use....to see what else they come up with.
This still is not true. If I place the super high level knight at the front gate because it's the most sensible thing in the fiction to do, it's not because I want to deter the PCs from attacking the front gate. My desires don't come into that decision other than the desire for things to make sense and the desire for the game to be fun for myself and the other players. That is also not a thoughtless decision. I put a lot of thought into what would make sense and why the knight is there.
It only starts to become a problem when more and more options are thus removed by the DM, not because the PCs fail due to dice rolls, but because the DM decides that they simply cannot work.
This is entirely dependent on what those options are and why the DM is deciding that they cannot work. If you have a jerk that is just removing options and saying no, because of what he wants to happen in the game, that's bad. If you have a thoughtful DM who is saying no to an option because it just flat out can't work, then it's not a problem at all.
If you only ever consider fictional justification....which as we've established can be almost anything you want it to be....then you're not considering the game that's being played.
I think you're looking at this through the lens of your preferred method of running the game. There are a lot of people for whom this type of play is most enjoyable. If I were to run it differently for my players, then I would not be considering the game that is being played. We emjoy a different playstyle.
If multiple paths are being allowed, then this is not something I see as a problem. The DM is considering alternate paths for the PCs to get into the castle.
To be honest, I rarely consider paths into and out of places. I just make the place in the way it seems like it should be made, given the fiction and have at it. Players are a very ingenious lot and will think of ways that work................and ways that don't. They may even figure out a way past Sir Invincible.
The fiction is not the reason that anything is happening. As you say, the DM is responsible for all the fiction. Therefore, whatever happens in the game is very much determined by the DM, not by the fiction.
The fiction isn't deciding anything.
I disagree. I look to the fiction to inform me of why to do something first and foremost. It's only after that fails, since the fiction far from covers all things, that I become the sole determiner for what will happen. For example, if the fiction has 1 or 2 probable ways something might happen, I will pick from those 1 or 2 options. The fiction is entirely the reason why that decision is being made by me. It's a shared determination, not solely mine. However, if there's nothing in the fiction to provide me options, then and only then is what is happening not being determined by the fiction and entirely my decision.
Plenty of other people have run adventures exactly as written. Plenty of other people aren't worried about X in this place, or Y in that. Other people may think just the right amount of information is given here and there. Opinions on this vary, obviously.
And plenty of other people use the Rule of Cool or anime style games. I'm not saying that my way is the only way or that other ways don't work. I'm saying that the way I run the game and my players like to play it, I cannot use any published adventure as written. They all(for the last 20ish years anyway) fail the to pass the bar.
I think this thread clearly shows that many folks cling too much to what's written, whether it's a published adventure or one of their own design.
Sure, but I think this is more due to inexperience than anything else. I grew out of that and all the DMs that I've seen from their early days on for a long period of time grew out of that. I'm sure some never go past that limitation, but I think most do grow.
You said that your players came up with 10 ideas, and that 3 could have worked. They chose poorly. Now, I don't know how you decide what would or would not work, but again, I'm talking about the cases where the DM blocks a certain path. So if 7 of 10 ideas simply cannot work, I'd not spend a lot of time on them. I'd instead focus on the three that may, or maybe on the three that may and then one or two of the impossible ones just for reference. I wouldn't want to spend 70% of our time on stuff that won't ultimately matter.
There were brainstorming how to solve the problem. What might or might not work. To show two of the ideas and how I made that determination I will give a bit of background.
There was an old woman in a small town of about 400 people. She was fairly mean and had lost her husband a few years earlier. He was what kept her from getting out of hand. In the few years prior to this adventure, she clashed a lot with several townsfolk. Some of the kids would tease her, because of how she acted and one broke one of her windows. Several neighbors got into arguments with her over one thing or another.
Nearby was a decrepit church to Beshaba(The Maiden of Misfortune) which had not been used in at least 100 years. The locals would not tear it down out of fear of Beshaba taking offense. The old woman eventually became angry enough to go into that church, knelt on the floor and prayed for misfortune to strike her tormentors. He prayer was answered, though not as she expected. The floor gave way beneath her and she plummeted into the basement and died upon impact, whereupon Beshaba brought her back as a ghost out for vengeance against those who she felt wronged her.
Every 7 days after she died, her anger built up enough that she manifested at her home and basically sought out one of her tormentors with single minded rage. The players discovered her motivation after talking with the townsfolk. They discovered her corpse in the church. And knowing what the PCs knew about ghosts, tried to decide how best to lay her to rest.
One of their ideas was to have all the townsfolk who had clashed with her go to the church and apologize at her body. I hadn't actually thought of that, but I knew that she rested at her body and was calmer during the 7 day build-up and would listen, and that would work. I'd have simply roleplayed out that scene and it would have worked. No roll involved. Another idea idea was to have the townsfolk at the site that she manifested and apologize as she came out of her hut. Since I knew that she was in a single minded rage at that point, the apologies wouldn't even register to her in that state. That method would fail. The nature of ghosts. They went with the second option and had to fight her off, at which point they gave up on the apology avenue and solved the issue another way.
If you're not talking about paths that you simply would not allow, then I mistook your comment.
I almost never say no they can't attempt something. If they wanted to try and jump a three mile wide chasm that is one mile deep, I'd let the player have his PC make that attempt and roll up a new PC.

I'm not about stopping them from trying things. Things will just sometimes result in an an outright fail.