I'm trying to find where the game defines the goals of play you list:
From RAW, it seems like gaining levels is the mechanic for goals...
Well, if you don't do anything towards the goals, or even take on challenging issues, you don't get XP. Burning down a house is something anyone can do, and takes no special effort - oil, toches, whoomph! Done. Why should they get a reward for that?
Last time I designed a haunted house adventure, the players did just that.
In G1: Steading of the Hill Giant Chieftain, a considerable portion of the 8 pages of original text is devoted to just why the PC's can't successfully burn down the dungeon, and what unpleasant things will happen if they try to do so.
So, in general, my advice is have a plan for what happens if the PC's turn arsonist right from the start.
IF burning down the house doesn't do anything towards the goals, then there's no XP to get, and no XP to "lose." So I agree with that part.
My point was that, if the house is full of dangerous enemies, then burning it down definitely DOES do something towards the goals. It does a lot. Awarding or withholding XP based on the PC's method of achieving the goals turns one of the primary decisions the players get to make into a meta-game decision. Rather than "How should we overcome this challenge?" it becomes "How does the DM want us to overcome this challenge?"
IF burning down the house doesn't do anything towards the goals, then why did the players decide do it? My players are not stupid people. If that really seems like the best course of action, maybe I've miscommunicated somehow?
I agree with [MENTION=97077]iserith[/MENTION] that the players do bear some responsibility for making "fun" decisions instead of purely pragmatic ones. For example, for most PCs, the purely pragmatic decision is to sell most of your starting gear and become a farmer. My group of PCs literally met at a bar (location 8, the Empty Net) and decided to adventure together for purely meta-game reasons.
One thing I'm considering is giving characters a Wisdom (Insight) check to get "hunches" about the consequences of major actions. Like, "maybe you'll miss important clues if you burn the house down." This would really just be an excuse for me to tell them things at a meta-game level, without breaking immersion. I'm hesitant though, because for some people doing that would break immersion even worse.
One thing I'm considering is giving characters a Wisdom (Insight) check to get "hunches" about the consequences of major actions. Like, "maybe you'll miss important clues if you burn the house down." This would really just be an excuse for me to tell them things at a meta-game level, without breaking immersion. I'm hesitant though, because for some people doing that would break immersion even worse.
My point was that, if the house is full of dangerous enemies, then burning it down definitely DOES do something towards the goals.
I seek a world in which every player decision, smart or stupid, leads to ever more interesting decisions...
That's impossible. In fact, it's self-contradictory. By definition, if the player decision - whether smart or stupid - always leads to ever more interesting decisions, then those decisions are not interesting. If regardless of what I choose, I'm going to get an interesting result, then the decision itself is not meaningful. I could roll the dice or flip a coin for every choice. What does it matter?
Fundamentally, an RPG is a cooperative endeavor that requires a certain sort of active participation by all parties. If the players make interesting choices, you can always have interesting consequences. But there are some sorts of choices you can't give interesting consequences and still have choice be meaningful.
...
There is only so much the GM can do to put in what the players are leaving out.
I agree. A lot the responses seem punitive and a bit railroaded. The players should do things a certain way. If they don't, all these other subjective factors kick in: the town responding negatively, all the enemies escaping unharmed to the caves, all the clues being destroyed.