I'm not the only one to bring up meta-gaming as an RPG crime...
I admit that at first glance, most people think I have a somewhat heterodox view of metagaming. I know it is popular to blame metagaming for everything, just as once upon a time it was popular to attribute to 'realism' the solution to everything. But I believe both are based on fundamental misunderstandings.
and you didn't object when I mentioned it originally.
I didn't object originally because your first statement was not as absolute as your later statements. I don't object to the idea that metagaming can be a crime. I object to the idea that it is universally objectively bad and further any good role-player should hate metagaming.
Before we get into why, let me note our agreement:
This is one definition that I know is well-accepted. When role-playing, meta-gaming is the act of having a character act on information that they don't (or can't) know.
Good. This definition accords well with the one I offered, so we are on the same page. Now, I'm going to hold you to that statement.
First proposition:
1) All players and all DMs always and with every action metagame all the time with every action that they take within an RPG. There is no action that you can take in an RPG that is not metagaming.
This follows directly from the definition that you just gave. Every player at all times knows things that his character does not know, including the fact that it is a game. It is impossible to act in a way that you disregard the information you as a player know that the character does not know. Even if you consciously note to yourself the things you know that you know that your character doesn't know, and try to disregard those things completely, you are still then acting on the knowledge of the things you know that your player doesn't know. Because you know these things, you can never unknow them. Your decision making process is forever tainted by this knowledge, and can never be pure and free from metagame knowledge. Although you can try very hard to be introspective and ask yourself, "What would my character do in this situation?", even then you are asking yourself that and engaging in introspection, and you are not yourself in the game world and cannot possibly answer the question except as influenced by all the other stuff that you know.
So when we talk about metagaming being a crime, we are perforce, speaking of a very narrow range of metagaming activity, since the whole of playing an RPG where you make choice for your character is metagaming activity. It can't possibly be the case that metagaming is bad, unless we also agree that playing an RPG is bad.
Which brings us to our second postulate:
2) Metagaming is actually good and necessary to playing an RPG.
Let's suppose that we were obsessed with the idea of playing a pure simulation of an imagined world, free from any metagaming. We could begin to create procedures of play that would ensure that we didn't act based on metagame knowledge. For example, faced with a situation where we know the character is facing a troll, but we also realize that deliberately not using fire in this situation is also metagaming, since absence our knowledge that this is a troll and is vulnerable to fire, either we might have stumbled on the idea of using fire anyway or the character might have heard in his long life within the setting stories of trolls and their vulnerability to fire (and many other traits). So we in this situation decide to flip a coin, in order that we can guarantee our decision is free from bias and we cannot be accused of the 'evils' of metagaming.
However, by the first proposition all decision points are identical to the decision point of whether to hit the troll with the torch. Everything is tainted by our out of character knowledge, and so every decision must be resolved by a coin flip of some sort. The result of our obsession is that the game will cease to be an RPG. Indeed, it will cease to be a game at all. What we will instead have created is a model. We can set it running, but once it is running, to maintain the purity of the simulation we - being outside the model and so influenced by our knowledge that the characters in the model couldn't have - can't in fact touch it. We won't make decisions at all.
3) When the DM creates a setting or scenario, he acts on knowledge no character within the scenario or setting could have - including the fact that it is a game. However, this is not (in and of itself) metagaming.
Why? Because you said yourself that metagaming involves acting on behalf of a character and a DM creating a castle or a city or a jungle isn't acting on behalf of a character. No NPCs wishes are consulted in that creative process. While he may at times after things have begun ask, "What might this NPC do in this situation based on what the NPC knows?", there is no way that he can ask questions like, "What might this setting be like based only on the information within the setting?" The setting can't be defined in terms of itself - it's always defined in external terms, by making reference to the one setting we have, reality. Even when it differs to reality, that's still referencing reality as a baseline.
Now, for the record, I consider your definition slightly too narrow. There are things you can do in a setting that are metagaming, and I mentioned a few (such as adjusting the nature of your setting to counter the abilities of the players) but merely establishing a setting is not metagaming. And, even if it was, it wouldn't necessarily be wrong.
It can include knowing that trolls are weak to fire (if the character wouldn't otherwise know that), and it can include making an assumption that a puzzle is solvable because the GM presented it to you.
So let's get to this now. I put it to you that you can't unknow that the character knows that trolls are weak to fire, and you can't but metagame about that. You can't unknow that and ever again actually simulate the not knowing that. At best you can do is try to pretend for a while you don't know that, and make some determination of how quickly your character might figure that out, but that entire decision making process will always be based on your out of game knowledge.
Secondly, the assumption that a puzzle is solvable based on the fact your DM presented it to you is metagaming, but it's a really lousy assumption. This is an example not only of metagaming, but metagaming based on knowledge that is likely wrong. The fact that a player is acting on out of game knowledge that is actually wrong is really the only reason I tell my players not to metagame. It's not that I think that there is any wrongness in a player metagaming, it's just that I think players are usually pretty terrible at doing it and further that if you get in the habit of relying on out of game knowledge, you'll tend to ignore end game clues. I don't tell my players not to metagame because I think it's a crime to do so - it's all metagaming. I tell them that because I think that players use it as a weak crutch.
In point of fact, I believe all metagaming is the DMs fault and reflects badly on the DM. A DM should never be upset with his players for metagaming, and never tell his players, "Don't do that. That's metagaming." It took me maybe 15 years to realize that, but telling your players not to metagame is proof of your own bad DMing and your own reliance on crutches. If the players are metagaming and you are upset about it, one or more of the following is true:
a) You became invested in some sort of 'gotcha' encounter, which is adversarial DMing.
b) You are trying to play the player's character, which is a violation of the baseline social contract of an RPG.
c) You did a bad job in your role of secret keeper, and let slip a secret at the wrong time, and you are doing an equally bad job of improvising. This suggests you need to work on your skills, you were bragging to the players, you were impatient, you didn't study enough before the session, or you just otherwise made a noob mistake. Now that the secret is out of the bag, it's ridiculous to act like the players can ignore the secret.
So the question is, why did this specific NPC make all of the necessary decisions that led them to become the preeminent villain of the setting?
Why not?
If the answer is "to provide a compelling foe for the PC" then that's meta-gaming.
Sure, although as I noted above, not by your definition. But if that is metagaming, so is putting 5 Bugbears in the room instead of 50 or 500, or not chasing the players down with an ancient red dragon, or deciding that the country you will start on is loosely based on medieval Scotland, or deciding to play a pirate game because you know that your players are beer and pretzels casual gamers that like to blow steam, not super heavy thespian types wanting a deep story line. The entire setting is crafted by the DM based on knowledge he has from outside of the setting. He's only requirement is that he make the setting rich, believable, and fun. And there is nothing inherently unfun or unbelievable about the protagonist, uber-talented scion of destiny that he is, has an older brother he is estranged from that is just as talented as he is, but who has chosen a different path in life.
While it's possible to have their motivations be entirely in-character, and for that to be established before the GM ever finds out who the PC will be, it is nevertheless highly improbable.
So what? An average RPG session involves dozens of highly improbable things before breakfast. As I established earlier, we don't pick a random character from the population of the world for the player to play. We pick someone to whom highly improbable and unusual things are not only likely to occur, they are absolutely guaranteed to occur. If we eschewed improbability, we'd be back to picking a character solely by dice rolls out of a model, and we'd likely end up with a subsistence farmer living a hard scrabble mean existence and never doing anything more glorious than managing to survive and maybe care for a family. And while that's a valuable thing and we might could tell a great story about it, that's not the sort of improbable character living an improbable life that we choose to focus on.
There are millions of people that could potentially rise to power in opposition of the Big Bad, and the likelihood of them being related is insignificant...
Sure. But there are also millions of people that the player could play other than the younger idealistic brother of the Big Bad, and the vast majority of people we could have chosen for the player to play are 'insignificant'. Fortunately, we aren't required to choose a random insignificant being to be central to the casting of our story.
(Note that you can raise the probability of such events significantly, if you let the players play nobility. If you're playing the Prince of England, then the likelihood that the mysterious villain is somehow related to you is actually pretty good.)
I don't really see how this follows. Regardless of whether we are playing the 'Prince of England', we aren't playing a nobody unless the player specifically tells me that what he wants for the character is to be a nobody that pulls himself up by his bootstraps.
At least according to the writers I follow on Quora, they usually develop the outline of a story before they figure out the specifics.
I suspect that is a formula for extreme writer's block.
That's not the case, for characters in an RPG. Their bottom line is not written yet.
Heck, it's not even the case for the characters in a novel. Plenty of authors will tell you they had no idea what the character would do until they happened. Sometimes, they are just as surprised as the reader by the turn of events.
The choices they make - the choices that we imagine them to make, while we pretend to be them - actually matter, because their world is actually controlled by internal causality rather than plot contrivance.
Sure. But that has nothing to do with metagaming. And again I insist, that if you write a novel which is obviously controlled by plot contrivance rather than internal causality, it's going to be a bad novel. The trick is to make your plot contrivances, whether in a novel or an RPG, seem not to be contrived. However, they are always contrived to one degree or another. If the appearance of your evil brother seems too contrived, that's a problem with the GMs artistry. It's not however metagaming.