enerally I find compelling is something that drives what the characters want, or somehow intersects neatly with thier alignments goals and playstyle. When the players find it not compelling it's usually because they simply don't like the plot details of what you've thrown at them. Or they react in unexpected ways. I have a friend who loves rogues and thieves and the whole underworld thing. He's a great DM but he often hijacks his own games by having a thieves guild or assassains guild do some over the top thing that pisses off the hero's and then his cool flavor moment becomes the hunt down and kill all rogues and assasains in the city. Because that's what hero's do right? Even DM's get lost in thier perspective sometimes and forget to consider what the individual players and character's value.This post has been prompted by a conversation with some friends, and reflection on my Torchbearer game yesterday.
To me, it seems at the core of mainstream RPGing is the player characters finding themselves in a situation, that is, some imagined circumstance where they have to make choices or take actions to overcome a problem or avoid a threat or similar. The PCs are controlled, as "game pieces", by the non-GM participants. Typically, the situation is established and adjudicated by the GM participant. PCs-in-situations is where those two roles - player and GM - meet, and interact, to create the distinctive experience of playing a RPG.
Not all time at the table is like this - eg if its established that a PC is at the market, and if the rules of the game allow buying gear to be resolved simply by changing the tally in a "money" column while writing new stuff under an "equipment list", then we've got play happening in some sense, but no situation. Still, I think player-characters-in-situation is the core.
What makes play engaging? The situations have to be reasonably compelling; and the resolution of them has to be reasonably exciting. Sitting around the table with your friends can still be fun even if the situations are boring and their resolution fairly pedestrian (I would put some of the combats in the original Giants modules in this category), as even a boring situation resolved in a pedestrian fashion can provoke enjoyable conversation, surprise and joy in lucky (or unlucky) rolls, etc. But when I talk about engaging play I'm thinking about the play of the game itself, and not just the social context that it occurs within.
One approach to compelling situation puts the emphasis on "who has the right tools for the job?" In my Torchbearer session, the burly Dwarf was the one who lifted up the stone lid of the sarcophagus, while the Elven Dreamwalker was the one who read the runes and interpreted the mysterious sigils. This is an important feature of traditional D&D play, and other RPGing that is inspired by D&D (like Torchbearer).
But I don't think that is the only, or the most reliable, approach. First, there's no guarantee that resolution will be exciting, unless the excitement simply consists in getting to use your tools. Which might be true the first couple of times, but probably isn't going to be sustainable.
Second, most of the time, in most situations, probably more than one - and perhaps every - PC has a tool that will do the job, especially if the players are at liberty to decide what "the job" is - that is, to decide what to make of the situation and what they want out of it. Before I got my group playing Torchbearer, one of our main games was Prince Valiant, and all PCs in that game are knights by default, and two of our three knight PCs had near-identical skill and equipment lists. Another main game was Classic Traveller, and while Classic Traveller PCs tend to be pretty varied in their skill lists and equipment lists, "who has the right tool for the job" wasn't the main way that situations were compelling, because most of the time most PCs had something they could do, be that talking or shooting or radioing for backup or whatever else.
Probably the most reliable way to make situations compelling is to make them clearly speak to something the players have shown their PCs care about. This is also more likely to support exciting resolution, because whatever it is that happens, some PC's situation will be changed in a way that matters. Something will be gained or lost, a friend made or betrayed, a goal advanced or set back, etc.
Particular techniques for doing this - eg the role of prep, the technical procedures for adjudication and resolution, etc - will vary across RPG systems. Nevertheless, and despite that variety, I think this is a reliable approach to engaging play. And it also makes a number of worries that people talk about - eg niche protection, spotlight balancing and the like - largely evaporate. Framing a situation in which the PCs stumble into an archery contest that might give the Robin Hood PC a chance to strut their stuff remains fine, but not the key device for keeping play engaging; and if the real action at the archery contest turns out to be a different PC establishing an alliance with Robin Hood's nemesis, that's fine too!