Glad I finally got around to reading this thread. I've tried some similar ideas in the past, but not with much success, usually because I eventually found the state-tracking tedious, especially if there are a dozen little rules required to figure out who is engaged and who is not. So I'm watching with interest to see if these problems can be overcome.
Focus fire is usually such a good strategy in RPGs because the single most important resource in combat is almost always the action. In a purely gamist sense one might say that is the point of combat. Furthermore, the importance of the action is often amplified because in most games opponents remain fairly effective until they actually die (and for good reason because death spirals severe enough to change that calculus are often a pain to use and introduce a lot of side effects.)
The approach I eventually developed ensures there is a tension between focus-firing and spreading out attacks at a slightly higher level of abstraction. It performs that task not by modeling engagement/disengagement based on specific actions or circumstances but by modeling the consequences of taking or refraining from actions with respect to other creatures.
It is a momentum system, where success grants a creature attempting anything of significance with a spendable benefit that can be applied to a future attack, defense, or skill use, etc. Creatures that are being ignored by their enemies tend to accumulate momentum (because they don't have to spend it to boost their defenses) or simply pump it back out in the form of more vicious attacks (and a greater chance to earn yet more momentum). This upends the usual power of focus fire because ignored enemies actually become more dangerous (or at least more flexible) the longer they are ignored. And even creatures that are being ignored have to be careful, because if they blow all their momentum on attacks when someone finally does attack them they might be left in an awkward position.
So engaged vs. disengaged is no longer a matter of specific actions or relationships, but a consequence of incentivizing other creatures to spend momentum in order to stay in the fight. The rules to achieve this don't need to be complicated as long as it is clear how every aspect of the game can contribute momentum and how it could spend it. For a game with a well-defined core mechanic that means a few simple rules probably cover 80% of all cases.
In play I've observed this lead to very dramatic changes in tactics as a creature shifts its focus between offense and defense simply by how it spends momentum, or as minor threats become major ones and back again. Combat has a distinct ebb-and-flow as creatures spread their attention around while also trying to finish someone off, and as the freedom to spend momentum more freely gets passed around the battlefield.
Here is an example of how it works for players. In my homebrew game momentum tends to be the major source of spell points for spell casters, even though they have a daily reserve. (Roughly speaking dedicated spellcasters of any power tend to have enough spell points to cast their best spell 4-6 times per day before running out, and a bit less than that at the very beginning.) A wizard that truly wants to go nova in this game must spend spell points from their daily reserve and then expend momentum to improve the spell's attack. That means players usually spend the wizard's daily spell points quite conservatively, since spending them without momentum is good but not amazing, and it also means a wizard without any daily resources isn't necessarily reduced to a crossbow or even his "at-wills". Of course, if said wizard is being attacked and needs to spend momentum bolstering his defenses instead he'll find that the spells he can cast will probably be rather limited. But the balance can always shift back if the creature attacking the wizard is ignoring the party fighter, because it won't take long for that fighter to absolutely mess up a single opponent, giving the wizard a chance to slip away. Thus, engaging a creature is both a way for the attacker to press his advantage or to dull his opponent's advantage before it becomes too great.
Now, this was implemented in a success-based system and how I might do the same in D&D is not something I've really considered. Some critical things in my implementation, like gaining more momentum for better successes, and making gaining more momentum harder for creatures that already have a lot, probably don't interpret directly into a d20 system. Nevertheless, one can start to see some possibilities. For example, a rogue with momentum 4 might simply be able to expend them to add 4d6 to sneak attack. Therefore a regular sneak attack might be good, a high-momentum one devastating, and the PC doesn't have to worry about whether they are engaged or disengaged, they can simply use sneak attack and decide if they want the extra damage or not.
Anyway, as alternate approach to achieving some of the goals of this thread I thought this basic outline may interest some of you. If Crazy Jerome would like to keep this thread more focused on the engagement/disengagment idea (which is worth pursuing) I wouldn't oppose forking the discussion.
