One solution to this is to replace the d20 in the attack rolls of monsters with 2d10. Instant swingy buffer.
Sorry, but I don't think this helps swingyness much, because you are still getting a binary result: hit or miss. For the obvious example, imaging a hypothetical monster attack: +8 vs. AC 19; 4d8+12 damage. You need to roll an 11 or better to hit. Rolling 1d20, that's a 50% chance of hitting. Rolling 2d10, that's a 55% chance of hitting. So it's the equivalent of a +1, but the real take-away is that a single die roll stiil means the difference between 0 damage and 30 damage, which is kind of a lot. In fact, the expected value of the attack is 15 damage, but there's no way to actually ever get that result (since the minimum damage here is 16).
What 2d10 does do (or 3d6, another popular alternative to 1d20) is clusters results around the center, effectively giving a bonus on attacks that are lower than 50% and a penalty on attacks that are higher than 50%. Imagine an extreme case, where monsters always Take 10 on attack rolls. Now, the PCs get in a fight, they are either going to get slammed hard on every round, or the monsters will be completely ineffective. To me, that's
extremely swingy: even though the results are predictable round-by-round, when the combat
starts the PCs don't know whether this encounter is going to be the TPK or the cake walk -- there is no middle ground.
To me, a rule that would decrease swingyness is something that would give a less binary result: A smoother
damage curve per attack, rather than a smoother hit curve per attack (which is what 2d10 gets) or a smoother damage curve per hit (which is what using average damage instead of rolling gets).
For example: Monsters all get +5 to hit, but do half damage; if they hit by 10 or more, they do full damage. Thus our hypothetical attack above (+8 vs. AC 19; 4d8+12 damage) no longer has a 50% chance of 0 damage and a 50% chance of 30 damage; it has a 25% chance of 0 damage, a 50% chance of 15 damage, and a 25% chance of 30 damage. The expected value is still 15 damage and is now the most common result, instead of an impossibility. (I'm ignoring crits, which are by their nature swingy.)
I'm not advocating something like this as a house rule (it seems like it would be too much trouble to convert the monsters, especially dealing with attacks that impose conditions+damage, or those that already deal half damage on a miss), just saying that I don't feel using a normal curve (2d10 or 3d6) reduces swingyness any when the check is producing a binary result. Any time the check can be reduced to a % chance of pass/fail, the die-rolling system is just a stand-in for 1d100, regardless of the distribution of the actual dice you are rolling.
-- 77IM