For what it's worth, the 3.5 Main FAQ takes the opposite approach - resistance followed by vulnerability.
If a monster has resistance and vulnerability to the
same kind of damage (such as fire), which effect is applied
first? And when does the saving throw come in?
Always roll a saving throw before applying any effects that
would increase or reduce the damage dealt. For example, if a
frost giant is struck by a fireball that would deal 35 points of
damage, it would roll its Reflex save, then apply its
vulnerability to fire after determining how much damage the
fireball would normally deal. If the save failed, the frost giant
would take 52 points of damage: 35 + one-half of 35 (17.5,
rounded down to 17). A successful save would mean the frost
giant suffered only 25 points of damage: one-half of 35
rounded down (17), plus one-half of 17 rounded down (8).
If the creature has both resistance and vulnerability to the
same kind of damage, apply the resistance (which reduces the
damage dealt by the effect) before applying the vulnerability
(which increases the damage taken by the creature). For
example, imagine our frost giant from the previous paragraph
wore a ring of minor fire resistance (granting resistance to fire
10). If the save failed, the frost giant would take 37 points of
fire damage: 35 (fireball) – 10 (resistance to fire 10) = 25, plus
one-half of 25 (12.5, rounded down to 12). If the save
succeeded, the frost giant would take only 10 points of damage:
17 (half damage from the fireball, rounded down) – 10
(resistance to fire 10) = 7, plus one-half of 7 (3.5, rounded
down to 3).
As a general guideline, whenever the rules don’t stipulate
an order of operations for special effects (such as spells or
special abilities), you should apply them in the order that’s
most beneficial to the creature. In the case of damage, this
typically means applying any damage-reducing effects first,
before applying any effects that would increase damage.
-Hyp.
