Blackwarder
Adventurer
But it won't be common damage, I figured it will be more the purview of traps and nature mischief
only way for monsters to dish ability damage should be rare and require a crit IMO.
Warder

Warder
You forgot the feats that requires stat mins and turns off if your stat is reduced below the min, and IIRC PrCs that turns off if you don't meet your feat requirements. In 3e, there's a lot of cascading.
As for dealing out penalties to attacks, ACs, skill checks etc. why not just call them penalties to whatever instead of ability damage if you're not going to actually impact the stat and everything that the stat influences.