Well, since this is something I might want to use for D&D, my first thoughts would be that maybe you have to spend 1 AP per die of damage or something? A fireball for instance might require 8 AP, with a successful save making it half that? Or maybe only 1 on a made save? I really don't know. Like I said, it is just a concept I am throwing out and seeing what comes of it.
Sure, of course then you have the whole "martial weapon proficiency is valuable mostly because their dice are bigger than similar simple weapons", etc., and "cantrips are balanced by trading die size for other effects" (vicious mockery suddenly becomes amazing), and all of the abilities and feats that allow you to reroll damage dice or treat certain rolls as other things, etc. So then you address that and all of a sudden you're either rewriting the whole game (though, to be fair, with the number of modifications it seems like you use you've pretty much already done that) or you're back to a pool that looks a lot like HP.
So maybe the way to go is to leave the concept of damage against the pool more or less alone and focus on the way the size of the pool is calculated / break the pool up into sub-pools with different narrative focus, and the ways in which it refreshes.
For example, I like the idea that the "tougher" classes are more durable not so much because of how many times they can get hit in a row before going down but because of how quickly they can bounce back from exertion. So maybe you have the damage pool be based entirely on CON mod, but replace bigger hit dice with a bigger share of your pool that can refresh on a short rest.
On the other hand, maybe the "dodge"-base classes get a pool of points like you're talking about where they can turn hits into misses a certain number of times (based on DEX mod), avoiding that damage completely, but when they're out of dodges they become vulnerable to physical damage until they get a rest to regain dodge points. Maybe uncanny dodge then becomes a quick-refresh dodge pool (refreshing when you roll initiative, instead of on a rest?) instead of halving damage, and evasion becomes an ability to use dodges in circumstances where you normally can't?
Meanwhile the "mental avoidance" idea could be implemented with a mechanic that lets you avoid the first hit from a particular enemy through prediction/feinting/insight, but then each subsequent hit they get a roll vs your passive intelligence/wisdom/charisma score (or use the formula for spell DC), to see if they've figured out your shtick, after which you can no longer use that trick vs that enemy. I probably wouldn't be inclined to have different mechanics for each mental stat, but maybe CHA is opposed by WIS and vice versa, and INT is opposed by INT. Something like that.
