This is *probably* doable. But why would you?
Instead of 20, 14, 11, 10, 10, 8...
19, 16, 11, 10, 10, 8 - your primary attacks will be as good for half your character's existence, you gain an AC, and your swordmage attacks are now viable.
20s are overrated.