You might as well just stab yourself, then. Taking your attention off of someone trying to stab you is a good way to die.
Well depending on what the caster is casting you might end up stabbing yourself.
You're better off defending yourself as you move to the caster. Just attacking the caster from 30 feet away makes no sense in any sort of realistic or semi-realistic combat.
In realistic combat there is no such thing as a caster, good thing we are not talking about realistic combat.
If you are into nonsense combats, then sure. Go for it.
All combat in D&D is nonsense.