This is a difference yes, but it actually works against the Wizard.
If the Wizard refuses to take LTH, SWD or Fly then he not cast them at all, ever, so he can't do his "primary job", as I chose to define it, and is not able to do it in the future either, even if the story progresses such that...