The only correction I'd make is that you wouldn't get a penalty to throw a dagger in your off hand, if you have the Two Weapon Fighting feat, if you're NOT USING the axe in your primary hand at the time.
This, however, is ultimately down to a DM call, I think. And honestly, how often will the...