I think there are some abilities (feat), like warcaster, that help with spell economy, or basically when you can cast a spell with like you weapon and shield or what have you in hand ...
I had issues in one battle where I switched from firing and using a loaded crossbow, to a sword and shield...