i mean...if you think that's a problem, you could just say you count as already having cast a spell on your first turn if you use the feature.
well, sage advice is about o5e. i wouldn't assume any of them apply to a5e unless, say, morrus (or your dm in the case of your own specific game...