No, you start having 4 uses assuming a 16+ Intelligence score, which I think is reasonable for the way that most people will make their Bladesinger. (Yes, we know that you personally might to do it differently.)
That is three initial uses, and then one back in a short rest. In a day with 4...