I don't see why not, I mean your concentrating on it, so you are not recasting it just doing an attack. Granted it's with a spell but how I see it is the spell buffs your unarmed attack to do X and you just poke them.
Barbarian seems to clash overall with what you are doing IMO, It will stop you from casting spells which is what the rest of your levels are going into. I think Bard (Lore) and maybe Cleric (Life) could make a better support build with all the buffs and support the classes provide.
Well I plan on making a Bladesinger Gish for a new campaign I am running tomorrow. Should I consider going Favored Soul and Bladesinger? or should I focus purely on Bladesinger?
Also should I multiclass at all for Bladesinger and if so what class?