For me it's simply the fact that once one creature saves, Instinctive Charm is done for the whole day. It's going to stop working sometime during the day. [Edit: This paragraph isn't correct; the ability only stops working on that one creature. But the rest of the post is just fine.]
By level 5, you should be able to prepare 10 Wizard spells, and when you add in the 2+1 that you know from Divine Soul, you've got an embarrassment of riches compared to most spellcasting characters. Spareing one for Shield shouldn't be a massive burden on a day you predict combat or adventuring.
I could see not taking Shield if your character is expected to be the healer of the group, but otherwise, I think it's just too good. Shield is just overwhelmingly potent, especially because it's unique in that it's a level 1 combat spell that remains just as powerful at level 20 as it is at level 1, meaning that it scales unusually well. It always gives +5 AC for a level 1 slot (meaning it gets cheaper as you gain levels). Since your AC typically scales from other sources, that means Shield scales with level without costing higher spell levels to do so.