I had a similar issue with my lore bard for a couple levels. Honestly, I struggled from about 5th through 7th level to properly challenge the characters. I was still throwing a bunch of hobgoblins at them and thinking that would be a fair fight. Um, nope. When I got better at selecting level-appropriate foes and putting them in tactically challenging environments, a lot of problems went away. Including hypnotic pattern. Also, I just got comfortable with the idea that spellcasters have "I win" buttons that they can pres a couple times per adventuring day. Win some, lose some. There are always more monsters...