I agree with what other posters have said from an RP/design standpoint but wanted to focus on mechanics for a minute.
Normally the specter variant of the poltergeist is not invisible, so you are creating a custom monster. Being invisible does not necessarily mean they don't know roughly where the monster is, but it does mean they will be at disadvantage for every attack which roughly equates to +5 AC.
Since you are modifying a monster, you can use the guidelines on page 274 of the DMG to get an idea of what DC the monster is after the modification.
Defensive Challenge Rating. Read down the Hit Points column of the Monster Statistics by Challenge Rating table until you find your monster’s hit points. Then look across and note the challenge rating suggested for a monster with those hit points.
Now look at the Armor Class suggested for a monster of that challenge rating. If your monster’s AC is at least two points higher or lower than that number, adjust the challenge rating suggested by its hit points up or down by 1 for every 2 points of difference.
Since your specter is 5 AC higher than expected, you've created a CR 3 monster (according to the guidelines). Considering your group, and the life drain ability, that's an extremely deadly encounter and you are quite possibly looking at a TPK (Total Party Kill).
The first couple of levels are the most dangerous ones for PCs, and while the guidelines in the DMG are imperfect (they're just a guideline after all) I think you've made a very dangerous opponent.
So the question is ... why is it invisible? A Specter is a CR 1, and with it's life drain a challenging opponent for your group.