Spells requiring line of sight wouldn't work. For Witch Bolt, you would follow the regular invisiblity/stealth rules. This was from a post by Dausuul in another thread, I think it sounds about right:
The 5E stealth rules are a hot mess. Here's how I read them in this instance:
1. When the wizard casts invisibility, everyone knows where he is, but it's tough to attack him effectively since you can't see exactly what you're doing. The PC knows where he is (no check), but has disadvantage to hit him.
2. If the wizard moves without hiding, he's detectable due to the sounds and visual disturbances of his passage. The PC still knows where he is.
3. If the wizard takes an action to hide, he rolls Stealth against the PC's passive Perception. If he makes the Stealth check, the PC no longer knows where he is, and he can move at half speed. If he fails the Stealth check, the PC still knows where he is, even if he moves.
4. If the wizard makes his Stealth check, the PC can take an action to try and find him again. The PC makes a Perception check against the result of the wizard's Stealth check. If the PC succeeds, the wizard is no longer hidden from that PC; the PC knows where he is. (The wizard can try to hide again as described in #3.)
5. No matter what, the PC has disadvantage on all attack rolls against the wizard, due to the difficulty of attacking something you can't see.
6. If the PC doesn't know where the wizard is, the PC can pick a location and swing at it blindly. If the wizard is in that location, the PC attacks with disadvantage. If the wizard is not in that location, obviously, the PC is just chopping air.