My guess is that your party will be fine without a controller.
Because this is the internet, and I feel the need to throw a little bit more out there, I'll add this.
It seems to me, and my 4E experience is largely limited to LFR, that the controller really comes into his own in paragon tier, maybe surpassing in importance / replacing the defender at higher levels.