I don't, and the unlimited casting piece is the benefit to counteract all the rather obvious drawbacks this comes with. Further, were it a 5e environment I'd also relax or even eliminate concentration mechanics for this class other than for maintaining complex illusions.
What it means is that one day you might be the most useful caster ever and the next day you're doing next to nothing, simply based on the spells you happened to pull. If your two 2nd-level spells today happen to be Invisibility and Levitate then sure, you can make the whole party invisible and get them all floating in the air but you're not going to do a shred of combat damage to anyone. Tomorrow your spells at 2nd might be Magic Mouth and Stinking Cloud - useful sometimes but not always. And so on.
Not nearly enough benefit for the drawbacks, IMO.