Thanks for all the input! It was very helpfully.
Here's a rough look at what I've come up with. Disclaimer: My group is pretty casual, so I'm not too worried about optimization, and I don't realistically think this game will go past Level 7-8, so I didn't bother to map out higher level...