The first question one should ask when trying to build an economy is "What's nearby?" What sorts of resources are there? You mentioned a mine and a few other things, but what else?
Rivers are important. Do people travel along them? Can they be fished sustainably?
Is there a port nearby? Or somewhere you can establish one? It can take many years for the effects of a good trading port to be felt, but if you had a decent one and upgraded it every so often, that alone could make for a sizable city in the future. If you're human, you might not live to see it though.
Monsters were mentioned, and are a great source of some things. Dragon parts have lots of uses (and you're about at the level where your party could take a small dragon, if you're all smart about it). Consider setting up a hatchery for magical critters, or other rare species that might be captured. Be sure it's well-designed, though; having monsters rampaging everywhere is going to cost you a lot.
Have the druids scout all the nearby lands. Tally up what sorts of herbs and plants grow there; some of them are used as arcane and divine spell components, some of them for poisons and medicines, and some of them might just be Elven delicacies! Not to mention, you might find a grove of darkwood or soarwood or something, possibly deep in the forest where men don't often go.
And most importantly of all, look for adventure opportunities! Maybe there's a hag nearby you could drive off. Maybe orc raiders come over the mountains every season. Perhaps there's a tunnel to the Underdark which needs to be scouted and secured? All great opportunities to loot treasure while also making your realm a safer place.
Don't forget to encourage your druids to help with the crops. Lots of clever spells for farming on the list. Clerics of Pelor (and other deities with similar portfolios) might be encouraged to make a trip out here to help with the farming, too, if you give them free reign to preach and convert - just be careful about which religions you encourage to preach to your people; some of them have agendas, and need to be managed carefully if you don't want to offend them.
And last of all, maybe you should consider a marriage of convenience to a ridiculously wealthy family? In most medieval societies, land-owners are highly respected. A man with land and a keep might be a good catch, or make a potential ally in the future, if a clever marriage were arranged early. It's time to start getting into the politics of the world! Don't be afraid to dive in face-first!
