Maybe.
But consider something like the Balrog in LotR. Sure the Dwarves dug it up and it slaughtered them, but outside of Moria few had ever even heard of it - including other Dwarves, as Gimli learned to his sorrow. It wasn't an existential threat to anyone, really, other than those it had already killed.
Going into an old mine and killing off some big ol' badass creature (and all its little henches!) that isn't (yet) a major threat to the surface world seems like just the sort of adventure the OP was looking for.