See, this is where definitions start getting tricky. If you have a timeline for Keep on the Borderlands, and the players not engaging in that timeline results in strong, negative consequences for the characters, there's not so much difference with an outright railroad.
Do X or get beaten with the punishment stick, or Just Do X amounts to largely the same thing.
Granted it's all about scale. If ignoring the Mad Hermit means that some NPC's die and this makes for some difficulty for the PC's, then fine and dandy. OTOH, if ignoring the Mad Hermit means he completes his ritual and Cthulu shows up, well, that's a lot closer to a railroad.
And the line between those two points is very blurry.
I'm really coming to the opinion that what people call story or plot based games and what people call sandboxes are nowhere near as far apart as they might appear at first blush.
Anyway, sorry for the digression.
----------
To the OP. There isn't a whole lot to add here. Talk to the DM in an open and honest manner. Tell him that you're feeling hemmed in by the campaign and there are some other elements you'd like to have to oportunity to pursue. And, actually HAVE other elements to pursue.
