I would just ask myself, what would be more fun? or how can I make this interesting? Forget about whether it works or not, I mean, If the players want to do it, and its somewhat feasible, I say do it but add some excitement.
I agree. Except, there's precedent to be considered. How you handle this now will influence how they approach the next time they have to deal with a fortification. And how you deal with this now ought to be consistent with the world - if the players can do this, then so can anyone else of similar intelligence and resources, right? So, why isn't this a common tactic?
We're talking E6, right? I don't know the details of feat availability in E6, but... Fly is a 3rd level spell, and Craft Wand is available at 5th level, no? So, wands of Fly are possible at 6th level, and 50 pound rocks are well within the lift capacity of the spell. Ergo, if 50 lb rocks are enough, some warlord will spend a chunk of change on a couple of these wands, and bombardment will commence with a *hundred* flyers, each able to dump several rocks before the spell expires! Why doesn't this happen? If it does happen, have you worked that into your world-building?
My answer would be that it actually takes 100 - 200 pound rocks to really bust up a fortified stone building, which is getting to be outside the effective limit of the spell. The warlord now needs to send 3-4 people up per rock, and that starts getting prohibitive as a general tactic.