And who, when the chance was there, failed to bump the wizard in mid-casting thus ensuring the whole party had to stay put until s/he could memorize up another teleport? The Druid, Fighter, and Barbarian did have a choice in the matter...
And given that the thing that was discovered (the portal) was itself magical, I don't see it as unfair that it took magic to find and 'unlock' it. Had it been a non-magical secret door, that's different...though I notice the party has no Thief/Rogue in its lineup so even there the options would come down to magic or hammers.