This is a perfect example of 'roleplaying' restrictions used to balance 'mechanics' benefits.
And look at what (almost) everyone is saying "Work out a way around it with your DM." (Or some variation.)
Not to be harsh, but, you picked a class that gets an inherent stat boost every leve in addition to the one every four levels, gets very good SR, etc. The trade off NO USING MAGIC. Period. Take your lumps, go through the portal willingly, because it serves a greater good (picture eating a worm to save a life), and be somewhat weaker for a week. Compare it to being stat-drained and having a long way back to a cleric able to Restore you. You are weaker for the first part of the trip.
It's part of the character, roleplay it.