Obviously in a 3.5 game an orge mage is too powerful for 4th level characters. I would suggest you have the ogre mage leave the scene for a few levels and return later for a revenge type situation against the party.
Give the characters some magic items from the ogre horde to replace the ones...