Just say the minotaur is growing up while he's adventuring. He's not getting bigger because he's killing things, at least not from a narrative perspective. The minotaur is killing things while he's getting bigger. At the same time, the fighter is getting more skilled, and the wizard is learning more spells.
If the minotaur just sat at home in his maze and ate pizza all day, he'd still get bigger, but then he wouldn't be a PC, so no one would care if he was earning xp or not.
Basically, you have to accept that there is a certain amount of coincidence between the minotaur's growth, and his gaining xp. Except from the rules perspective, these are totally seperate things. They just happened to be aligned to help make a balanced game.