I like the comparison to talking - have it be a free action with logical limits, as determined by the DM. A paladin can raise a hand from his greatsword to cast a spell and then put it back in the same round. But you should have some limit to it before it becomes a larger (and pointless) action. Twice is okay with me, but three times could have been avoided. Changing grip with every attack in a full attack action obviously isn't permissible to me, but maybe some DM's would accept that. Just like different DM's will have different limits to how much you can say (and I've had strict DM's with that - only x words, no more)as a free action, different DM's will have different limits about changing grip as a free action.