As I said, it seems like the best way to adjudicate the situation would be to require the stat-boosting item be there whenever the character is interacting with his followers. So if he takes the cloak off to bathe, it's not a problem - he doesn't have to enforce loyalty while he's bathing, anyway. But if he goes out to address the troops in his bathrobe, some of the followers who weren't quite so keen on him to begin with will say "screw this" and wander off.
