I vote you call his bluff, leave your horse outside a dungeon and see what he does. I mean what's he going to do to you? And if he does somehow mess you up, next time you attack roll all the dice you have for damage and say something along the lines of "well since you don't seem to have to...