Well the DMG does say, "As Dungeon Master, you wear several hats: storyteller, rules arbiter, actor..." Fact of the matter is that the DMs job is clearly spelled out in the DMG, and being a referee is clearly pointed out as being one such aspect of that job. If the player likes to quote rules...