DM is not just referee. He also chooses the opposition and the location, among other things. If you screwed up and seeded the PCs against a top 5 team it turns out they never really had a chance to beat, that's your fault. If "homefield advantage" works out for team Monsters better than you anticipated, that's your fault.
I don't think this line of reasoning applies to all methods of play. Some GMs *are* just referees because they don't create adventures: they use modules and/or some system for adventure development (the random dungeon generator in the 1E DMG, frex) and therefore only arbitrate. If a GM runs just, say, Paizo Adventure Paths or Goodman DCCs, then he is in fact just a referee (if he so chooses).