The wizard said that I get to go to my fantasy world. In my fantasy world, and I think it's the same with many other fantasy worlds out there, PCs never get challenged by something they can't defeat or run away from.
If the wizard gets me to my fantasy world, as a 1st level hero, and then a dragon appears and eats me - well, I can't ask my money back, but he did cheat me, 'cause that's not my heroic, high-fantasy world, but is instead a realistic and grim'n'gritty world.
I think most everyone will agree that there is a lot of difference.
The deal was for my fantasy world. No buts, no ifs, no excepts.
That's one of the things I would ask the wizard before going. If he tells me, "Well... actually, it would be a place looking the same as what you're thinking, save that the presence of danger and the lethality of combat is a lot more realistic" - then I'd have to think about it for a while. I'd probably go anyway, but I would be very careful to place me in a situation where I can grow very powerful while being sufficiently sheltered.