Your wizard retreats past the party, triggering your readied action. His movement stops in an occupied square (probably the clerics or something) and gets shunted to the nearest open square, which happens to be the one you just left. I'd rule this as legal.
See, this is why you take tumble as...