Maybe we should just leave twisting words into the worst possible result for genie wishes or even better, never.
They expressed their intent, said nothing about breaking in, walking through walls, any of this other stuff. They said they 'went to the castle' or 'went into the castle', a thing...