From the PHB:
There's no mention of it not transforming because it is in a confined space, and it happens instantly, so as a DM I would just rule that the pickle jar is obliterated. How you ruled if they used a stronger container, like an adamantium jug, is kind of up to you - I would still rule that this is a magical effect, so the creature has to assume its original form instantly, and anything getting in the way would be destroyed or knocked out of the way.
That seems consistent with the wording of the spell, and also prevents the players from cheesing it. Well, from cheesing it too easily. There are a lot of ways that clever players can exploit polymorph, but I would definitely make them try harder than a pickle jar. YVMV.