Problem 1 - Player - March Order
Any march order that allows the entire party to be consumed by a 10x10 cube has critical weaknesses. This march order will get them killed quickly with Area of Effect spells, traps, breath weapons, and many other effects. Furthermore, the scout (or at least a character with good perception skills) needs to be advanced out front to perceive threats and prevent surprise rounds.
Problem 2 - Player - Reaction to Cube
Characters with good Reflex saves could have tried evading it. Otherwise, an AoO beats off the Engulf attack, or you can run away. I guess they all failed their attempts to escape.
Problem 3 - GM - Souped Up Monster
Adding 8 STR points is pretty hard core, and makes the thing much harder to deal with. It would be like coming across an NPC mage using save-or-die spells with 26 INT. I guess you did increase the CR, but with a save-or-die area-of-effect monster like the cube, it increases the chances of a TPK.
Problem 4 - GM - Grapple Rules
Already addressed above.
Problem 5 - Players - Defense vs. Grapple
Grapples happen, occasionally. The party needs some way to deal with them: Someone should have escape artist, or Grease spells, or Improved Grapple, or a high strength character that can win a grapple check, something.
Problem 6 - GM - Fireball AoE
If the fireball couldn't penetrate the jelly to get out of the cube, the other characters probably shouldn't have gotten BBQd, RAW. However, I don't have a huge problem with this one -- a fireball going off 5' in front of your face should cause some damage, even if there's a wall of jello in the way. However, as a GM, I would have considered showing leniency at this point merely to avoid the TPK if I thought a TPK would derail the campaign. You can always claim that you weren't sure about the rule, erred on the side of the players to keep things moving along, and will fully research it for the future. Kind of a one time "get out of jail free" card.