Mistakes were probably made on both sides.
"Long story short..."
Engulf is not automatic. You do get a saving throw. So lets here that long story.
"Squeezing down a 5x5 corridor..."
Normally, squeezing requires a penalty. Oozes are amorpheous plastic beings, and as such (thought its not explicit and it should be) they effectively have high epic levels of the escape artist skill. In other words, oozes get no penalty for squeezing.
"Not having leverage..."
You always have leverage in a grapple. At the very least, you can lever yourself against whatever force is holding you. If no force is holding you, you aren't grappled. If a gelatinious cube's interior was completely non-viscous, then you wouldn't have leverage, but you also wouldn't be grappled and could escape with a Swim skill check.
"2) I ruled that all the characters had an effective dex of 1 for reflex saves, since they were trapped inside the thing."
No. The rules are pretty clear here, and you should read them. The characters loose thier DEX bonus (if they have one) when grappled for the purposes of AC only. There are no other penalties.
Now, if they are paralyzed, that is a different story. Paralyzed characters are helpless, and thus automatically have a DEX of 0.
Paralyzed characters do not automatically fail reflex saves. They aren't even automatically hit by attacks (although with a Dex of 0 and a +4 bonus to hit them, its a fair bet they will be hit).
"Strength and Gelatinious cubes"
You just learned one of your major lessons as a DM. Minor changes in the abilities of a monster can have enormous impacts on the lethality of the monster, if what you change is one of the critical weaknesses of the monster. In the case of grapplers, increasing thier size and strength, especially if their size or strength is initially relatively low, will enormously increase the challenge. Your estimate of CR 5 is probably about correct, assuming you didn't do any other enhancements to the creature, but what you discovered is that one of the assumptions on gelatinous cube CR is that the players can recognize the creature and will take appropriate actions (like running away). Since you were dealing with inexperienced players, and since the Gelatinous cube is a TPK machine if not properly dealt with, I'm not surprised at the results.
If you really want to make the Gelantious Cube killer, fix its other two weaknesses - make it fly and increase its speed.
"Fireball exploding on impact"
Yes. Although the spellcraft check to know that should have been relatively low since its a property common to spells in general (and not just fireball), DC 8 would have been appropriate, and certainly not higher than DC 13. If a 4th level character had invested in spellcraft much at all, I don't see how they would have failed a spellcraft check of a DC like that. Using a typical 4th level Wizard I get a spellcraft skill check of +12 (+3 Int, +7 spellcraft ranks, +2 synergy from 5 ranks of Know (Arcana)), and that's not even counting a possible circumstance bonus if the Wizard had used the wand before.
"Cube blocks line of effect"
Yes it does. Only the wand user and the cube should have been effected by the full blast of the wand. The other players trapped in the cube effectively had full cover, though I would have ruled that if the blast dropped the cube to less than -10 hit points, that the blast burned through the cover (I don't want players arguing that a sheet of paper or a gauze curtain blocks line of effect on a fireball, even though under the RAW it does), and the players were exposed to the excess damage (probably only a couple of points) and that even then I think I would have given the party the equivalent of evasion, so that if they made thier reflex saves (paralyzed ones having 0 DEX) they would have avoided all damage.
"Characters knowing about Gelatinous cubes even if thier players do not"
I think its reasonable that the characters might know something about Gelatinous cubes, but rather unreasonable to assume that they automatically should know about them. My personal feeling is that oozes are 'natural' (non-magical) beings in a fantasy world, and thus fall under the province of Knowledge (Nature). If you think oozes are magical, then they should fall under Knowledge (Arcana). Unless thier background prevents it, I personally allow characters to make intelligence checks on any knowledge that they aren't trained in to reflect the fact that people will collect bits of information about fields even if they aren't skilled in them. A gelatinous cube is a fairly rare creature, not ordinarily a part of an average person's experience, so I'd give like a DC 20 Knowledge (Nature)/Intelligence check to recognize one (as a type of dangerous ooze) when it had been spotted, and a DC 25 Knowledge (Nature)/Intelligence check to know significant things about it (it paralyzes by touch, it engulfs foes by running over them, its slow moving, its harmed by ordinary weapons).
