Awesome that you're converting X5. The tentacles are an interesting thing to convert. I would suggest treating it somewhat like Swallow Whole. Thus, once someone is trapped in the tentacles the Malfera can continue to act unhindered and without taking a -20 to grapple checks to hold the victim. The victim can use a light piercing or slashing weapon and if it does enough damage it cuts enough tentacles to free itself. Alternatively it can escape with a grapple check and presumably land in an adjacent square. This is just like the Swallow Whole rules except the rules for getting caught are as per Improved Grab. Otherwise, it's as if the Malfera had a stomach on the outside (ew) and if/when you break free you end up in a neighboring square, just like escaping a stomach.
For extra fun, you can allow allies to attempt to wrest a victim from the tentacles. They would make a touch attack (provoking an AoO as normal) and if successful make a grapple check to grab hold. A subsequent successful grapple check could let them pull the victim free or an Aid Another attempt could be made by beating a DC 10 with the grapple check, giving the victim a +2 on its grapple check to escape.
Good luck!
