Cleon
Legend
Let's give this a try:
Voluntary Barbed Defense (Ex): A vampire squid can invert its toothy maw as a defensive measure; in this form, it cannot use its bite attack and takes a -X penalty on tentacle attacks. However, any creature striking it with handheld weapons or natural weapons takes X points of piercing and slashing damage from the squid's exposed teeth. Note that weapons with exceptional reach, such as longspears, do not endanger their users in this way.
The basic idea's sound but I don't like the name that much. I'd also like to include a save - probably Reflex for half damage, probably with a Con-based DC.
We also need to say what action is required, and I think we should note it can not jet or run when in a defensive ball.
I wouldn't bother with the attack penalty for the tentacles.
So, I'm thinking something like...
Ball of Fangs (Ex): A vampire squid can invert its toothy maw as a defensive measure, surrounding its body with its sword-long teeth. Inverting its maw takes a swift action, as does reverting back to its normal configuration. An inverted vampire squid can not use its bite attack, charge, Run or use its jet special ability.
Any creature striking an inverted vampire squid with a handheld or natural weapon takes 1d10+6 points of piercing and slashing damage (halved on a successful DC 15 Reflex save); except for opponents wielding handheld weapons with exceptional reach, such as longspears, who can attack the squid without taking damage from the exposed teeth. The save DC is Constitution-based.