I'm thinking just a minor annoyance to balance out a powerful weapon.
I'm thinking it would be like having an extra minion during battle to fight, but with some kind of special ability to set it apart from normal minions.
Another thought I've been contemplating is that perhaps the sword you find is part of an armor set that belonged to a powerful warrior, and that this goblin you fight is what remains of that warrior.
As you collect more of the armor set, this little goblin guy becomes stronger. When you collect all of the armor, you can transform this guy back to his true form and return him to his former glory.
...So... something that can be leveled up during the course of a campaign.
One other option is to have the item level itself up over the campaign - for example, turning +2 at 3rd, +3 at 8th, +4 at 13th, etc. If it's doing that, having the monster inside level up (in the normal way, just gaining a level and +1 to the relevant stats as it does so) makes sense.
I'd be tempted to either do it the easy way I mentioned earlier, where you just take ongoing damage (say 3 * Enhancement, save ends) from it chomping on the person or spend a healing surge to have it out per encounter... or stat it up as a minion of that level that springs forth at a time of its choosing once the sword is drawn... where a time of its choosing is almost always during a fight with other things, but could occasionally be to yell and draw attention or to chomp through a rope, or something similar.
I'd probably stat it up like a brute, so take the DMG p184 rules and just slot level in for everything. Ie, Level +3 vs AC is his attack, Level + 12 is his AC and all defenses. I'd set his damage to 5 + Lvl / 4. Now, that is a little boring so you might want to give him some special ability - I've been visualizing him like a redcap - all mouth and teeth. Maybe not what you've got in mind.
Finally, you could stat him up as a full fledged monster and have him show up immediately when the sword is drawn - they'll draw the sword outside of combat so it will just be him, and he'll get destroyed and don't be surprised if they get tired of fighting him.