I wouldn't give a set bonus. I'd give a big bonus for the first round- +5 or more- because the target just sees some moron waving his empty hands around, weaponless.
"Oh NOES! A KILLER MIME!"
However, once the first blow lands, I'd give them a single Wis based perception check- IOW, Spot- each subsequent round to let them reduce that bonus by 2 per success, to a minimum of +1, as they gauge the weapon's real dimensions. (I'd have to think a bit about setting the DC for that check, though.)
Yeah, its a bit of bookkeeping, but it has a good feel to it.
Of course, I'd also let someone actually able to see the sword negate the bonus entirely- without making any Spot checks- and actually get a +1 to their first attack vs the guy wielding the invisible weapon as he lures him in (with a successful Bluff check).