Heh. I'm going to start out seeming to take one side but end up at the other!
If the off-hand weapon is being used for no purpose other than to look really pretty in the fighter's off hand, then I see no absolute reason to give the penalty. [Although if a DM said that the penalty stuck I'd find no absolute reason to argue, either]
However, this would also negate any benefits from the Two Weapon Defense feat (including the bit on fighting defensively or using the total defense option within the TWD feat). If the person was only wanting to use one weapon, then they should only use one weapon for purposes of defense as well.
So, if the character in question wanted the bonus to attack and compensated by removing any bonuses from the TWD feat (if they had any) then I would see that as an agreeable compromise.
BUT. As a DM and as a matter of convenience I would say to the player:
"Listen, you most-likely have a BAB of +1 (or greater) and therefore can draw a single weapon, a double weapon, or two single weapons as part of a move action. Since drawing the second weapon gains you no bonuses or penalties in any way, shape, or form (unless you plan on using it) and since you can draw the off-hand weapon in any round that you want to use it later as part of a move action ... just don't draw the off-hand weapon until you need it. That way, this argument doesn't even have to come up."
Ultiamtely, then, I would put this on the player's shoulders to really only draw the weapon when the player wanted to use it. My reason has nothing to do with reading the rules, it has everything to do with the fact that DMing is messy enough with keeping all the numbers straight. If it makes my job easier to say 1 weapon draw is this set of numbers and 2 weapons is this set without having to go through the whole "is the player using the second weapon for any benefit - either AC or attack" line of thinking ... well, I think that is a fair compromise.
As a player, if you have a BAB of +1 or greater, just do your DM a favor and don't put them in that situation. Why start an argument when it really gets you nothing anyway?