Not strictly true. From the PHB glossary we have an unarmed strike as a 'successful blow .... from a character attacking without weapons.'
The phrase "attacking without weapons" is not the same as the phrase "attacking while not holding weapons". If I punch you in the face, I'm attacking you without a weapon, whether I'm holding a sword in that hand or not.
Saying that, and since non-monks would have to take an extra feat to be able to easily attack with an unarmed strike anyway, I'd rule that it's not against the spirit of the rules to allow unarmed attacks to be made as off-hand attacks.
Honestly, I think the rules support it, at least indirectly, anyway. For instance, the Combat I portion of the SRD has this to say:
"Unarmed Attacks: Striking for damage with punches, kicks, and head butts is much like attacking with a melee weapon, except for the following:"
The following exceptions are that unarmed attacks provoke attacks of opportunity, deal less damage than most melee weapons of the character's size, and don't deal lethal damage normally.
Actually, strike that. The SRD outright states, in its definition of unarmed attacks, the following, under the heading '"Armed" unarmed attacks':
"Unarmed strikes count as light weapons (for purposes of two-weapon attack penalties and so on)."
Therefore, you could use unarmed attacks as off-hand attacks. It also mentions that unarmed attacks include such things as "punches, kicks, and headbutts", so it's simple to infer that you could choose to make an unarmed attack, such as a kick, even when you're holding two swords.