By that logic, any fighter carrying a shield, which CAN be used as a weapon, gets the TWF penalties.
Also, any one carrying a torch in their off-hand , whcih CAN be used as a weapon gets the penalties.
Not to mention if the TWF is moving and CAN'T use their off-hand weapon...