My own take falls more along the "no taxation without representation" side of things: no penalty without concurrent benefit. The benefit is you gain an extra attack (or more with later feats), the penalty is the reduction in attack accuracy.
The way I see it, yep. If you want to give up the benefit of a free hand - casting somatic spells, using a shield, holding an object, using a weapon in both hands, etc - in exchange for some versatility or style, then by all means.
Heck, I'd even let a character intersperse what weapon is used in the course of a full attack without penalty as long as they don't exceed their normal number of swings. Designate which hand is your lead and which is your off for purposes of strength modifier and go nuts. The only spot where it gets zany is with defending weapons, but it's pretty easy to keep that on the level by applying the same thinking - no bonus without penalty (which means actually attacking with that defending weapon, not just holding it in your off hand).