The problem is that that is not entirely a valid complaint - the way the defects system is designed takes care of this, and more besides - technically speaking, when useing defects you can also account for: Strength, Aglility, Endurance, Intelligence, Wits, Perception, Luck, Willpower, Charisma...