The min/max argument - right, and D&D doesn't already have a billion little tools for this sort of thing.
Flaws don't work in a system that is designed only to add things to a PC. Flaws also, unless completely generic, are world specific, as are the value of certain merits/feats (a wilderness campaign will see little use for feats like "Mafia connections").
That said there are ways of bringing Flaws and Merits into a system that don't equate to power gaming.
1. Everyone has one flaw. All flaws are equal in power - say a -2 to two related skills (which the PC can use normally) or one -2 to a class ability (such as spell DCs), or -1 to a significant ability (saving throw, AC, Hit Points, etc). Since everyone has this flaw the effect is balanced.
2. Flaws and Merits are mixed blessings. One game (whose name escapes me) uses this method. The mechanical benefit of a flaw (the merit side of it) only comes into effect when the flaw is triggered. One way of looking at this would be a pseudo-2e Ranger's chosen enemy - you have an unending hatred for a certain type of creature, one that grants you benefits (the merit) when dealing with them offensively, however, you are also blinded by your hate (the flaw), possibly comitting Alignment infractions and other bad things. Another possibility in this area would be using the flaw as a "dynamic editing" of the story - i.e. the player have an impact in the direction of the plot narrative in the hopes of triggering their merit/flaw. The obvious example here is the nemesis - the character gaining bonuses when fighitng their nemesis but allowing the Player to insert their nemesis when needed (note that the Nemesis, to count as such, is an opponent worthy of the character, they are not mooks, and killing the nemesis merely resolves the plot, you lose the benefits forever).
3. The math is weighted against min/max. Another common way around the min/maxer - Flaws are simply worth less than the attendant merits. For example, you must take a at least two -2/-2 flaws in order to take a +2/+2 merit. If two is not enough then think about making it 3 or more. That way they still get the +2/+2 feat for "free" (not through level advancement) but the result is rarely worth it.
BESM d20 and Tri-Stat X would both have these features, as does Silhouette Core. But anyone really interested in seeing how Merit/Flaw systems have evolved are encouraged to look at White Wolf - in particular Adventure! and Mage Revised Player's Guide.