The basic way to discourage min/maxing is to arrange the incentives in your system so that characters can more easily become powerful by being well-rounded. In particular, it needs to be easier to improve at something you're bad at than at something you're good at.
I don't know the system that you're working on, but I think I can illustrate what I'm saying using the d20 system. The same principles apply to any system that you build.
1. Skill Points
The d20 skill points system encourages min/maxing of skills. The reasons for this are subtle. First, because it often suffices for only one member of a party to pass a skill check, it makes sense for the members of a party to specialize in certain skills. Second, because the DC's of skill checks often increase with level, you really need to keep up with the skills you have for them to remain useful.
In any case, here's a simple rule that would encourage characters to pick up more skills:
The first skill point gives a +5 bonus to checks in that skill. Every additional skill rank gives +1.
With this system, there's a much bigger incentive for characters to put at least one skill point in many different skills, since that single skill point makes a bigger difference in a new skill than in a skill that a character is good at.
Of course, this system will encourage characters to take only one rank in every skill. If you want to make the choices even more interesting, you could make a sliding scale:
[table="width: 500"]
[tr]
[td]Skill Ranks[/td]
[td]Bonus[/td]
[/tr]
[tr]
[td]1[/td]
[td]+5[/td]
[/tr]
[tr]
[td]2[/td]
[td]+9[/td]
[/tr]
[tr]
[td]3[/td]
[td]+12[/td]
[/tr]
[tr]
[td]4[/td]
[td]+14[/td]
[/tr]
[tr]
[td]5 or more[/td]
[td]skill ranks +10[/td]
[/tr]
[/table]
This has the disadvantage of increasing the complexity, but it really encourages characters to put as many as four skill points into skills that they're "bad" at.
Of course, if you adopted this system, you would need to increase the DC's of most skill checks, since characters would be receiving such large bonuses for skills that they're good at. You might also want to give characters more skill points (e.g. an extra 2 per level) so that they have enough to spread around.
2. Ability Scores
The d20 system already has a mechanism to prevent min/maxing of ability scores at character creation, namely the "point buy" system:
[table="width: 500"]
[tr]
[td]Score[/td]
[td]9[/td]
[td]10[/td]
[td]11[/td]
[td]12[/td]
[td]13[/td]
[td]14[/td]
[td]15[/td]
[td]16[/td]
[td]17[/td]
[td]18[/td]
[/tr]
[tr]
[td]Cost[/td]
[td]1[/td]
[td]2[/td]
[td]3[/td]
[td]4[/td]
[td]5[/td]
[td]6[/td]
[td]8[/td]
[td]10[/td]
[td]13[/td]
[td]16[/td]
[/tr]
[/table]
Using this system, it usually makes sense to have a nice range of scores, with some very high, some medium, and some low. Of course, it would be possible to increase the costs of the very high scores, which would discourage min/maxing even more.
In d20, characters also get a +1 increase to a single ability score every 4th level, and characters often choose to increase just a single score (usually their primary stat). Unfortunately, this encourages min/maxing. Here's an alternative rule:
Every fourth level, characters get 3 additional points that they can use to "point buy" higher ability scores.
For example, a character could choose to increase a single 16 to a 17, or could increase a low ability score by 3, or three low ability scores by 1 each. This system would encourage characters to spend points to improve their low scores, although they would still have the option of improving their best score.
Finally, characters also use magic items to improve their ability scores. Again, there is already a system in place to discourage min/maxing, which is the sliding cost scale for ability score improvements:
[table="width: 500"]
[tr]
[td]Bonus[/td]
[td]+2[/td]
[td]+4[/td]
[td]+6[/td]
[/tr]
[tr]
[td]Cost[/td]
[td]4,000 gp[/td]
[td]16,000 gp[/td]
[td]36,000 gp[/td]
[/tr]
[/table]
Because of these costs, it usually makes more sense for a character to get two different +2 items than a single +4 item.
If you want to discourage min/maxing even more, one way would be to make magic items available that improve more than one ability score. For example, there could be an 8,000 gp magic item that grants a +2 enhancement bonus to all physical ability scores, and another one that grants a +2 enhancement bonus to all mental ability scores. Or, if you prefer, there could be 6,000 gp items that grant +2 to one physical and one mental each, or a 12,000 gp item that grants +2 to all ability scores, and so forth. The point is, if you want characters to spend money to become more well rounded, then incentives to do so need to be built into the selection of magic items.
3. Other Things
The same general philosophy applies to any part of the system: *if you want characters to be well-rounded, you need to build incentives for this directly into the system*.
For example, most melee characters in a d20 system take exclusively combat feats, to the exclusion of all other choices. If you want to prevent this, you would need to sort the feats into categories, e.g. "combat feats" and "skill feats", or some such. If you want characters to take both, there are a few choices:
1. Mandate that every character take some combat feats and some skill feats. For example, some of the bonus feats that characters get could be required to be skill feats, and some could be required to be comabt feats.
2. Instead of mandating, just make it easier for characters to pick up feats in their non-preferred category. For example, you could make a rule that, whenever a character has four more combat feats than skill feats, then they can pick up two skill feats for the price of one.
You may even be unhappy with the way that the d20 system encourages characters to specialize in either fighting or magic. If you want to see more gish characters (on the grounds that they're well-rounded), just make it easier to gain combat power when you have lots of magic power, and vice-versa. For example, you could make a rule that any character with at least six more levels of wizard than fighter can get two levels of fighter for the price of one.
The point is, players will always tend to build the most powerful characters they can in a given system, and for most RPG systems this means min/maxing. If you want to encourage well-roundedness, you really need to put incentives into the system itself to encourage this. It needs to be easy to pick up new abilities and improve at the things that you're bad at, but hard to improve at the things that you're good at.
Finally, none of this will matter unless it's also to a character's advantage to become well-rounded. There needs to be some in-game reason why every single skill or ability is important to every single character. For example:
1. For every single skill, there would need to be some reason that it's advantageous for every member of a party to have that skill. For example, if you want everyone to feel like they should personally take a Knowledge skill, there needs to be some personal bonus to having knowledge that can't be communicated. Maybe passing a Knowledge check on a monster should give you a +2 insight bonus to attack rolls against that monster for the rest of the encounter . . . something like that.
2. Every feat or ability needs to be useful for every character. If you want wizards to improve their combat abilities, then there needs to be some reason that wizards commonly find themselves in combat. For example, it might be common to run into monsters that are immune to magic, or perhaps the way magic works is that wizards need to "recharge" (not cast spells) for a few rounds after casting a powerful spell.