The first question you have to ask is why you wish to do this? Is the game going to be better if your players don't min/max?
Off hand, I think you are fighting a problem that can't be solved entirely via rules. Regardless of the rule set, a character's success depends on having reliable choices. A smart player never trusts his success to a coin flip. If a character's powers don't succeed the overwhelming majority of the time, they aren't really worth having. It's far better to be reliably good in one area and reliably fail in every other area, than it is to fail half the time in every area.
The reasons for which I think the game would be better if min-maxed characters were not used are the following:
1. The "ideal" campaign, in my view, would not involve combat and combat alone, but rather it would be more of an "adventuring party simulator," and I'm already trying to develop rules to reflect that. To keep the game fun, there would still be a significant amount of combat; however, being big-time heroes who defeated the evil so-and-so of whatever won't make you exempt from having to eat, pay taxes, and otherwise manage your life. So far, I haven't seen any game like that.
2. The way I've designed the game so far, investing in a stat (or ignoring it) causes HUGE differences in your character's capabilities, even to a degree which I did not intend (though I'm not at the point where I'd say that I regret it). For example, like I said in the original post, the stat you would use primarily to determine damage would not increase any static values like in DnD, but instead it would allow you to roll more damage dice. This also makes weapon choice more important; the difference between wielding a dagger that has a d3 damage die and a giant hammer that has a d10 damage die will be much greater than in DnD. Given all that, a min-maxed character would quickly and easily get to the point of extreme overkill in the one area that they're good at, and that would take away from the game's overall "fun" value.
None of the rules you describe really get around that. In fact, you almost seem to have gone out of your way to make a system that encourages min/maxing and system mastery. Sight unseen, it sounds like a number fiddlers paradise on par with GURPS. In particular, your estimation that being classless supports the viability of well-rounded characters is exactly backwards. Classes support well-rounded characters. Removing the stricture of having to adhere to a class structure allows for maximum min/maxing.
To be clear, I'm talking about min-maxing as a specific form of number fiddling. That is, min-maxing, as I understand it, is when you invest in a narrow aspect of your character at the expense of everything else. As far as the classless system goes, I was thinking more in terms of how classes, as they're usually implemented, tend to narrowly limit what your character's function is, and how trying to go outside of that limit usually punishes your character more than anything else.
However, the real issue here in my opinion is less of a rules problem than a encounter design problem. As long as the group can depend on another member of the group to solve a problem in their area of specialization, there is no need to build a well balanced character. A well balanced party will tend to be far superior because everyone's problem solving will collectively be more reliable. To mitigate against that, you have to force everyone into situations where they are reliant on their own resources.
I'm not saying that characters shouldn't have focuses; in fact, I agree with what you're saying here. The problem is that, in my game, not only are there plenty of things which can be invested in to help a character with his/her focus, but in fact, to invest in only one or a small amount of those things will be less effective than paying some attention to them all. Like I said earlier, there's the example of Morale, which is boosted primarily by Resolve, an attribute that a warrior-type wouldn't even consider a tertiary attribute. Without a decent Morale defense, however, all your warrior's points in Strength would be rendered useless every time an enemy said "BOO!"
For example what genre is the game, and what tech level?
It's actually pretty anachronistic. Most of the technology in the core setting is somewhere in the range of 0-1400 AD, but in my game's universe, magic is based on scientific principles, and any adequately-capable wizard would have knowledge beyond even our own. Depending on the culture, a wizard's knowledge could be spread to others and applied beyond the spells he/she casts, such that, for example, even relatively uneducated peasants would know that being clean is important, even if they don't understand exactly why.
On the other hand, there's a nation which exists in an area where there's very little ambient ether (energy in its purest form, made into other forms of energy by the activities around it), and thus magic is less useful, since there's less energy to power it. Without magic, they've had to develop advanced technology such as guns, internal combustion engines, computers, and the like, all with a steampunk aesthetic.
And on another hand entirely, there's a nation of barbarians, where success is equal to physical prowess and most knowledge is seen as being ways for the weak to succeed without honor. Despite having little in the way of magic
or technology, they have successfully maintained and defended their nation due to the fact that, in my game's setting, the limits of what can be biologically achieved are much less restricting than our own; in other words, with sufficient exercise, a warrior could eventually be able to survive a direct hit from a tank shell, then destroy the tank with his/her own fists.
