Treat these guys as a 13-14th level party.
Party size+50% standard
Hit points ~ +30% standard (average 65 10th (d8 con +2)
DPS feels around +20-30% standard (pure guess)
Get the feeling they also have high stats. What's their total pluses (standard +7)
The bold doesn't seem right. A 10th level character with d8 HP (e.g. rogue) and Con 14 will have 73 HP at level 10 (10 * 7 + 3 bonus at first level), or 68.5 if they roll for HP.
But yeah, if there are fighters in that group with 115 HP they either rolled really well on stats + HP, or they are leveraging spells and abilities like Aid and Inspired Leadership.
I really like what you're saying here. The trick is in the mentality of your player. I have a player in my group that will ALWAYS min/max no matter what you do...and then complain that a) it's not challenging enough, or b) I'm nerfing him to make the challenge too difficult.
The changes I made in DMing a specific group of players (I have two) where one always min/maxes while the others tend to play what they want because it's just plain fun is this: I have encounters that are tough and some that are very easy. For instance, the real world is full of things that are easy and challenging. And so goes the world of D&D. A 15th level group finds a raiding kobold group on the edge of town that's pestering farmers. The group goes out, handily wins the day with everyone contributing, but no EXP is earned. The story is advanced and a certain amount of noteriety is earned, however. I haven't had to force anyone to change their playstyle, everyone had fun mopping up the kobolds (sad really, poor things), and it sends the message that I will put you into ANY encounter...ANY (you best be running away from stuff that's too big to tackle!). I rarely see this in games but I think it's a great way to balance the game. The OP player walks away feeling OP and the other feel that they're not left hiding behind an OP shield through an encounter.
I have players at my table that are so competitive that if I throw more monsters at them, they do not learn to dial the minmaxing into a different direction like you suggest, instead they say "Ha! More XP for us.". I just threw an "ambushing assassination squad" at the PC as they were traveling down a road that came within a hair of two PC deaths and my expectation is that the first thing they will probably do when we play tomorrow is change how they spread out and their marching order when traveling down a road from now on (a player has already Emailed the group asking what can be done). Throwing a deadly encounter at them doesn't get them changing the direction of their optimizing so that it is less obvious, it changes their tactics.
That is also why I enjoy playing hyperintelligent monsters with good intel access, because then I actually have an excuse to pull out some of that "what you do if..." content against the PCs without breaking the 4th wall. It is fun to use good tactics like drow stealth, hobgoblin fortifications with 3/4 cover, beholders drilling through ceilings with Disintegrate, beholders shielding themselves from arrow fire with wooden barricades lofted by Telekinesis[1], etc.
[1] Unfortunately this one doesn't work reliably with MM beholders because their eye-rays are random. Either you accept that fact and have the beholder stay mostly ground-bound, moving in short bursts whenever they roll a telekinesis ray; or you say, "That's lame, untraditional for D&D, and unrealistic: beholders can choose their eye rays in my game as long as they don't use the same one twice in one round." Per DMG guidelines that doesn't change the CR, so you're good.
Yeah, that's bogus. Only Wild Magic Sorcerers or Int 1 or so creatures should do random stuff. Everything else should use the best ability for the situation (assuming that they do not want to conserve resources).
I'm not 100% convinced about that. If I wanted to keep the beholders random, I'd say they're schizoid, and that each eye is controlled by a different aspect of the beholder's personality, and that the "death ray" mind scores political points for killing an enemy, so the "enervation ray" mind may throw in with the "charm" mind this turn to prevent "death ray" from scoring too much, given that the beholder only has enough magical mojo to power three shots per turn (plus Legendary Actions later). This intense politicking appears random to an outside observer, resulting in a random choice of eye rays.
This theory is completely consistent with the beholder's racial history of intense xenophobia and especial distrust for other beholders in particular who are almost completely similar to themselves save for minor details (slightly longer eye stalks). Distrusting yourself is the next logical step in that progression.
If one feat, in one situation could instantly unbalance the whole shootin' match... what would several humans with feats do?