... snip
To be crystal clear and define my terms, I'm not talking about low-hanging fruit like synergizing race/lineage bonuses with your chosen class, or a rogue taking expertise in stealth or sleight of hand. What I'm talking about are the game breaking combos that...well, break the game.
In my experience, optimizers relish the thrill of the hunt away-from-the-table and want to show off their finds at the table. The trouble is being a DM at a table with optimizers. There seems to be one of four possible approaches to dealing with an optimized character and an optimizing player. First, you outright ban optimization. Second, you ramp up the combat challenges to such a degree that the optimized character is properly challenged...which will almost guarantee the non-optimized characters die regularly. Third, just never feature combat. Fourth, do nothing and let the optimized characters constantly walk all over any and all combat challenges.
... snip
TL;DR: optimizers ruin the fun for everyone but themselves at my table. Help.
Let me use a separation and define terms here so we are on the same page in discussion of terms.
Having insane HP, AC, Initiative, Stealth, Slight of hand, persuasion or any other skill is an Optimizer or often called a
Min/Maxer... but not game breaking.
Maximizing Damage is usually called a "power gamer" in my experience and is really the only thing that is game braking. You perfectly display with the second paragraph above being all about combat. Your not complaining about the high HP character that takes a long time to kill but can't kill anything. The high AC character which is virtually immune to attacks from melee fighters and archers but easily killed with AoEs and saves. The high initiative character who always goes first and is never surprised but has no advantage for the rest of the fight, the stealth character who is invisible and gets the jump until NPCs hold action and attack them when they break stealth to attack. The athletics/strength character who can break out of any trap. The Persuasion character who gets insane persuasion but you simply don't allow a roll for things an NPC would never agree to not matter how persuasive the player character is.
I find the best way to deal with "optimizers" is to ask them to be a min/maxer not a power gamer. I am a min/maxer my self. I like my character to be really good at something because it feels heroic. However, the heavy investment into maxing out one thing means minimizing in almost every other way meaning they are more flaws than perfection. The party can be composed to hit all the key areas in session 0 so that everyone has their own max without stepping on toes and in turn their own chance to shine i.e. preventing the ruining of fun for other players at the table (not that every area need to be or even should be covered, Party weaknesses are good too for story points too).
Most min/max gamers will not be happy without "the hunt away-from-the-table" and character design is huge component of their fun. Focusing their effort on anything but damage can generally satisfy their hunt without breaking the game in away you can't deal with. For example the persuasion king might allow greater access to resources and short cuts as could a super scout, but neither mean that they can't just go the longer path with less resources if that player/player character misses a session unless the GM just makes it that way.
In your, "yes, I've tried the standard 'why don't you try talking to your players' routine." , did you ask them not too optimize at all or did you ask them to optimize (min/max) in something other than damage?
If you have then your current answer, "So I basically have to choose. Which group of players will I run the game for. I don't have time for both." is correct. I do that too, but I also make sessions for specific players one at a time. The all participate as a group but when you have players of different styles who want to play in the same group you end up having to do this. Its is also true with RP players versus strategic players even without max/min or power gaming. Some players want a session of nothing but talking and shopping and others nothing but combat and nothing but combat. Your best bet is to write for each type of player in turn and make a note of your player types to consider which group might need some love.
“You can please some of the people all of the time, you can please all of the people some of the time, but you can’t please all of the people all of the time”.”
― John Lydgate
