Interesting thread. Frankly, having DMed for optimizers in every edition of D&D, 5e optimization isn't even a thing, IMO. Compared to DMing 3e for 10 years, including up into high levels, the notion of 5e optimization is laughable. In 3rd edition, I had to deal with literal demi-gods stalking the battlefield. I don't have this issue with 5e.
Frankly, there is no build I have seen yet in 5e that I would even consider a speed bump as a DM, and I pretty much allow everything WotC publishes and play by RAW. Perhaps there is some optimization board theorycrafting uber build I haven't encountered, if so, I'll maybe need to house rule something. But that day hasn't come yet. Though, we should make a distinction between optimizers and actual rules abuse. I have no issue with players who leverage the rules of the game to a make a hyper competent character, assuming that the game rules themselves are balanced and well designed. I do have a problem with players who exploit vaguely worded feats and spells to distort the intent of the design. That's easily handled with some house rules or errata, both official and unofficial. But in some cases, the game itself is just poorly designed and balanced. 3e had this problem in spades. It was frankly inevitable that casters would dominate non-casters. It was baked into the game design.
But in 5e, the difference between a competent build and an 'optimized' build is at best 10-20%. 5e rules are pretty tight. But I certainly can see the case where optimization may appear to be a problem in 5e if either of two things are true:
- The DM is not themselves tactical or optimization minded. This frequently manifests as the DM feeling threatened or overwhelmed by optimized characters since they themselves don't employ efficient monster tactics, or design encounters suited to their group. Frankly, this is a DM problem, not a player problem, and not easily solved. Anyone can become a good DM, but it does take skill and experience. And it involves skills in program management, and people management, and yes, you do need to understand the underlying math of the game. You should be able to read a rulebook and be able to identify and analyze potential optimization tactics before your players ever ask you if they can play it. All good DM's do this, even if its subconscious. Not everyone feels comfortable with this. That's ok, but if you aren't, then you should consider that DMing may not be for you at least without some investment to grow your skills. I've never seen a good DM who couldn't do this.
- Some players are deliberately designing sub-optimal characters. Again, if PC's are not being designed to a baseline level of competence for their level, that is the fault of those players, not those who are actually making competent or even optimized characters. Talk to them out of game and explain the issue. Give them an opportunity to tune up their builds. Perhaps asking advice from the players who are better at optimizing. Otherwise, they either accept that their PCs will be consistently outshone, or find a different game. Early on as a DM, I struggled with this dynamic when I tried to adopt the conventional message board wisdom that optimization = bad. Then I realized the optimizers weren't my problem, it was the ones who refused to engage with the mechanics of the game at a sufficient level that were causing all the disruption.
To explain it differently. If a competent build is an 8 out of 10, and an optimized build is a 9 or a 10. Then your game will be much smoother if you are designing for the 8-10 range. But if you have to deal with builds that can range anywhere from a 1-7 in competence and ability, then the game won't work. And the fault isn't with the 8-10 builds. Its with the 1-7 builds. And likewise as a DM, you should feel comfortable DMing for 8-10's. Of course, if the game itself is broken and an optimized build can be a 9-20, and totally break the scale, well then I would find a different game to play (I'm looking at you 3rd edition)... But 5e doesn't really have this problem.