I think the most complex rules are the ones I see people screwing up constantly.
1. When you cast a spell as a bonus action, you can't cast another spell on the same turn, unless it's a cantrip. I get the reason for this rule, but have yet to see a player who has access to bonus action spells NOT mess up and try to cast a regular spell on the same turn. Let's scrap this rule.
2. Don't add ability modifier to damage with an off-hand weapon. Again, I get it, and that's kind of an elegant balancing mechanism. But holy @#$%! does this confuse people. Let's just not worry about it.
3. Hit Dice: How do they work??? A lot of new players have trouble with this for some reason. Just when someone learns that "2d8+4" means "roll two 8-siders, add them, and add 4," along come Hit Dice. Now you have to roll 1d8+2, see if that heals you enough, and then decide whether to roll another 1d8+2. It's really not hard, but does it need to be that complicated?
I'd much rather all characters have a "recovery pool" equal to their hit points. During a short rest, you can transfer points from the recovery pool to your hit points. (I think Iron Heroes had something like this?) It's mathematically similar to Hit Dice, but much easier. I think the only reason 5E has its wonky Hit Dice system is because the term "Hit Dice" is a sacred cow that needs to be in the game somewhere.
4. Use Wisdom (Perception) in situation X but use Intelligence (Investigation) in situation Y. The only reason Intelligence (Investigation) exists is for low-Wis rogues to be able to find traps. Honestly, I really wish they had called it Intelligence (Searching) because then it'd be clearer when to use it.
A lot of DMs solve this problem by developing a set of criteria and heuristics for when to roll one skill versus the other. I've found a simpler solution to be: Roll whichever skill you want, I don't care. Pick the one that makes your character look cooler, and let's get on with the game.
Similar problems arise for Athletics vs. Acrobatics, Nature vs. Survival, Performance vs. musical instrument proficiency, etc., and I employ a similar solution.
5. Readying: State your trigger, state your action, use your reaction, you can't take multiple attacks, you can't move plus attack... This doesn't match anyone's expectations. I introduced a pretty simple "pass" rule (that is similar to Savage Worlds' "on hold" mechanic): If it's your turn, and your not ready to go yet, we'll skip you and you can jump back in later. You can't interrupt another character's turn, though.
With Readying, if you don't want to act, you still need to spend a lot of time thinking about what you want to do and what you are reacting to. We still use Ready for things like "If he attacks, I shoot him" or "If anyone comes into melee range, I hit them," because those reactions can happen during someone else's turn. But if a player just isn't sure what to do, instead of Readying or Dodging (which is often the same as losing a turn), people can just take their turn later.
6. Disengage doesn't let you move. This is an odd one, but I've seen many players, both new and old, make this mistake. When you take the Disengage action, it doesn't give you any movement (that would be the Dash action). In fact, when you Disengage, literally nothing at all happens unless you also move using your movement. That's very counter-intuitive for people. They expect the act of disengaging to imply some movement as part of the action.
I don't have any good solution to this one that doesn't trample all over Cunning Action. My preferred solution would be to roll the benefits of Disengage into the Dodge action; if you spend your turn defending, you don't suffer those Opportunity Attacks. This also reduces the number of different action types by 1. But it would be unbalanced to make Dodge an option for Cunning Action. Although, honestly, I've always felt that Cunning Action was really created just to paper over the fact that Hide is an action...
7. Hide is an action; if you don't take it, everybody still knows where you are. This is another one that makes total sense to me from a game-balance perspective; hiding can be very powerful, and making it use up your action is a great way to balance it. (Plus, it lets rogues be extra cool because they can hide as a bonus action.)
But, this is really not how players think about what's happening in the game. I've seen so many PCs shoot someone, then run around a corner and try to hide. I mean, they are totally out of sight, right? Even worse is when someone casts invisibility for the first time and then runs 30 feet and I have to explain to them that, no, they aren't hidden, and so even though they are invisible, everyone knows exactly where they are.
I don't have a super great solution to this either, and considering how problematic Stealth has been over the years, I'm hesitant to propose one. I think the invisibility thing could be fixed with a change to the spell or to the condition (e.g. "While you are invisible, you can take the Hide action as a bonus action"). I'd also consider something like: you can hide as a bonus action, but at disadvantage. (Unless you're a rogue with Cunning Action.) None of this makes the game any simpler, though.
Honestly, anyone who can come up with truly simple Stealth rules for D&D should win a Diana Jones Award for Lifetime Achievements in Gaming.
8. Object interactions: what counts? how many? huh? During the playtest, instead of giving players one free "object interaction" each round, the DM could give players as many "incidental actions" each round as seemed appropriate. I wish this were still the rule, because it solves a lot of problems with weapon-switching. Can you stow a bow, draw a sword, ready a shield, and attack with your sword, all on the same round? Officially, no; you'd have to spend a turn switching weapons.
That's boring to me, so I let players switch weapons for free. And I'm OK with a character picking up an object, opening a door, AND attacking someone on the same turn. This seems to me like one of those rare situations where "the DM just goes with whatever feels right" is actually the simpler rule.
