I can't speak to your experience, but a lot of players I know avoid casters for precisely that reason -- and while in other editions you can just not play a caster, it's baked into every class in 4e.
My experience is that, yes, in old-school D&D and 3.x, and even 5e, people avoid casters because the Vancian system is onerous, having to know all the available spells well enough to make good choices among them every 'day' when you prep your spells can just be too much, and because it's counter-genre and just unintuitive, and, particularly in 3.5, because they're just too broken to be fun. Not even the Wizard, the only class in 4e that was remotely Vancian, approached that level of issues, though.
For me, personally, it was the organization of spells in 3.0 that put me off casters for a bit (until I tried out a Sorcerer, that is). In 1e, I could play a first-level magic-user, go through the 30 first level spells, and get a clear idea of what I had a potential to do, then either pick (or have randomly assigned) 4 known spells, and just figure out how to use those really effectively until I found new ones, then, at 3rd level, repeat the process with 2nd level spells. The spells were all neatly arranged by class and level, so I could just browse through and familiarize myself with what a caster of a given level could potentially do. 2e, same thing. In 3.0, one little change - all the spells dropped in one alphabetized list - just blew that for me. 4e went back to the 1e layout, except, in even smaller chunks, 4-6 powers instead of 10-30 spells at each choice point. 5e, in spite of being so classic-feel across the board, inexplicably went back to the 3e format.
I mean, point taken, and the comparison seems valid, but it's still a problem for at least some people's preferences.
I really question that old saw. Yes, some players would balk at Vancian - particularly because it was nothing like what they expected from magic. And, yes, some would balk at the complexity of the system (even the simplest RPG is complex compared to something like monopoly). But players overwhelmingly choose a class because they like the idea of the archetype, not because they want a simpler/lower agency experience. Once they obtain some system mastery or internalize some community preconceptions, that can change, of course... they start to realize that there's the character they want to play, the character that will be most effective in theory, and the one that might be the most fun to actually play, and that they're all different.
6. Drawing your weapon as part of the attack action is fine, don't even worry about it, but you can't draw both weapons before attacking with them unless you're implementing an entirely optional system of character customization and wait until level four when you will be allowed to sacrifice your basic stats in order to pick up that capability. Seriously?
How hard is it to let slide, though? Let anyone draw two weapons to fight two-fisted. Let anyone draw as many daggers as they can throw, as well as as many arrows as they can loose. Let the drummer play his drums as an action, even though he has to pull out the drum and /both/ sticks (OMG, so many things!).
7. Anyone with a swim speed can swing a maul (or flail) underwater, with no issues whatsoever.
Rule otherwise. In an underwater scene, rule that everyone has resistance to bludgeoning. Done. It's not a mistake, but a missing detail: fill it in.
(Yeah, I know, I'm doing '5' again.)