So just a point. While swinging a sword may take slightly longer, it takes about
six hundredths of a second to throw a punch. Yet people respond to and block those punches. In D&D you can react to and block a dozen or more punches from different opponents as long as you can see them all.
I just don't see why people would have enough time to react to those attacks, but not the attacks of someone that leans out from cover, aims, times it so they aren't hitting their ally
when they know where the attack is likely to come from. Totally unexpected? Okay, it kind of makes sense. The second or third time? You have as much warning as the fact that ogre #3 that's directly behind you is going to throw a punch while ogre #2 to your side and ogre #1 in your face is doing the same.