Do you actually, in practice, only hit what you're "swinging" at in your 4E games about 50% of the time? Is it really, on average, taking 5+ hits to drop any given opponent?
If so, then I really do feel your pain. That really would make every combat a grind. In practice though, you ought to...