It's not much of an exaggeration to say that the real world pretty much operates on that principle, depending on how you scale "one point of damage." One would argue that you get ridiculous results like household pets killing people... but sometimes household pets, stubbed toes and tripping down the stairs do kill people. War isn't about people slugging it out until someone keels over, but trying to be lucky enough to not get hit at all.
So it wouldn't end violence, it would just scale up intimidation and the use of armies to exert power and bodyguards to protect you. In short, it'd become more of a political and strategic game than a tactical one (in the sense that most D&Ders mean "tactical," that is "interesting exchanges of blows between two individuals or two very small groups." It would certainly fulfill more classic definitions of "tactical," in the sense of "winning a battle between armies."
(Come to think of it, most wargames, especially ones with minis, just gives everyone "1 hit point," unless you're a totally awesome hero character)
Oh, and fast-draw/initiative would be king.
I'd play it, but I play GURPS, in which a single solid hit might not kill you, but it can easily be a fight ender, even if it comes from a mook. In my last GURPS Samurai game, one of our elite 350 point super-warriors got floored by a 25 point Yakuza goon with a crappy shortsword because he wasn't paying attention, let someone get behind him while he was otherwise occupied and got shived in the back. His buddies swooped in and made quick work of the goon, but it really highlighted how careful you have to be.
Of course, chambara (samurai stories) really focus on how fleeting life his and how dangerous battle can be, something I'm explicitly trying to emulate, but it wouldn't be appropriate to all genres.