maybe you should consider it a kensai/kensei, not a samurai...
are you familiar with the second edition oriental adventures? the kensei/kensai (sword-saint) is pretty much exactly what you're describing. bonus to AC when unarmored, basically only uses one weapon (katana) unless you want to use two. I agree with starting with the swordmage for the build but just as a very basic outline. you're not going to use the aegis, you're not going to have a magic shield, and you're not going to use magic...this is a martial build and your stats should be dex, str, wis. +1 to AC/Ref, quickdraw should be a class bonus feat, use the combat superiority and mark abilities from the fighter class, and then get into some house rules. This is basically a guy that specializes in fighting enemies by himself right? so give him something like the sneak attack from the rogue class except you get it when you're the only person fighting whatever you're fighting. +d6 damage when dueling, +2d6 at 11th, +3d6 at 21st. for powers do a combo of fighter/ranger/rogue and add in some of your own like a wave attack from you swinging your sword at somebody. Also, you need to increase his defenses a little faster than everybody else since they aren't supposed to use magic really, maybe add an additional 1 at every level a new daily power is available (+7 at 29th level). this is just off the top of my head, hoping other people will have waaaay better ideas spring from some of this crap i put down. As for the Katana - it's a bastard sword, no doubt about it, make it superior and high crit, i don't see it as a brutal weapon.