Execution (that is, killing someone) is not "good" in any fashion; it is lawful. Problem is most people equate Law and Good to mean the same thing. Generally if you perform an evil act for a good reason it could balance to neutral. I don't mean a justified reason; I mean good.
The way I see it, OP was acting within the bounds of neutrality. I don't believe that being an old lady distinguishes you from being a young man with a sword on the moral scheme of things when we're talking about swift justice (whether truly just or not). Why is killing someone in self-defense any different morally than killing someone because they are a threat to your authority? At the end of the day, you still took someone's life. I think it is well within a party's power to neutralize a hostile target without killing them; thus the fact that 95% of people who would argue alignment in the first place feeling that self-defense is justified, non-evil killing means OP's scenario should be, as well. For all intents and purposes, that hostile level 1 warrior is just as helpless as a noncombative old woman. This is D&D we're talking about, where a single mid level character can wipe out an entire town. Even if this were not the case, are we comparing alignment to individuals or to the world at large? If wiping out threats to one's authority by any means necessary allows one to act in the capacity for greater good, does that not by definition make one neutral? It certainly doesn't make you good, and by progressive reasoning it wouldn't make you evil, either.
There again I could argue that any kind of killing is immediately neutral, further modified by motive and personality and all that. I simply refuse to agree that a group of people turning on their friend and slaying him because he slew someone else is "good". It reeks of vigilante justice, which is almost by definition Lawful Neutral. The entire concept of punishment is an ethical realization and has nothing to do with goods and evils. As a matter of fact, the entire concept of the paladin class is silly, as a truly good individual would not mete out harsh justice to any living creature on sight.
Executing a murderer isn't a "good" thing, it is a "lawful" thing.
I hate alignment so much that it makes my face hurt, but it's fun to debate. In our games, races that aren't outsiders, undead, or paragons of an alignment are neutral. Trolls? Neutral. Orcs? Neutral. Dwarves? G...Neutral. In a world of angels and demons it is very easy to assume that human (read: mortal) capacities for alignment axes simply cannot contend with the extent to which outsiders can go. Characters are people, whereas outsiders define the concepts to which mortals adhere.