Personally, I saw Wolverine killing the Brotherhood mutants as pretty much in character. He's had a lot going on in the film, and the series as a whole, and he's not the best person at dealing with emotions.
Add the fact that he's probably psychotic (i'm not sure of the exact definition), violent, has razorsharp adamantine claws, and an excuse to use 'em (the fact that *they* were trying to kill *him*), well...
As has already been mentioned, he volunteered for the Weapon X process.
He's not a nice guy.