It seems to me that it really is something physical that was at the monastery, probably a sword. Anyone see the sword that Buffy is wielding in the previews? I also think that just because the writing says that he cannot wield it, it doesn't mean that he doesn't have it. Basically, I think that Buffy had the right idea, but she just didn't know what it was that they were looking for. As far as that goes, it is much easier to do a raid going after something specific than it is to just go in and kick butt.
About Caleb not killing Buffy. That eye thing that Giles talked to basically said that the First is going after the slayers right now because it can, the slayer line has been weakened. Whatever it is that caused that weakening, Buffy is the focus of it. For the First to kill all of the slayers, it must kill all of the potentials, and then it can kill Buffy. This is one of the reasons that I think Buffy being kicked out is bad. The First can now hit the house as hard as it wants, and not have to worry about Buffy dying at the wrong time during the fight. Of course, that will only work if Buffy being kicked out is what it was trying to do. If it was trying to get Buffy to do exactly what she did, but successfully, then it doesn't know that she got kicked out, yet.
