Yes, you can combine both. The PBP I run uses both the Buffy & Angel corebooks (though we use just the Buffy character types; Heros & White Hats).
The buffy magic system assigns a power level to each spell. For the spell to go off correctly, you need to get that number of success levels. Otherwise, the spell either flubs or 'something interesting' happens.
It does a good job of capturing the flavor of magic in the series, I think. Namely, magic hard and not alltogether a goodness. To what ends you use magic for can affect your life and how things go if you don't manage to get the spell off right. Also, not exactly the system to be throwing about attack spells if you want to be a 'goody two shoes.'