Agreed. While I like the idea of skills vs defenses (or a variant of it), the 4E skill system would knock the math way off making it too easy or too hard to hit.
One drawback in the case of too many skills, is where a system of skills vs. defenses starts to become unwieldy and messy. (ie. Such a system where there's over 50 or 100 different skills).
Another drawback is determining how to allocate points in a skills system, such that it minimizes munchkin style minmaxing. Everybody knows that munchkin powergamers will allocate all their points into the combat and defense skills, while all the other skills becomes "dump skill" stats.