According to PHB P.33, a fallen cleric (Ex-Cleric) loses all spells and class features. I read it as he loses spellcasting ability itself. Now he does not have any spells on his list. So he cannot use scrolls or wands (unless he use them with Use Magic Device skill).
Regarding a wizard who lost his spellbook, he still retain his spell casting ability. It is just that he cannot prepare any spell other than read magic until he finds or creates a new book. So he can still use scrolls and wands.