Vow of Poverty lets you use a lot of things, you just can't own them.
Scrolls are legal as things are now. So are potions and oils. Wands and their relatives are supposed to be off limits, but if your DM is okay with them being used once in a while that's up to them. Just don't make a habit of it.
Most important on those things though is that if they are used once in a while, make sure there's good faith. Not, "Well these potions are really his, I just have to carry them for him".
Want your Paladin to use VOP? So let your Paladin use a quarterstaff. Or a Mace. Or a Morningstar. Or a sickle. Or any other simple weapon.
If you allow armor of any kind, then edit out the AC bonuses the VOP gives. They shouldn't get both.
I might allow a shield, if it's one that has the family crest, or the symbol for the church or noble that they serve. Better if it's cast aside when the battle begins though.
Over all, while we tend to see Paladins as the classic "Knight in Shining Armor", we all know from play that the armor doesn't always shine. And if that armor is provided by righteousness instead of by a smith? So what. Maybe they "shine" without armor when the VOP AC bonuses come into play.
If you're feeling generous let them have a second simple weapon, like a crossbow or a sling, so they have some ranged damage potential.
Sorry if I'm the wet blanket. I guess I've played with too many opportunistic rules gymnasts.