Items:
*Since you use a longsword, I'm guessing you're a sword & boarder. If you have power attack, I would invest in Armbands of Might (~4K) from the Complete series. They give +2 to all strength checks (meh) and +2 damage when you power attack for -2 or more (makes your power attack efficient, even with a one handed weapon and shield).
*+1 armor and +1 shield would be good. They're very cheap and well worth the money.
*A +1 vest of resistance or ring or protection is an alright purchase. However, note that these provide Resistance bonuses to saves. Protection from Evil provides a nice +2 bonus to saves from evil creatures, which doesn't stack with the Resistance items. Thus, I'd be far more inclined to pick up a wand of Protection from Evil or some scrolls than I would be to stock up on Resistance & Deflection bonus items.
*Get a cold iron sword and a scroll of magic weapon. Ta-da! +1 cold iron long sword.
*Get a silver weapon -- even a dagger or short sword should be fine for those Devilish or Lycanthropic emergencies.
*Bane ammunition is pretty cheap and well worth your money. Opening up a fight with a heavy crossbow bolt doing 1d10+2d6+3 damage is beautiful at low to mid levels. As a paladin, I'm guessing you'll run up against Evil Extraplanar creatures and the Undead at some point; a Bane Goblinoids bolt is also handy, merely, because it covers three common humanoid bases (goblins, hobgoblins, and bugbears), and even a goblin with some class levels can be annoying as all heck to kill.
*+1 Mithril chain shirt barding is the next armor upgrade for your mount, and its very pricey for the value right now. Definitely a low priority.
Feats:
*Divine feats are your friend. Divine Might is great because it is a free action to activate. I like Divine Vigor a whole lot -- temporary hit points are great. Divine Shield is alright, but the standard-action-to-activate thing really stinks.
*Extra Smiting is also good for a charger as the extra damage is multiplied on a charge for lots of damage. And free actions are your friend.
*If you don't have it, Power Attack is good for a charger as the damage is multiplied on a charge.
*If your DM will let you customize your mount, see if you can have it take a useful feat like Improved Toughness at HD 6. Also, the DMG has some optional rules for variants on mounts. For example, -1 effective paladin level in exchange for applying the Celestial Template. DR 5 and a bunch of energy resistances? Yes please!
PrCs/Multiclassing:
*If you go in for level 5 paladin, I'd stick around for 6. The +1 to all saves, +1 caster level, and Remove Disease are nice to have.
*I'm assuming you're building a charger. That being the case, I'd suggest 1-2 levels of sorcerer. For my level 1 spells I'd pick True Strike (to gurantee that my charges struck true and to fuel a maxed out power attack) and either Featherfall (no somatics) or Mount (never be without a steed, even if your celestial warhorse dies or is unavailable). Duskblade from PHB2 might also be a good pick for a 1 level dip as it has full BAB and solid HD, but you don't get the nice Charisma Synergy.
*You might find it necessary to squeeze in a level of fighter or something that gives fighter bonus feats. I'm guessing you have Mounted Combat --> Ride By Attack --> Spirited Charge right now, which is great, but not very flexible. And if you should ever get unhorsed, your combat utility drops like a rock. Even adding Power Attack would increase your versatility a lot, and frees up your level 6 feat for a Divine Feat.
*Hospitaler (Complete Divine) might be a good PrC to dip for a few levels. It should be easy to get into, works nicely with Paladin, and gives a Fighter Bonus Feat at level 1. Levels 2-4 advance Divine Spellcasting.