I'd say that if you're going to dump the "mount" part, at least keep something similar, either a special weapon or an animal companion. The problem is, once you get past level 10 or so there's no need for rideable creatures (once you can teleport, wind walk, gate, plane shift, etc.), and they actively impede your ability to go through dungeons or other planes. So, just replacing the horse with a non-rideable creature, keeping all other bonuses the same, results in an increase in power. On the other hand, at high levels the standard Mount is often so vulnerable that it's not worth bringing into combat, unless you buy tons of horse-sized magic items, so it could USE an increase in power... unless you use the rules for having Celestial horse mounts, or Dragon mounts, in which case they become too strong again.
A while back, my group switched the base Mount rules to a more customizable system; any Animal would do, even if it wasn't large enough to ride, and you selected abilities from a small list as it levelled up. If you wanted a rideable creature you could have one, but if you wanted a more combat-oriented companion you could do that, too.
I posted an early version of it on this board a couple years back:
http://www.enworld.org/showthread.php?t=48419
That post is up-to-date except that we made the change mentioned in a later post in that thread, where the bonuses are split so that there's always something at every level.
It's a definite increase in raw power for the Paladin, but it hasn't been unbalancing, and it makes things a lot of fun. In our campaign we ended up having two Paladins at various times; one was an Elf with a horse Mount that closely mimicked the PHB progression, and the other was a Halfling with a Dire Weasel "companion" that would fight side-by-side with him. Both ways worked well.
