It's really standard that when something teleports, anything mounted on it, that it's grabbing, etc comes with it. See dragons in Pern, and, yes, if you're actually well versed in fantasy, a variety of situations where when someone's in skin contact, a teleport transports them both.
That's not actually important, of course, as the real question if how this works in D&D. It's not unreasonable that being essentially mounted on a teleporting creature will cause you to teleport when it does. But really, the question is one of rules. If teleportation is "movement", then you'll teleport with your mount when you or your mount are hit with forced teleportation (but not necessarily when you use a teleport power) and Close Quarters will let you go with a creature when it teleports, uses planar travel, or whatnot. If not, not.
The con is easy -- movement is movement, teleportation is teleportation; never the twain shall meet.
For a contrary opinion, I'll reference the rules definition of teleportation: Many powers and rituals allow you to teleport—to move instantaneously from one point to another.
The rest of the definition features a set of ways in which teleportation differs from normal movement, forced teleportation differs from normal forced movement, etc. So, teleportation is movement -- movement that's instantaneous and doesn't need to follow line of sight rules. As such, there's nothing particularly wrong with applying the forced movement while mounted or Close Quarters text to teleportation.
