The catch for me is that all those people just don't have their crap together, i.e., no self-respect, no self-esteem, no enlightenment... which means I have no real respect for them... which makes them uninteresting. And I have trouble remaining focused on things that don't interest me.
I've actually had people (including a shrink) jokingly tell me I should start a cult, but being a cult leader requires a major investment of time and energy, and I just don't have the sort of focus and dedication required - I'd get bored or frustrated pretty quickly.