In the assumed setting of 4e - let's say the Nentir Vale - psionics are thought to be the universe's auto-immune system response to the creeping cancer of the Far Realms.
In Eberron, psionics are tied to the planes of Xoriat (i.e., madness) and Dal Quor (i.e., dreams).
But I suspect that part of the appeal of psionics is not that people want something better than magic, but, rather, that magic has essentially become internalized and approached by D&D as a science, and so people want some form of magic that actually feels more magical or subtle than what it has become in D&D's Disneyfied magic.
Huh. Having psionics be the less understood and science-y version of magic is 180 degrees from what I would expect. Did the whole fantasy vs sci-fi side discussion start because psionics was called more sci-fi than fantasy? Of course, you may be focusing on the more subtle, but that's not be a terribly strong hallmark of psionics in D&D, either, what with energy manipulation being a large component, or body manipulation. Harder to tell who's using it, maybe, but not really more subtle.
I think, fundamentally, psionics is just another power source for magic. We already have arcane and divine, so psionic is just another tag. While arcane and divine magic use the same system, the access to these is gated through different class structures that are different outside the leveraging of the same magic system. It's very unlikely that psionics will get a new magic system all it's own, so it'll have to share the magic system. Given this, what makes a psionist class distinct from the Cleric, Wizard, Sorcerer, or Bard? It cannot be, "they don't use components" alone. What's the thematic bits the make it distinct? Can that distinct core class support the subclasses that you think need to be included to cover the themes necessary to make psionic rich enough?
These are the basic design questions that need to be answered. Honestly, the 'no VSM' bit is too far down in the weeds to deal with until these larger questions get answered. And they need to be answered in a way that facilitiates design, which means they need to be reasonably specific and testable (as in, does this design feature meet the design goal?). Vague statements like 'more subtle' don't provide enough design guidance.
Personally, I'd like to see psionics avoid the full caster approach altogether and instead come at the issue with a core class that focuses on a limited set of powers/spells common to psionists and then building up of class abilities that allow for enchancing and empowering abilties. Subclasses would be where real diversity shows up, with additional powers/spells and additional subclass enhancement options selected to meet the theme.
Something like starting with some basic powers, maybe cantrip level, at will, and then a few 1st level spells that are thematic for psionists (good opportunity to add a new spell or two). Add to this the psi-die, usuable to add to attack damage or skills as in wild talent. Spellcasting requires components (don't panic), but non-expensive material components are replaced with a focus, like a totem or crystal. The focus glows when powers are used.
At second, gain the ability to roll your psi-die to enhance your powers, like removing V,S components (crystal still required), or increasing range/area/targets, or imposing disadvantage on saves... rough ideas, not balanced yet.
At third, gain speciality. You get some new powers that align to your speciality, like scorching ray/burning hands for a pyro-kinetic, thunderwave and shatter for a kinetist, detect thoughts and blindness/deafness for a telepath, etc. Again, need fleshing out, this is rough.
Then, as you level, the core class gets a few more general spell abilties, but at a slower/lower rate than full casters. Most of the new powers come through subclass improvements. The core class does add new ways to enhance/empower these powers, and, crucially, add more psi-die at useful intervals (maybe at proficiency breakpoints). This lets the psionist expend dice on being able to do more enchancements on only a few core powers, giving the flexibility and depth.
The core concept here is really class that uses a limited number of spells, but can enhance them frequently to match the power level of full casters, but in clear specialities. No generalist psionicists. Looking the above over, it needs a lot more fleshing out and a strong balance pass (anything allowing mix and match enhancements needs a careful tuning), but this is a class I'd be interested to see. Not because it's psionics, because I'm at best ambivalent about the concept, but because I think this is an interesting design and a kind of class I'd enjoy seeing in play, regardless of it's flavor. Ultimately, this last bit is the most important thing -- something that offers an interesting new way to interact with the existing ruleset, and not necessarily that it has a specific flavor.
