This alone would not address the issue. I mean, it might help, but keep in mind that half of those things are still spells anyway, either directly or indirectly. So we would have just ended up with half these "spells" instead being "spell-like abilities", which are just spells with more steps.
For me, I think we just need to have a design which understands that bespoke abilities are not bad in and of themselves. That doesn't mean we should let them proliferate without use. We shouldn't. They should be for a purpose. But the way the 5e design team treats them, you'd think they actively wanted to make EVERYTHING a literal actual spell, even things that aren't even magical at all, get turned into spells. I just...I don't get it.
Doubly so when "everyone is a spellcaster" has been a major, major complaint about game design in the past.
But, more importantly, the way you make species design better is not (strictly speaking) by enforcing the above categories. It's by making it so that each species has active abilities that make them distinct. Consider that elves have had the whole "trance instead of sleep" thing for ages, but it never comes up as the thing folks love about elves. That's a really really big clue! Passive abilities have their place, and shouldn't be totally excised any more than bespoke abilities should. But passive abilities aren't what makes something cool, fun, or memorable. Active abilities are what do that.
"High Elves" (I still prefer the term "Eladrin") have a teleport. Dragonborn have a breath attack. Tieflings have Hellish Rebuke (which should not be a spell, sorry Warlocks, it just shouldn't). Orcs have their blood rage. Etc. Tap into that sort of stuff, and you'll have people beating a path to your door. Because you've made it so that each species feels different to play, because they actually DO different things, rather than merely having slightly different numbers.
This comment in particular helped me to organize my thoughts and gave me some motivation and inspiration, so hats off to you, EzekielRaiden!
Giving Tieflings Hellish Rebuke (as a core ability, not a spell), while an absolutely amazing idea, does require a bit of tweaking.
See, Hellish Rebuke is alright for the 1st level Warlock spell it is, 2d10 damage that uses your reaction, scales fairly well (which synergizes well with its original class), a good target range of 60 feet and not being restricted to only melee attacks like Armor of Agathys (although it gives out scaling flat damage in a game that relies on dice, so the trade-off's fair), so it seems fine to just plug it in as is, right? However, that's only because it's a
spell.
See, with spells, the intent is to create an
option. Something that you
could choose, but aren't
expected or
forced to. Hellish Rebuke is a damage option, plain and simple. It gives you a way as a spellcaster to use your reaction to deal damage, potentially freeing up your action to cast other spells (or Eldritch Blast). But if you desire other spells for utility, like the aforementioned Armor of Agathys or Hex, you don't have to take Hellish Rebuke. However, given that this is now going to be a multi-use ability that won't require a spell slot, or have a one off use like it was originally, and how it's going to become the signature feature of Tieflings, it simultaneously needs to be more than just damage while also needing to be toned down due to lacking the resource cost of a spell slot to balance out some of its qualities.
Starting off, the range should probably be shortened, 30 feet seems fine, and the damage will reduced to 1d10 to start with. We'll go with cantrip scaling, 2d10 at 5th level, 3d10 at 11th level, and 4d10 at 17th level. The damage type dealt will be determined by your Fiendish Legacy.
Next, I believe that you should be able to target enemies that attack your allies as well as yourself. Our range is 30 feet already so why not share the rebuking to our allies as well? If this isn't given right out the gate, then 3rd level is fine for this improvement.
Finally, while I think that each Legacy should get a rider effect for Hellish Rebuke (to give them all the same ability, but different ways of using it), I think those should probably come in at 3rd level or 5th level as a balancing measure. Thus, I think Hellish Rebuke needs one additional effect attached to it, just to give it a little more bang for the buck.
Some ideas I have are:
- Giving the target of Hellish Rebuke a penalty to attack rolls and/or saving throws until the start/end of your next turn (either 1d4/1d6 or your proficiency bonus).
- Adding life steal to Hellish Rebuke, allowing you to recover hit points equal to the total (maybe half until 3rd/5th level?) damage dealt to the target.
- Lowering the damage dealt by the triggering attack by either a d8 or by your proficiency bonus.
- A damage over time effect (someone said this was unbalanced so probably not going with this one, but I thought I should throw it out anyways).
Something along the lines of debuffing the attacker is what I think it should be. And for Variant Tieflings, the effect would change depending on several fiends you could be descended from, like a Chain Devil option grappling or restricting the target, or a Succubus option employing the life steal should that not be the effect given to base Tiefling. (If it is, a Charm effect would also work here.)
Perhaps there could be several options that you choose from to make it a kind of toolbox ability, like Dhampir's Vampiric Bite (side note, I
really like that feature and how it's designed. It's flavorful, gives you several options that can be useful, and it expanded upon the idea of species having natural weapons and gave them a purpose beyond being a better Unarmed Strike).
Also, and this is just a concept so far, I think there should be a more active use of Hellish Rebuke given that it uses a reaction and you have to wait until an enemy's turn to use it. Maybe expend a use of it as a bonus action to give the rider effect exclusively to a target? Or expend a use of it to recharge another potential feature for the species? Something worth thinking about.
I would add more damage types here, but I think everyone knows what I am talking about. 
