Consider Charm Person as it appears in the current playtest:
The spell charms it target until the caster or a companion of the caster does anything harmful to it.
And a charmed creature cannot attack it charmer (and the charmer has advantage on social interaction with the charmed creature).
Put these together, and until the caster or a companion harms the target of Charm Person, it cannot attack the caster and the caster has advantage on social interaction with it.
This does not take a creature out of a fight unless it is the only creature, or unless the caster can - via social interaction - persuades it to abandon its friends/allies. A rogue could have the ability to do these things without magic, say by bluffing a creature into surrendering.
To take a step back, Charm Person is one of how many spells a wizard will have available to him (not the spell list, but the choices of spells he can choose to cast)? At first level, the Rogue having the equivalent of Charm Person might be a reasonable combat ability. That L1 wizard will likely only have a couple of offensive spells, and both may have no great combat choices if they run into the wrong opponent (here, a non-human), and be back to their weapons. However, it has to scale up with level - at L3, L5 and L12, that Wizard has a lot more options that a Charm Person and another minor offensive spell, and the rogue should have combat abilities that are similarly enhanced in power, effectiveness and/or versatility. Sneak Attack powers up without becoming more versatile, but it already works on most targets, certainly far more than Charm Person does.
So, short answer, I'd agree this is a combat ability, but not that it is sufficient, by itself, to replace Sneak Attack at all levels. Perhaps this could be a choice to replace 1d6 of Sneak Attack, but I'm not even certain of that.
My idea is that a rogue should be able to choose between:
a Sneak Attack that deals Xd6 damage
a Defensive Roll that reduces Xd6 damage
* A Defensive Roll that reduces Xd6 damage: This isn't very useful unless the rogue is otherwise capable. It is nice if there is a macguffin for the rogue to steal and run off with, but great defenses without offense only works as long as anybody cares if you are there.
I think the defense is a good ability, but only if we play in genre - opponents still try to attack the rogue, who can strike back with his very limited weapon damage, but he ties up an opponent. If, however, the PC's and/or NPC's just say "he can reduce our damage, so just ignore him and attack the other guys", then the ability loses its value. This could be effected mechanically - the Rogue also gets an ability to taunt the opponent(s) to be affected by the reduced damage, and those enemies are either forced to attack the rogue, or are markedly penalized if they do not.
a Henchman who has Xd12+X HP
Workable - seems like a reskinned animal companion, and needs to be comparable in power to such a companion. Perhaps the Henchman should have X character levels, with some choices for the Rogue to select from (nothing wrong with a Cleric or Wizard, or even an apprentice Rogue, henchman). But then it is becoming much more a "second PC" on the board.
an Explosive that deals Xd6 fire damage
Again, workable (less so if one adopts the Pathfinder alchemist class, but I see no indication D&D will do so). It doesn't have to be fire - concussion or shrapnel damage would work fine. The fluff could be an issue (what stops him handing these out to teammates, for example), but it seems a very close replacement for sneak attack.
Switch a combat ability for exploration ability or social feature with the "power level" of Sneak attack would make if harder for DMs and designers to make good assumptions for their adventures
Agreed.
Adding some of my own. Again, not deeply thought through.
* Ability to daze enemies for a round, Save DC 10 + Cha +X, where you can spread X out over several targets if you want. Some noncombat utility for chases and the like.
* Ability to create entanglement in an area for a round (caltrops/overturning carts, as appropriate)
These all seem reasonable. I come back to the need for these abilities to scale with level, or to be taken as only a partial replacement, perhaps with some abilities being available in lieu of +1d6 Sneak Attack (so if Sneak Attack starts at 1d6 and works up to 7d6, a Rogue could instead have 7 social-flavoured abilities like the Charm Person and Daze options by top level, usable independently or in tandem, or seven environmental abilities like the entanglement one. Some of these abilities could be specific growth to earlier choices (like, say, expanding that Charm Person to a broader target group, either more than one person, or non-person targets), or be level-restricted (eg. can't be taken before 5th level, likely matching a level where Sneak Attack would get an extra d6).
These start to look like Rogue Talents/Special Abilities - did those flow through to Next? I get the sense that Next has worked to reduce spellcasters to a power level more comparable to martial characters, contrasted with Pathfinder's addition of abilities to the martial characters with less or no increases to spellcasters, which may become a significant differentiator between the two systems.