To answer the title question: If you are accurately characterizing his behavior clearly the Ki-Rin/Artificer player (I'm not 100% clear if that is your brother) is a problem player. I mean we have "shouting" at you for "half an hour", having a "tantrum" over a clearly broken magical item, and complaining about his power level when he is 13th level and hybridized with a magical creature, which is more power than most 5e D&D characters will ever see. He also seems to have spent a round in an obviously critical fight complaining melodramatically rather than doing anything productive, which I wouldn't care about as a DM but which I would being pretty irked by as a fellow player.
This doesn't mean that you can't play with him, that he can never get better, or that he doesn't have redeeming qualities as a player. It also doesn't mean nothing is your fault here. But if you are characterizing things reasonably accurately then sure, we have a "problem player", whatever that is.
The Artfificer moaned that with his passive he would have been able to see them(didn't actually say if I had rolled stealth), but I thought that since these devils had the sole purpose of getting the jump on someone, they would know how to hide from people. Is that my fault for making a wrong call? Naturally I'm biased on it., and would love to hear your opinions.
The proper "spirit of 5e" approach is generally to roll stealth. Rolling with advantage and/or proficiency they don't otherwise have might be appropriate if this stealth mission is their whole deal as you seem to say, but best practice is still generally to roll and maybe they fail miserably. That said, just deciding on a fixed DC to detect enemies on a specific location is also acceptable. Regardless of what you did or didn't do, I wouldn't put up with a player whining about their passive perception not being respected unless they had made specific investments in it like the Alert feat or getting expertise in perception.
How do I solve the definitely deadly encounter they are in.
Unless you feel like you railroaded them into it, it's not your responsibility to save the party. Sometimes D&D characters lose fights. It may be time to end this campaign which clearly has a variety of issues. An unplanned TPK is never going to be the satisfying way to do that, but it is sometimes the one necessary to emphasize that D&D takes place in a world of danger and consequences.
If you want to make it go on I recommend just having the enemy capture them. If you want to do this you may want to have more enemy minions arrive to induce them to just surrender or to ensure that they all get taken down before one or more can fail too many saving throws.