Okay bear with me here, because this is going somewhere.
D&D is NOT a good system for this kind of game. Some of the reasons for this are listed above, but to reiterate, almost all of the abilities in D&D are provided to bypass, overwhelm, or convert challenges to the players' benefit. This means that the system, as a whole, is focused on the PCs defeating a challenge. The result of this is that the resolution system generates binary results and, outside of combat, non-granular results. It's not well suited to social intrigue, politics, negotiations, etc., that all will feature prominently in the style of game you're proposing. D&D does direct challenges well to okay-ish, and is very bad at the kind of murky, maybe games that go with a gritty crime game.
So, the short answer to this is try a different system. Blades in the Dark pretty much nails everything you list in the OP -- it was built around those concepts to do only those concepts, and it excels at them. Or FATE, which can do a good job of that kind of game as well. But, I'm going to guess that this isn't an option, so let's steal a few things from Blades that might work okay in D&D.
Foremost -- Factions. Build up a number of factions in the city, but leave them as thumbnail sketches only -- a distinguishing trait, a goal (maybe), a stomping ground. Treat everything in the city as a faction -- the noble houses, the guard, the military, the watch, the thieves' guilds, the merchant guilds, the urchins, the beggars, the creepy old ladies in that one house. Treat all of these them same. Assign them a tier (I'd go with the I, II, III, IV tiers already in 5e) to represent their relative power. Then, when you start the game, go ahead and have your players tell you why one of those factions likes them and why one of those factions doesn't like them. This immediately creates tension in the system and lets you start pushing and pulling the PCs to interact with the various factions. Whenever the PCs interact with a faction, have that interaction change the opinion of the faction. You can track the favor of a faction using the factions rules (clunky) or as if they're a NPC and the PCs are trying to influence their opinions (so, friendly, neutral, unfriendly, hostile, that stuff). If you keep a sheet listing the faction names and a quick mark for favor, you can prep quickly and dynamically as play goes.
Remember those tiers you assigned? Use those to set the DCs and CRs for that faction. A Tier I faction is low tier CRs and low DCs as a base -- DC 10 should be what you use for anything in Tier I. Tier II is DCs 10-15. III 15-20. IV is 20+. Same with the CRs of any combatants -- the higher the Tier, the tougher the mooks, leaders, and personalities. This way the city will "gate" itself in play as access to or competition with a faction will be heavily influenced by the tier of the faction.
I would avoid trying to map out a plot for a city game. Too many variables. Let the plot develop naturally, picking up ques from the play as the party angers some factions and pleases others. Remember to 'hold on lightly' to your plans, in case the players throw you a loop.
You will still find that the system will not do a good job of supporting the kind of play you may want, but you can maybe bash it into shape. I'd strongly suggest reading up on using skill challenges in 5e (especially the dynamic fiction nature of skill challenges) as this can be a good tool to reduce how the system treats complex social interaction or complex infiltration/theft challenges.