First, this isn't a parody. I honestly want to understand what you would do. I still don't get why there couldn't be a dialog where the PCs are questioning the shopkeeper but since you refuse to give an example of what the dialog would look like I give up.
Sigh, you keep saying this, but then you do not listen and instead insert your imaginings in place, like saying you don't understand why there couldn't be a dialog. Where have I, or anyone else, said there's never a dialog? This is your assumption, and it's wrong.
I'm not interested in imagining a dialog between players and me in my game. I felt yours was fine up until the point I said 'full stop' because it reasonably looked like something that could happen in my game up until that point. It, in fact, at that point that it became farce.
I don't want to put words into your mouth but since you refuse to give a concrete example, I'm assuming something like:
I've given plenty of concrete examples. Are you demanding that it be in the form of dialog now? How does that help?
DM/shopkeeper "So I locked up the store as usual, set the normal traps and went upstairs to bed."
Player: "You sleep above the shop?"
DM/shopkeeper: "Yes, it's part of the compensation, and I'm single so it works well for me."
Player: "So no witness and you didn't hear anything at all during the night."
DM/shopkeeper: "No witnesses and no I didn't hear anything. But this building is very solidly built for a reason."
Player: "And there was no sign of forced entry, the traps were still set."
DM/shopkeeper: "That's right. In fact the traps are supposed to ward against magical entry as well."
Player: "I don't believe him, I think he's hiding something."
DM: "He's telling the truth"
This to me would ruin all the mystery of a who-dunnit like this. No thanks. The shopkeeper
should be a primary suspect. The reason to ask for an insight check is to maintain that air of mystery and doubt. The shopkeeper is less likely to be involved, but there's no way to be certain.
It's not what would happen at my table at all, though, which is why I put a full stop in above and explained what needed to happen. Here, the issue for me is that you've decided that the shopkeeper being a suspect is important. That's great! If I did that, there would be all kinds of things going on that led to that, and it wouldn't be 'is he lying.' To me, that would be very boring. If I flat out told the players the shopkeeper isn't lying it would be because it wasn't interesting to anyone at the table except idly. Again, the issue here is that the construct you've built wouldn't happen at all in my game. If I ran this, then the shopkeeper would have many things going on that would not, at all, pivot on if the players think he's lying.
And, this really touches on another point of play -- what does a successful check actually mean in this case? That the player confirms their guess and absolutely knows the shopkeeper is lying. What does this enable that isn't also enabled by the suspicion of lying?
And please don't bother responding if all you're going give me is more platitudes of "I would be such an amazing DM this could never occur" unless you can show how. This is an extremely simple scenario that is very typical of games I've played in or run. It shouldn't be hard. Assume it's a new player and this is the beginning of the session.
Because it's not an scenario that exists in my games, no matter how "awesome" I may or may not be. As I say above, if I ran this scenario, the shopkeeper would have lots of interesting detail, but lying might not be one. This doesn't mean there wouldn't be lots to engage with. Perhaps the shopkeeper's brother used his familiarity to rob the store without the shopkeeper's knowledge, but the shopkeeper suspects, that's a neat thing that would come out in the interaction and could be sussed. Using Insight to discover the shopkeeper's ideal for honesty could help, or maybe a flaw that they always excuse their brother? Lot's of neat things I might do with a heist scenario involving a live-in shopkeep that do not involve using insight to tell if he's not lying.
Well let's talk about three possibilities with my jewel heist scenario.
1) The shopkeeper is really telling the truth and is not particularly nervous or agitated.
This means that the shopkeeper is just ambulatory exposition, and I'd not even bother with this. Or, the issue wouldn't be the shopkeep lying, but some other thing involving the shopkeep. If the sole challenge is "do NPCs lie to us" and the shopkeeper doesn't, then why bother?
I mean, I know why, it's to preserve a sense of uncertainty in the game. I've found that I can straight out give my notes to players (of all calibers), and they'll still create their own uncertainty and will definitely screw up by the numbers. This is why consequences for failures are so critical to my style -- they create roadblocks and adversity without me having to spin the players along with uncertainty that what they rolled means anything at all. My choices to do it this way are my choices, I certainly had a lot of fun in the past playing the other way, I just have less fun now. YMMV, and all that.
2) The shopkeeper is the thief but is not any good at deception.
This is much more interesting. At this point, that the shopkeep is lying is pretty much a foregone conclusion, so I'm not going to make that a crux point -- ie, the shopkeeper getting away with lying will not be that happens if the players investigate. The 'mystery' will be why, to what ends, and what does this mean to the PCs and their goals. Again, I'm not going to have a situation in my game where a store was robbed and the shopkeeper did it as the whole of anything -- this example would be just a front to a larger issue and the shopkeeper lying would not be the challenge.
3) The shopkeeper is the jewel thief but he's really good at deception. Good enough that a passive insight isn't going to catch the deception (which I would handle as an automatic success and let the player know).
Again, interesting. But, as I've said, I'm not going to hinge an investigation on an Insight check to tell if the shopkeep is lying. Insight to get that the shopkeep is smug and unconcerned, maybe, because that would push players to look more closely at other things. And, really, the other things would be what solves this mystery, not an insight check.
And, this leads me back to something I said above: what's the point of the insight check? What does it actually do?
Let's assume we're in #3 above, and the players succeed at a very high Insight check to tell that the shopkeep is lying about the theft. What does this do in your game? Is this proof, of any kind, that allows players to arrest/kill the shopkeeper as a criminal? Normally, this would obviously be false, but I think this is actually what the check does -- it enables the players to have the moral right to punish the wrongdoer. It, in many ways, functions like the old Paladin Detect Evil, which was sometimes used as a crutch to enable killing anything that pinged. I'm not suggesting this is the same, here, because that's extreme, but I do think a successful Insight check to detect lying is really just a gate that provides justification to escalate within the scene. I do not enjoy this for a couple of reasons. One, I don't see that I need to validate your character's thinking so that they can act upon it -- that's up to the players. Two, I don't find that interesting. It's a tell to the player that this person is bad in a game where bad things are usually killed, but many GMs might also use that as a means to then take the expected player action and pervert it by having them get in trouble with the authorities. And, if the authorities are important such that real proof is needed, then why am I validating player suspicions in a pass/fail way if it really doesn't make a difference? This is one big reason why I do not use Insight as a lie detector, nor do I create situations where telling if someone is lying by talking to them is a crucial point of the game.
Since you've asked for examples, let me present a shopkeeper example I might run:
Initial PC information:
The shop has been burgled. There are numerous mundane and magical traps that have neither been disabled nor set off. The shopkeeper lives in the store and was present all night long.
Initial DM information:
The shopkeep didn't have anything directly to do with the burglary. He didn't hear anything, and didn't see anything. He does, however, do occasional fencing work for the local thieves guild, and so is guarded when answering questions. He suspects that his brother may have been involved, and, indeed, he was. The shopkeep's brother is a wanted jewel thief, but uses a different name so it's not obvious. The shopkeeper has the following BITFs: Trait: guarded with authorities and those representing them; Bond: will do anything for his daughters; Flaw: I've always made excuses for my brother's behavior.
There now I'm done with prep. In play, this can go a huge number of ways. If the players inspect the traps, they might discover evidence that they've been disabled and expertly reset. Pointing this out to the shopkeeper would result in the shopkeeper being surprised and then suspicious and pointing out that he'd not have needed to do it that way as he has a much easier way to disable and reset the traps that wouldn't leave that evidence. Further questioning on this line wouldn't result in much, as the shopkeeper has no reason to reveal anything else. The party might then look at what wasn't stolen, and could find out that some of the left merchandise was previously stolen and confront the shopkeeper about his fencing. Having a party member affiliated with the guild would make this go very easily, and thus the connection with the guild would be exposed, indicating that it's not likely a guild job and thus likely an outsider. Looking around the shopkeeper's apartment would uncover that he has daughters he's supporting (a letter, or painting, or kid's drawings, etc.). Using this, and asking how he could possibly continue to support his daughters if the party doesn't recover the merchandise could result in the shopkeeper revealing his jewel-thief brother.
Or, something totally different might happen, depending on the player's stated goals and approaches to those goals. I do not ever write solutions, I write problems and then enjoy seeing what happens in the game.
To give you another example that actually happened in my game, here's some of that dialogue you asked for:
Scene: the dwarven barbarian and his cleric companion have entered a seedy bar seeking information about the person that murdered the dwarf's entire clan, leaving him the only survivor. The dwarf doesn't know why this happened, only that it did and the person responsible is named Terak. The dwarf has information that Terak has been in the city recently and is searching for him. They enter the bar looking for Butcher, who, in a previous scene, they were told might have information. The bar is dimly light, close, and full of regulars at the end of a hard workday (picture a coal-miner's bar and you're not far off). They immediately notice Butcher, who is immense, occupying two bar stools, heavily muscled and overweight, with bright red skin, a single curling horn, and is wearing a butchers apron with a massive cleaver on his belt. The duo approach and sit at the bar next to Butcher.
As they sit:
Butcher: [without looking] what do you want.
Cleric: who says we want something?
B: you don't belong here, you don't want to be here [turns to look at the cleric], and you sat next to me. That means you want something. [turns back to his pitcher sized mug] So, what is it?
Dwarf: We want to know where Terak is.
B: [visibly pales, starts, tries to cover it] Why would you want to know that?
C: So, you know who he is?
B: [winces] I might.
Dwarf's player: Okay, I'm going to activate my smoking armor and my glowing eyes helmet and then threaten him.
D: [now with smoking armor and glowing red eyes] You'll tell us or else!
Me, as GM: Okay, cool, make an intimidate check. You don't get advantage for your gear because this is Butcher's stomping grounds and you've notice quite a few thugs around paying attention, so you're pretty sure he'll have some support if you start something.
D's Player: sounds fair, I got a... 4. Darn it.
Butcher: [glancing at the dwarf before turning back to his ale] That's a cute trick. Why don't you get out of here before something bad happens you'll regret. {I'm attempting to close the scene here.}
Cleric's Player: I'm going to look around and see if I can notice anything that might help fix this. The butcher has friends here, clearly, anything about them jump out?
Me: Sure, you look around and notice that there's a plaque opposite Butcher's spot at the bar that says, "In appreciation of Butcher and the support he provided after the Foundry accident. The families of the Foundry thank him for his generous assistance in their time of need."
Cleric's Player: Oh. So, he helps around the community, I guess. I'm going to try to leverage that.
Cleric: Please forgive my friend, Mr. Butcher, he's lost his entire clan to this Terak. I'm sure you understand how hard it can be to suffer a tragedy like that and try to do anything to help. Are you sure there's nothing you can tell us to help us stop this Terak from destroying another community like my friends'?
Me: Ooh, super nice. He's pretty pissed though, so make a persuasion attempt but definitely get advantage on it.
Cleric's Player: An 18!
Butcher: [deep sigh, looks at the plaque] Yeah, I get it. [turns to the dwarf] I'm sorry that happened to you. This person you're seeking is very bad news, and I'm sorry I ever got mixed up in anything with him, even if it was just supplying rare meats to his household. He's put those shipments on hold indefinitely, and I have it on good authority he's left town, and I don't know where, sorry.
Dwarf: Is there anything you could tell us that might give a clue as to where he's gone?
Butcher: I overheard a conversation I wasn't supposed to between some of his goons when I made my last discovery. They said that their boss was looking for an artifact called the Great Wheel and had just gotten a lead on a piece of it so they were packing up to go. All I know, I tried to not learn anything at all. I suggest you grieve for your lost and forget about it.
D & C: Okay, thanks, have a good day.
{scene}
So, in there, there were a bunch of possible checks. A few automatic successes based on approach (the mood in the bar, noticing the plaque), and two that had checks called for: the intimidate attempt and the persuasion one. The intimidate failed, and I moved to end the scene with no success to the overall goal of finding information on Terak. The persuasion succeeded, because it leveraged a bond, and so reversed the move to close the scene (it's intent) and also automatically succeeded at eliciting further information. The dwarven barbarian has this Terak as a central conflict in their backstory, so I'm leveraging that to provide hooks. This whole scene occurred because of a previous hook I had set, and now the players are driving this plotline, not me. I don't know where Terak is, but I know he'll show up, likely at the worst time. We're gonna play some more and find out. For instance, right after this another member of the party found information on a possible location of another piece of the Wheel, so they've cowboyed for a hop to the near Astral and a fragment of what was once a Primordial Cult temple from before current recorded histories in case that's the piece that Terak's after as well. I wonder if it is...