Think of yourself now, playing rules from back then. When we played, we didn't have such questions: were we constructing an SP experience now, we likely would! Players, as well as games, have evolved.
Back then however, Basic was not limited to the dungeon for us; not least because of all those pages in B2 on interacting with NPCs in the Keep, and then very soon after X1 took us out of the dungeon for extended hexploration.
I've read several posters position SP as addressing D&D as game. If your representation is right, that's not narrow enough: it's addressing D&D as boardgame. For that, 5e using the Fantasy Grounds VTT has been the tightest version that I have experienced. We didn't play D&D as a boardgame back then. No one I knew did.
What we found in the published adventures - and our own interests - moved the rules immediately out of D&D-as-boardgame. I suppose there must have been some niches in which players adopted what they saw as SP and stayed inside the dungeon. Outside of organised tournaments, I never once encountered them.
That doesn't diminish SP as a concept. It does argue against Basic D&D being equivalent to SP. It's silent on Basic D&D having the strongest valency to SP.
I follow your line of reasoning. Typically, the fewer and simpler rules in a game the more tightly defined the play can be. What I question though is the premise that tightly defined play == SP. So long as the context is gameful, a more generous rule set creates a greater abundance of opportunity for SP.
You - and maybe
@Ovinomancer - seem to be suggesting that fewer rules and tighter process affords greater SP. I believe that more sophisticated rules covering more ground afford greater SP. I suppose that is in part because when I think about an SP-axis of play, I noticed it more in 3rd-edition onward. If I had to pick one version of D&D that most afforded SP, it would be 4e. Have you played much using a structured VTT like FG?
EDIT Further - as I added in my response to
@Ovinomancer above - I am denying the line you draw between tightly defined and SP. In part because if that line exists, the opening question has no pertinence. If SP can only occur in D&D as boardgame, then there can be no tension between story and SP. Those modes inhabit different contexts. It is only to the extent that you allow them to share a context that the question in the OP can have any value.