As a minor aside, RPG testing and iteration isn't solely about how well its mechanisms function, it's also how well those mechanisms give rise to fiction of the desired kind.
I'm not feeling the force of this contrast.
Giving rise to fiction of the desired kind is the central function of a RPG's mechanisms. Not something that is different from the mechanisms' functions.
The experience of choosing which mechanisms to include, which to retune, and which to cut feels authorial when instigating the design, and editorial when finalizing it. That's if editing includes preparing a text (in the broad sense) for consumption or performance.
Part of preparing a text for consumption is settling on the page size, the margin width etc. This is editorial only in some loose sense. Choosing which ink is best for a given paper is also part of that preparation, and I don't think that's editorial at all.
In the context of a RPG rulebook, choosing which rules to include is editorial in the sense that
it is a decision about what to include in a published work. But the process of winnowing out by testing is not editorial.
The analogy would be a cook book: deciding which recipes to include, and how exactly to present those recipes (eg with or without metric/non-metric conversions), is editorial. But the process of testing the recipes to see which ones are worth including is not editing. It's cooking.
Suppose that James Watt wrote an instruction manual for building a steam engine. In writing the bit about how to prepare the leather for the seals, a decision has to be made about what to assume the reader will know and do, and what to leave out: that's editing. But the process of working out what systems of seals will work is not editing: it's engineering.
RPGs create a bit more room for confusion because not only do they resemble recipe books and instruction manuals, including written descriptions of rules/processes for doing a thing; but the thing that one does by following those rules and processes is the creation of a fiction. But I think it's better to avoid this confusion than lean into it!
An example could be if an early version of an Oracle contained a prompt to "Writhe" and it was felt that writhing would be jarring to the intended conversation. I suspect that the Stonetop discord channel is replete with further examples, as Strandberg proposes, playtests, accepts feedback upon, and revises the rules etc.
In that light, I'm not sure I entirely agree with the point about "not the subject of extensive rehearsal". It is only that they're not rehearsed by the group at the table. They are rehearsed by groups from whom conclusions are drawn about how they will most likely play out at the table. Blades in the Dark works as well as it does because Harper was able to extensively playtest it (a sort of rehearsing on behalf of). It wouldn't be surprising if there were lessons to draw from theory around editing for other media, just as there have been lessons to draw from theory around narratives presented via those other media.
I also prefer to maintain clarity of terms, rather than blur their use.
A rehearsal is when a performer, or a group of performers, practices a thing - a play, a recitation, a musical piece, etc - to get better at performing it. A composer working on their piece, and on their score, perfecting each, and also perfecting the relationship between the two, is not rehearsing. They are composing. Suppose they call in some musicians to perform the piece, or some part of it, to help work on both the composition (and its sound) and the score (and its adequacy in conveying the piece to a musician): that is still not rehearsing. It's part of the process of composing.
A group of people play-testing a RPG rule to see what sort of fiction it tends to produce, and then using that experience to improve the rule and/or the written presentation of the rule, is not rehearsing. The group are designing and/or developing a game's rules.
The way they use the rules in this process might also give some insights into better ways of writing those rules, and we could call those insights, if we speak broadly,
editorial insights. Vincent Baker gives an example in the AW rulebook, where he says that he wanted to start with 4 dots for improvement, with the fifth "dot" triggering an erasing of the 4 filled dots and earning an improvement, but this didn't work: players would treat the 4 dots, when filled, as complete, and so he had to introduce a fifth dot to fill in even though, as soon as it is filled in, all five dots are erased as the improvement is earned. But this is not any sort of editing of the fiction that the play of the game will produce; it's all about improving the communicative qualities of the game rules, which are a set of instructions.
These (which I do not have any strong disagreement with) would fit with the ludologist's dislike of presupposing critical methods for traditional forms of narrative will say anything useful about games. However, there have been arguments put about narrative structure (development of themes, rising tension, etc.) that do seem to express or rely on a familial relationship between RPG and other kinds of performed fiction.
Cues and other signifiers (rules, descriptions, perhaps parameterizations) are expected to lead to performances that are recognizably of a kind. There's that superficial likeness to a script, which is also expected to lead to performances that are recognizably of a kind. The difference seems to be spontaneity as you said, including freedom of interpretation and arrangement.
The development of theme, rising tension, etc, is a property of
the fiction that is created in the play of a RPG. It is not a property of the cues that are used in the course of that play,
except for the special case (that I already mentioned) of a pre-plotted adventure where the play of the game consists in the particular group, in their play, reproducing the pre-authored fiction of the adventure.
Even in that special case, we have to make some careful distinctions. The way a pre-plotted adventure encodes and presents a fiction with theme, rising action etc is different from how (say) a novel does, and from how (say) a script does, because it is written to be used by the GM of a group of RPGers to lead the other participants through the fiction. A novel, if well-written by mainstream standards, will draw the reader into the experience of the theme, the rising action, etc. A pre-plotted adventure module is not likely to produce that same sort of experience just by being read though. Even more than a script, it is likely to need to be experienced in play to actually generate those narrative phenomena.
Once we turn away from the special case to RPG rulebooks more generally, we wouldn't expect to, and we won't, find theme, rising action etc in the books themselves, any more than you will literally find good things to eat in a recipe book.