Other than perhaps degree of organization, once the houserules become extensive enough what's the difference?
A playtest document is designed to be distributed for easy, rapid testing. It is barebones, focused on efficiently communicating the mechanics in question, in a way that helps the user quickly apply said rules, and actually
test them. Meaning, it explains to some extent the purpose/intent of the rules provided, especially if they have changed relative to a previous playtest document, and it specifically puts the reader in an analytical mindset relative to its content.
A playtest document is--or should be--a serious thing, presented carefully in such a way as to highlight that which is being tested, and to make it as painless as possible to rapidly perform such tests and provide detailed, specific feedback.
A house-rules document is simply record-keeping.
It's the difference between schematics and illustrations. Schematics do contain visual depictions (in most cases, anyway), but not vice-versa.
Take the system I use. It's fully written up (except for many of the monsters), sort of vaguely organized, and consists almost entirely of tweaks, kitbashes, and houserules on top of the 1e chassis. The only thing that perhaps prevents it from being a "playtest document" is that it's too tightly tied to the specific setting I use; doing a more generic version is very much on the docket, but don't hold yer breath as it's a bit down the priority list.
If I may use an analogy?
This is "Grandma's annotated cookbook". It is not an experimental procedure being shared so that the procedure can be refined later. Grandma's annotated cookbook is a beautiful thing and, in general, very usage-focused--but it is not for
testing purposes.
Oh, I've got an even better idea. Imagine an engineering company working on reviewing all sorts of aspects about cars-in-general to find ways to make them cheaper, safer, more fuel-efficient, less resource-intensive, etc. You don't know what things about the car need to change, so you make it extremely modular, allowing you to swap engines, transmissions/drive-trains, brakes, wheels, exhausts, windows, whatever. All of these things can be easily popped in or out, specifically because the engineers have designed them to work that way. They create various design models with different elements, and provide them to drive-testers to analyze and report back about. As bits get refined and streamlined, they begin sticking to specific parts again and again, but the inherent hot-swappable modularity remains in this model, which would not be present in the finished product.
Now imagine you are a hobbyist muscle car enthusiast who tinkers in your garage. You've re-built and re-imagined a genuine classic car--say a '65 Shelby Mustang GT350--with top-flight, hand-machined parts, improving nearly everything about the vehicle except the upholstery. Your vehicle would not tell an outsider a damned thing about what makes a Mustang "tick", about how to
do Mustang-type cars differently. It's simply designed to be a real damn fun ride, for those who are looking for that kind of ride. It has no
testing value, and if you gave it to someone else to drive, they might pick up an idea or two they like to take back to their own muscle car in their garage...but they'd have to hand-machine it all over again, just like you did.
A playtest document is one specific "engineers' modular car". A house-rule document, even if made neat and pretty, is one hobbyist's muscle-car rebuild. The former is an experimental platform designed for testing purposes. The latter is a cool object to view (or, in this case, to use). And neither one of them is a production-model car (=manuscript ready for publication).