Again, you're confusing ad hoc with improv. Improv is not just "make stuff up." If that were sufficient for improv, then we all already do it to a pretty good degree. Your opening question isn't something that makes me say "oh, yeah, that's improv" because it's present in every RPG game there is -- players having the choice to try something with their characters. This is the fundamental level of agency in RPGs -- it must be present. Citing it as "improv" rather than "what must be present for an RPG" is odd, especially in a thread that's concept is to encourage improv in games -- why encourage something we're defining as always present?! No, ad hoc, random, and procedures are not improv. Improv is where every player is building and not blocking in play. These block. That's not a bad thing, mind -- blocking is a very useful tool in a lot of RPGs. In D&D, it's essential for the GM to be able to block to enable the approach that D&D is built upon. But blocking is antithetical to improv. There are other games that do not block. Those typically are games that do a better job of supporting and encouraging improv.
And you get it wrong.
Improvisation can be free form, as you claim.
Or can follow parameters.
In a D&D context, improvisation always follow some parameters. Otherwise, you do not play D&D, you improvise.
When Robert Downey Jr improvise on a set and that impro is better than what was originally written, it is not free form improvisation but an improvisation which follow parameters such as a movie scenario, known character's behavior, history of the previous movies and so on. Robert is given parameters and he improvise on these. Sometimes there a flashes of genius, sometimes, it is not even taken into consideration. But he follows parameters. And within these parameters, almost anything can be possible. But Tony Stark would not all of a sudden be able to do magic out of nowhere. A non parameterized improvisation would allow exactly that.
To do improv in D&D is hard, mostly because the system is so dense that some preparation is required and that prep is often rather specific (these monsters, that trap, this map, etc). If play heads in an unepected direction that could use that kind of prep but it wasn't done, there are no tools in the D&D toolbox to bridge this.
Not at all. It is quite easy but requires a few things.
1) An extensive knowledge of monsters.
Flipping through the pages is bad. Right of the bat, I can play any monsters in the MM and be almost perfectly right. Memorization of the stats with a small margin for errors is more than enough. So far, I have read the MM about 20 times. (but I am a fast reader, I know not everyone has this advantage..)
2) An extensive knowledge of the rules.
Again, flipping pages is bad for improvisation. You need to know most rules. This one is simple enough to understand.
3) Know your players and their characters.
This one is not easy, the more players you have, the less it becomes possible. At 12 players, my memory can have some lapses but usually, I can easily play each character of every single players with ease. I sometimes, even remind them some of their equipment that they are forgetting. With that knowledge, you do not need to flip through pages of notes, you can easily stir the improvised events in the direction you want. This allows for complete improvisation on the adventure.
Even If I said the random dungeons and encounter table can be used. I do not always do so. I know my players and their tastes. I know what will "work" and what will not with them. One fact is to be considered though. When you have an open game with random players, this can't be done with any efficiency or consistency. An AL type game needs more restrain than a personal games with friends. It maybe why you say that improvisation is hard in D&D. When faced with random players, you simply can not do that.