So to get back to my question: say that i have a chapter which explains the initiative rules, what kind of art would you use and why? (This works for any kind of rules in a rulebook)
Some sort of combat illustration, it's not like every rules concept needs an piece of artwork to match. But one or two pieces would spice up a combat chapter. Using the PHB as an example, the combat chapter opens with a full page illustration of a warrior fighting off multiple opponents at the top of some stairs. Initiative is three paragraphs on the opposite page, you have a similar amount of text for surprise, order of combat, your turn and bonus actions. This opposite page which is all text, use headers, and a boxed section to break it up.
The next double spread has three columns of text where a fourth would be is an illustration, I know to be a fire giant, next to the text about creature sizes. Following pages in the chapter no double page spread is without art on one of the two pages, usually a full column but right near the end (page 196-7) is mostly text with a small illustration of a potion, and pestle and mortar, right by the text on stabilizing a character, so again related to the text.
* No offense intended but the RPG readers are all educated persons and add a picture of a gnome along with its description to my eyes is a bit sad: it's like a primary book where you have Ape+picture, Bee+picture and so forth.
Well kids of that age haven't seen an Ape or maybe even a Bee, so an illustration makes a visual link with the word. I don't know about you but I've never seen a troll in person, but I've seen several different interpretations of one over the years, but what does D&D consider a troll to look like? It is a lot easier to use artwork to illustrate what one looks like, that then leaves word count to be used to describe things like tactics, it's habits, cultural information, etc.
What does D&D consider a gnome to look like? As the only gnomes I've seen IRL
look like this fella. Surely it makes sense to include an image so we are all sharing a similar imagined setting, not some people imagining something like Dobby from Harry Potter and others imaging the Smurfs?