In my opinion, the amount of adventures going to the end-level is reduced because the target market for adventures is even more low-level oriented than the general market. Most "new" DM will want to stick to published adventures for their first few tries, because they are learning the ropes of DM'ing. Once DMs are experienced, they can either stay in the market (because they lack the time or inclination to build an adventure from scratch) while a portion of the DMs will homebrew their settings and adventures. So, the market for high-level adventures is more reduced than the market for adventures in general. Plus, new DM will probably be happy to start their campaign with level 1 PCs 'cause they are more manageable.
Along the same lines, once you reach high level (even if 5e doesn't become as ludicrous as 3.5 could) the PCs are much more varied in their abilities and I think designing published adventure is more difficult as you need to take into account many more variables; so high-level play needs more prep time for GMs, hence reduced usefulness of the published materials. As other have mentioned earlier, high-level charcters more often drive the action than low-level characters (because the latter lacks the mean to significantly influence the world... a 3rd level party must obey the laws of Cormyr, and 18th level party could reasonably decide a regime change in Cormyr is in order instead of following rules against paying with foreign gold pieces... and declare this goal in the middle of an adventure), hence less need for high level adventure.
However, this increases, and not reduces the NEED for high level adventure. So why are they published? My take is that even if the campaign peters out due to real life constraint, the appeal of going after a big bad you love to hate is great. And you can't really care about a recurring villain you've only seen once or twice, so the campaign need to develop over some levels. And it's not a total waste of money on the DM's part because more often than not, those adventures are salvageable for parts. In my experience of a campaign so far around level 16, it's difficult to find balanced encounters: players tend to wallop over opposition at these levels; so it's quite possible that the higher-tier adventures could be used with lower level PCs with minimal fuss... which campaign in D&D assumes that the characters have access to planeshift? More often than not, they include some elements to run an interesting trip and include a sidebar saying that probably the party will just teleport over... So, to summarize, my opinion is that including the higher-tier makes the adventure more compelling (so the DM decides to run it) and said DM doesn't think he spent money badly by buying the whole AP, so, there is a target market, a bigger market than just the DMs looking for something to play as is.
I don't take small spell lists as a sign of lack of development in character class, because at the higher end, the rules to design your own spells should probably be used... Those Leomund and Bigby spells are enticing players to come up with their own signature spells -- this is wizard-centric, i know.