The only time your list would ever be less than maximum is when you have just leveled up and haven't prepared spells since. Nothing "unprepares" a spell except replacing it in the list.
So you agree then! You can have a list with less spells than level plus mod! (j/k - you clearly don't agree in principle)
The rules only specify that you
can prepare Level plus ability mod (L+A) number of spells. I think everyone can agree that you can't prepare more. This is an upper (soft - because domain spells etc) limit to the number of spells on the prepared list. But the rules for changing the list are rules for changing the list, not changing spells. In other words, List A versus List B.
Now RAW, I think the rules specify (L+A) as the size of the list. So RAW, yes, you can't have "empty" prepared spells if you read RAW as absolute in the list size (ie you "MUST" prepare L+A even though that language is not used). I can accept that as RAW. But, is "nothing" a valid "spell" to put on the list (my null spell argument - certainly not very popular here!)?
But RAI, I think (L+A) is intended to be a max, a balancing limit. So, as RAI, I think it's a valid
ruling to allow for less than max prepared. Whether I had a "full" prepared list the previous day or not only matters in terms of prep time. I would argue, that should I rule that you can prepare a smaller list, you need to spend the prep time "unprepping" a spell, but that's not the center of this discussion.
After you prepare a short list, RAW demands you wait until after another long rest to be able to prepare a long list (thereby effectively "filling that slot" as it were) - There is some debate on when that occurs, but I think its fair to say it happens only once by RAW and RAI.
RAF - To allow this as I've written, a spellcaster would have to purposely limit their prepared list. During "downtime" this might not be a bad thing, but in the middle of an adventure this is a penalty: you have less choices for that adventure day. Then following a long rest you have the ability to increase flexibility
If and only if the adventure allows for enough prep time to do so
and that flexibility can be greater if the DM allows for prepping at any point in after a long rest (such as a few minutes before you need the spell). Given that there is penalty that almost perfectly balances with the reward of flexibility, I don't see this as a balance problem. It can be narrative/gamist/meta-game problem (ie PC preps small list during downtime so as to have flexibility on day one of adventure) but that is a table problem. So it is fun? (that is rhetorical, we all have to answer that to ourselves and our table).