Bard: Magical Secrets is not part of spellcasting, so I don't actually think you can swap out those spells. That said, you're always swapping a spell for a spell of the same level, so even a more lenient approach doesn't really bother me.
Oh, no, you can definitely swap out magical secrets. Bards don't have 2 separate lists of spells known.
When I level up from 5th level to 6th level on a lore bard I get extra magical secrets. At that time, I can select spells from any class using the magical secrets rules. So I could add fireball and alter self, for example. The magical secrets rules state specifically that those are bard spells from me.
The rules for swapping out spells state I can change a bard spell I know for a spell on the bard spell list. If I decide I don't like alter self I can swap it for any spell
on the bard list because alter self is a 2nd level bard spell for me as specified by that secrets choice when chosen. I cannot swap it out for lightning bolt using the level up mechanic because I'm swapping a bard spell for a bard spell on my bard spell list using the swapping spells mechanic. How alter self actually became a bard spell for me is irrelevant to the swapping RAW. Any spell not on the bard list is off limits to swap in because the swapping mechanic doesn't create a condition to consider lighting bolt (from the example) a bard spell or on the bard spell list.
Mistaking spells known via secrets as somehow special bard spells is what leads to confusion. Treating them exactly like any other bard spell is how they work and why they are part of the spells known table. This is how the UA version works:
"Whenever you finish a long rest, you can replace
one spell you learned from this Spellcasting feature with another
spell from the bard spell list."
Again, magical secrets gives the bard a bard spell regardless of list. A bard spell is a bard spell is a bard spell and how it became a bard spell is irrelevant. The language on the swap is also specific to the bard spell list and doesn't include any mechanic to swap a bard spell for a non-bard spell.
Don't overcomplicate the definition of a bard spell.
EDIT: the fact that it says "from this spellcasting feature" would prevent swapping entire lists during downtime because the one swapped in is always the one swapped out RAW.
