It's not that the fluff doesn't matter. The fluff is yours to do with what you want because it's not part of the mechanics. If I want to make an eldritch knight (for example) and take the entertainer background and history that can easily be a variation on the type of bard I want.I feel that if the fluff doesn't matter, then why even have the fluff? Why fix the things into combined packages in the first place? If we have splats, be they species, classes or backgrounds, their fiction should matter. If we don't want that, then do not have splats. Have freeform character building system instead, where people can buy whatever combination of stats and features and give it whatever fiction they want.
It's forcing mechanics for the fluff or vice-versa where we create our own problems. Keeping those independent of each other gives us that room to do what we want with our fluff.
If that makes sense. ;-)