In terms of risk management, this is... not a wise way to think about it, either for your personal health, or the community at large.
The flu vaccine is cheap and safe. Both the flu and covid are generally common and easy to catch. And if you have one, the added risk to you from the other skyrockets.
Meanwhile, your current risk mitigation can be entirely breached by unforeseen circumstance, leaving you without any other defense. But once you've gotten the flu vaccine, that protection is yours for the season, and cannot be taken away.
Layered defense is far superior. Unless you are allergic, arguments that the flu vaccine isn't a good idea are hard to come by.