The main issue is that the 3ish players are not the same each time. There is little consistency for story and I don't have the ability to plan ahead for encounters, etc.
So why don't you run a game for the ones who want to play D&D & are leaving in order to do so?
As for those 3(ish) players who're unreliable? They get to play characters who's sheets stay with you.
When they can make it? Great, here's your/a character.
When they can't make it? Those characters are NPCs/run by the other players/maybe not used at all.
You write your adventures based on the regulars who will show up
{they're the stars of the story, but you just don't advertise that to the others} & plan encounter difficulties for 1 more than that.
If more show up? Add an extra monster/increase HPs & AC etc. Or just shrug & accept that the evenings encounters will be a bit easier than expected.
This is exactly how my Thur 5e game runs
(though the group has opted to not have anyones character played at all if they aren't there). I've got 3 regulars atm, 1 guy with a fluctuating work schedule, & 1 gal who now has a very busy school schedule.
Work guy adds a bit of firepower when he can make it.
School gal is mostly just there for RP & hanging out with her friend (one of the regulars). But even when she could make every session she has
never been a factor mechanic wise..... Fun to play with, but no threat to the cosmic balance.
You set game night to a time that works well for you & the regulars.
You give a short recap of what went on last time at the start of each session.
If the regulars are still having trouble keeping track of what went on? Then you'll have to scale back the scope/detail of the campaign.
And don't be afraid to scrap/indefinitely pause the current campaign in favor of something that'll work better.