Over the years Ive occasionally had the same problem with players not showing up or just dropping out of games. The worst case resolution may be to just take a break from playing from awhile. Youd be surprised how the issue usually resolves itself. Chances are those casual players wont return to the game, your loyal players will and youll probably wind up with a new player or two. Start up a new game and hopefully that will solve your problem. I've done this in the past and it worked for me a few times.