That's not just a small party, it's a weak party.
The Fighter has just hit his first "kick in the nuts" level. They don't have any healing, they don't have any "fight-ending" magic, and they don't have a consistent skill set -- at least ONE of them is clanky, right?
I'd look more to the individual abilities of CR 1, 2 and 3 encounters. Some allegedly appropriate critters will hit them disproportionately hard, especially melee monsters -- those are usually best handed with magic.
Cheers, -- N