Sounds to me like the problem is one of meta-knowledge, and the best way to resolve that is to push for non-meta justification, and non-meta consequences.
So: they saw you
as a player roll a save. What did the
character see? Was there some kind of tell, and if so, what was it, and
how did they see it?
Maybe they saw your eye twitch at a bitter flavor. How did they see that? Since you consumed the fruit with no visible issues (having passed your save), the signs must have been relatively minor. That sounds, to me, like some kind of Perception check, or possibly an Insight check against the host to determine their true intent.
Beyond that, let's just say for the sake of argument that the DM gives them this meta-knowledge, some kind of reason or even just gut feeling that the fruit is unsafe. You mentioned hospitality is a big deal in this culture, which usually means that
breaking the laws of hospitality is a HUGE no-no. The guest and the host must equally refrain from injuring one another, so long as they are bound by the guest right; in cultures that observe this sort of thing IRL, a host who violates their sacred duties has committed a
horrible offense and will likely be ostracized by the community as a result (and, at least in myth, should expect a special delivery atmospheric electrical discharge courtesy of everyone's favorite divine philanderer.) This then means that an
accusation of violating these sacred duties is an incredibly serious charge, one that should not be made lightly.
Hence, I would expect some kind of save or social-skill roll to do that without giving massive offense. Just because they
believe the fruit may be unsafe, does not mean they can simply refuse outright. They've been caught in a trap of social expectations, and have to find their way out.
There may be other ways. Perhaps eagle-eyed or medically-trained characters could spot something awry about the fruit itself, or deceptive characters could try to bluff the host into
thinking they've enjoyed the fruit when they haven't, or some other way of wriggling out of this trap without actually setting it off. But they can't just blanket refuse, without
in-character reasons, and expect that to be the end of it.
Edit: That said, yes, the DM could also have altered their approach so as to prevent some of this, by asking how each player responded to the feast
before detailing what it was like. At that point, more diverse responses open up, e.g. the players have
taken fruit but get a chance to try to
spot that eager-fruit-eater
@iserith had an odd response to it, and thus a chance to change their behavior just in the nick of time...or not.