The problems I have with skill challenges and forced participation are twofold:
One, Immeral isn't a social character. It would, in the most literal sense, Out Of Character for him to get into a protracted social "challenge" of most sorts, at least at the present time; that's why I had him withdraw from this one (BTW, Osric: Immeral never even offered to help with carrying the trader, let alone started to do it!

He had already left at that point). It would be similarly OOC for other characters to be nosing in on areas where they're intentionally ignorant. Forcing participation is a method of railroading and I very much dislike that the DMG includes it as an implied rule for Skill Challenges.
Two, failure is failure, regardless of how severe it is. Directly contributing to failure isn't fun. Being
forced to directly contribute to failure is about as far from fun as I can imagine.
Also, the math of the Skill Challenge system has a lot of flaws. You have to be very precise in how you set it up or it becomes either far too difficult or far too easy for the party to succeed in the challenge. Even changing the target DC by 1 can have a dramatic (20+ percentage points) shift in the probability of success/failure. If you just follow the chart in the DMG, the chance of success for an average party is something like 20%.
That all said, I trust Redclaw not to make the game unfun.