I'm running the Obsidian system pretty much as written and haven't had the party fail in 6 skill challenges (with party size ranging from 4 to 8). This is probably due to the fact that I'm a bit lenient on the use of non-standard skills (allowing them whenever a player can give me a good explanation) so I'm not too concerned about it.
There are three issues that have cropped up though:
1) I'm running the Scales of War adventure path and thus converting skill challenges written under the
WotC rules into Obsidian. As is, this is hard because there is no guidance on how to translate the Complexity factor. If I simply ignore it (since Obsidian doesn't have one) then a Complexity 1 and a Complexity 5 challenge both convert to the same thing, which upsets my players when they are paying attention come XP time. As written, the Obsidian system seems to only be able to really convert Complexity 5 skill challenges (based on XP equivalency). Short skill challenges are probably a step in the right direction, but I haven't had the chance to try them out yet (and that would only really take care of one of the 4 other complexity levels). Larger skill challenges are for moving beyond Complexity 5 (since they tie together multiple full Obsidian challenges). It would be really nice to see a system for how each of the different Complexities should map into Obsidian style skill challenges.
2) Along the same lines, standard skill challenges generally list 4 or more primary skills, while the Obsidian system is setup to handle just one or two. How should the DM adjust the skill challenge in the Obsidian system to account for such a large number of primary skills? Also, what does one do with the Secondary (Other) Skills that are usually listed in a standard skill challenge? These skills don't contribute to success or failure, but provide bonuses on subsequent checks. Obsidian's round based system makes it nigh on impossible for this kind of thing to work because it's generally better to try the same check twice than to use a secondary skill to make it easier to succeed on one primary check.
3) Combat Skill challenges need some work. The information provided is only basic and lacks hard numbers. I would love to see something like page 4 for Combat Skill challenges. Also, there needs to be a provision for failure. In Siege at Bordin's Watch, for example, there is a Magic Crossbow Turrent trap that can be deactivated at the control panel by a successful skill challenge. However, there is a kind of built-in anti-tampering device in that the control panel explodes should the person playing with the panel fail the challenge. Obsidian's Combat skill challenges don't provide for this kind of dynamic. (I ended up resorting to the standard skill challenge for this one.)
And yet another word from our sponsors
