Having all rituals being available as complexity 4 skill challenges still means that *at best* it would take 10 rounds to complete, probably more and with a good chance of the ritual failing completely.
Wait, I may be misunderstanding you. Are you saying that it would be a complexity 4 skill challenge that only one person (the ritual caster) could participate in?
If yes, then I pity that poor wizard's player. I'm going to spend the entire fight doing nothing but making Arcana checks. That doesn't sound like interesting options, that sounds like, "You, go sit in the corner." Yes, he could stop every few rounds to toss a fireball, or put up a wall...but that's one more round that the skill challenge isn't being worked toward. Something we've seen with traps that require multiple successes to disable is that it basically sends one player to the corner for a large chunk of the fight, and heaven forbid you fail a check or two. If this is what you mean, then I can understand why you'd think that it would be hard to "bust" an encounter (although it's not even encounters I'm worried about, as much as the flow of the adventure); I think this would be terrifically boring for the PC in question, though.
If you're saying that no, anyone could participate, three characters could fail a skill challenge in one round, or complete it in a little over 2 rounds at best (assuming they spend action points to make multiple skill checks). I can think of tons of ways this can bust adventures/encounters. For example, the jailbreak adventure. Break in, free your buddy, escape. Teleport rituals as skill challenges make that "break in, free your buddy, hold the bad guys off for a couple of rounds and avoid 1/3rd of the adventure." Now, that might be an interesting scene in and of itself, but if you want that option to be out there then you're better off putting it out there for that instance (a scroll that can be cast as a skill challenge) rather than make it something that can be done every. single. time. It's really hard to put the rabbit back in the hat. Maybe that's why I favor the more conservative rules with more aggressive exceptions approach.
Again, I'm not saying that I don't think that allowing rituals to be performed as a skill challenge is a bad idea, I just don't like it as a blanket rule. I do like it as something that can come up from time to time. However, the thing to remember about blanket rules is that every book that gets released, every new ritual that comes into existence, has the potential to test the limits of that blanket rule, and sometimes even break things. That's a big thing you have to think about when doing game design like this--how it's going to work with things going forward, and how it's going to interact with a near-infinite number of options.
Could it make for some interesting, tense scenes to allow rituals to be used as a skill challenge? Absolutely. I just think that if you make it a blanket rule, you're back to the whole "magic can solve any of our problems" issue.