Ha, it's all good. I'm sure most of us have been there. You should see me when someone tries to criticize Bladerunner or Dark Sun.
I'm inclined to think otherwise, but it's probably just because of terminology. I like the exploration part of D&D quite a bit - to me, exploring hallways and interacting with weird elements outside of rules elements are a big part of the fun (probably one of the biggest parts, actually). And I don't really want to turn that haunted hallway into an "encounter" - it's more just some weird flavour that the PCs can observe and question, and then move along their way.
That does seem to be just an issue of terminology. Whether it's interacting with your environment between encounters or an encounter with your environment - your still interacting with the haunted hallway in an interesting way. Framing it as an encounter may help those that need a sense of pacing and might make it easier to define power use outside of combat ( I do wish there was a more defined use for 4e powers outside of combat) but for experienced DMs - there's little difference.
As for skill challenges... I never really understood them in the DMG presentation, and I ignore that incarnation. However, I use Stalker0's Obsidian Challenge System now, and find it works a lot better with how I prefer to run D&D.
That being said, I've never had a problem with running non-combat encounters, so that particular part of 4e wasn't built for me - it actively got in the way of my fun, so it was dropped. No biggie.
I think Stalker0 has an excellent system and prefer it myself, but yes it's just a tool and not everyone needs it.