By raw, I'm pretty sure you are ruling it correctly. To the best of my knowledge, there's no rule about hitting interposing objects.
That said, it seems totally legit to claim that you must first hit the arrow slit. The obvious thing would be Dex vs 10 AC (the AC/Ref of a tiny object) with...