I love it when my players spam ritual spells in dangerous situations, because I always play intelligent enemies as... intelligent.
Enemy Group #1 hears an approaching caster rambling away with magic words every round, usually from 50-100 feet away. In a quiet house with no nearby car traffic, I can easily hear someone talking in a normal voice through a closed door from 50 feet away, so that seems eminently reasonable. If the group is intelligent, they will be able to recognize spellcasting - it's not a normal conversational cadence; it's one dude/dudess chanting away in a monotone, round-after-round. Enemy Group #1 immediately leaves their room and gathers Enemy Groups #2 and #3, forming a single Deadly+ encounter. Further, any enemy caster will take the time to cast preparatory defensive spells, archers will overturn tables and get behind cover, stealth-specialists will prepare ambushes. The party will never surprise the enemy, and will be facing them in optimal defensive positions. This happens virtually every time in my games, and the party knows that they'll be "pulling aggro" on nearly everything in the dungeon if they give the enemy ample warning.
If they stay in one place and try the same, it's simple enough to measure distances. Unless the party is at least 100 feet away from the nearest enemy, and behind at least 1-2 closed doors, and the enemy has no motivation to send patrols... then they can expect to be attacked. And, once more, the enemy will come en masse; it will be multiple encounters gathered together in overwhelming numbers, with pre-cast spells and other defensive measures.
So, does this invalidate ritual casting altogether? Not at all. Fundamentally, rituals are for casting after you've cleared the area out, or just before you enter the "dungeon" (tavern, tower, ship, forest, whatever) for the first time. They're great for that first room or two (until the duration expires, or concentration is broken), and for tying up loose ends at the end. In the specific context of see invisible, it's ritualled to find invisible defenses in the first 30 minutes of the dungeon, or find invisible "phat lootz" after the boss monster is killed.
...
So, what if the party pulls back out of hearing range to cast it again, before moving forward? Easy enough; the enemies discover the bloodied signs of their intrusion so far, and prepare ambushes for when they return. Perhaps not every time, but often enough to make it unprofitable as a tactic. Besides: how many invisible stalkers, sprites and prepared wizards are they running into? There are so few creatures that are actually invisible, and so many creatures that have excellent natural stealth. If you're casting see invisible to spot the minority of enemies, but giving the majority of enemies advance warning of your approach and allowing them to prepare a non-invisible ambush, what's the point?