To get around this, a simple limit that the spell cannot be successfully cast again while any skeletons from any previous casting remain animated solves the problem. Or, if that's too harsh, a limit that says that your maximum number of skeletons (i.e. the number you'd get if you cast this once using your highest level slot) is the most you can have going at any one time no matter how many times you cast the spell.
I don't mind the ritual idea. If it bugs you, then sure - remove the ritual tag. But still keep the casting time at ten minutes - this isn't something you cast once combat begins, it's something you cast ahead of time. Or, see below regarding digging time.
I still don't think this spell fits with what is a typical ritual spell, nor are there any ritual spells that are limited in their casting. All ritual spells are utility spells that offer no real benefit it combat. Being able to effectively summon 5 minions for free, minions that last until destroyed, means this very much is a combat relevant spell. No spell or class feature in the game allows this kind of resource for free.
Aside from the issue with this being a ritual, the only difference between this and an Animate Dead spell upcast to 5th level is the casting time and the lack of requiring line of sight. Without the ritual tag, I don't think there's not enough in this spell to differentiate it from Animate Dead.
From experience in other editions, let's just say that can open up some cans of worms best left closed.
No disrespect, but saying "Trust me, this is broken" is not really a helpful statement when we are actively trying to dissect a potential new mechanic and it's impact on the game. How can we be sure if the issue is in fact related to the mechanics, and not just the idiosyncrasies related to your table and how you play the game? Especially when the basis of your statement is on your experience with other editions, and not the one we are discussing.
That being said, how would this be a can of worms? There's nothing inherently more powerful about animating a small or large undead, especially if the only change is the size, and maybe increasing the damage die by one step (which is not overpowered for a higher level spell). I'm not saying allow animating zombie beholders or minotaur skeletons, though allowing something like that would differentiate the proposed spell from an upcast Animate Undead and might be a reasonable addition to the spell, though still not as a ritual.