If anything, the fact that it doesn't apply all of the time makes it almost as annoying. That may sound counter-intuitive, but hear me out. If it did apply to every single check, I could just forget about it and let the players always add the 1d4. But, since there are quite a few situations where it doesn't apply, I have to consider whether or not the players could use guidance in that particular moment and remind them not to add the 1d4. That gets just as annoying as people announcing it constantly. If anything, it's more annoying for me as the DM because then it puts the burden of remembrance on me rather than the player. Either way, it's a real nuisance.