This is an oddly worded ability apparently (I haven't seen the actual text). If it doesn't cost anything when it doesn't help, why would you not choose to use it on checks you think are really important. Thus it could have simply been worded: "before making an ability check, you may spend [resource] to add [dice] to the result of the check. If the outcome of success doesn't change, you regain the spent resource." Oh well.
Another option is to set the DCs of impossible tasks (finding a door/creature that doesn't exist) infinitely high, allowing the player to use the ability, which won't affect the outcome. Social skills are a bit different, and I would simply ask after a roll "do you want to use Knack?" If they do and it doesn't matter, then they don't spend the resource.
Your players could simply choose to act correctly in-character. The PCs don't have the information, so the players shouldn't act as if they do. Our group does this, and it's sometime hilarious when one player succeeded with Insight, but most of the party failed. Everyone knows the truth, but the only person who succeeded has to find a way to convince the party of it.Does this ability interfere with any mystery since they automatically know if they succeeded or failed in a roll?
How will you handle this and/or am I reading this wrong?
Another option is to set the DCs of impossible tasks (finding a door/creature that doesn't exist) infinitely high, allowing the player to use the ability, which won't affect the outcome. Social skills are a bit different, and I would simply ask after a roll "do you want to use Knack?" If they do and it doesn't matter, then they don't spend the resource.