If you get terrorized by someone telling you that you possibly are in legal trouble (and what the maximum sentence for that could be) over not returning the cards you should not have in the first place but refuse to give back, then that is on you.
First of all, you could have returned them when they contacted you, did you expect they just go away after you tell them ‘no’?
Second, I would not have been terrorized, and neither was the guy from what I have heard of him. They had a not so friendly chat, nothing I would not shrug off. The guy never believed that legal ‘threat’ and I agree that was a stretch, so what.
If you cannot have a ‘talking to’ don’t try to play hardball.