Okay, this guy is pulling out all of the manipulative tricks he knows to get to you back down. He's using Intimidation, Bluff, and otherwise undermining your right as a DM and attacking your confidence to run your own game.
What I would do is call or meet with each other player individually and explain the situation (because I bet you that he is doing the same thing.) Ask them if what you have done is unreasonable. I bet you that you will get a story from other players on how he has intimidated the other DM to allow him to keep using that die, or you will find out that other players also have their suspicions. Ask them what they would do if they were DMing and they were uncomfortable with how a person was generating their die rolls. Ask them what you should do, and inform them that this is an issue that they guy will not let go and will probably work very hard to destroy the enjoyment of the game for everyone else if he doesn't get his way. But if he gets his way, then you have essentially allowed a bully to rule the group. Ask them if it is OK with them for this guy to rule them.
There's a certain amount of respect that a DM needs to function as a DM. Although it is good to incorporate player feedback into your game, you have the final say in what goes or doesn't go.
So there are a few of ways that this could go. You could run your game and he would use his die. You could run your game with him using a differnt die (and probably poisoning the environment.) You could run your game without him (thus completing his plan of making YOU the "bad guy".) You could choose to not run you game and keep playing. Or you can leave that group and start a new group with selected invited players from that group. Or you can leave entirely.
Before you choose to give in or not run your game. Check to see how the other players would like things to go.
One other thing, if you let him use his die, you can keep track of the die rolls,
then use them for his opponents' rolls. 