Odds are it's one of the following.
1) It could be your dice. Spend some time rolling your dice. Since you're doing 4E, I'd focus on the good ol' d20. Roll your d20 about 100-200 times & write down each number. Then check out the distribution, focus on the average & the top 5 numbers rolled. If the top 5 numbers are 1,2,4,5 & 8, with your average around 6, then maybe it's time for a new dodecahedron friend.
2) It's probably all in your head. People have a tenancy to remember facts that reinforce a preconceived notion & discard evidence that contradicts it. It's Psychology 101. There's no such thing as being "cursed" or natually unlucky. Probability is just a difficult concept us to understand some times.
See Money Hall Problem So, after you've verified your dice are good, (see #1 above) keep track of your rolls during a game. Every time you roll your d20, write down the number. After you've gotten to around 100-200 rolls, check the distribution. It should be pretty normal. And you can be comforted in that knowledge. And if they're still way off, even though outside if your game they roll normal, well....
3) You're cursed by the dice gods. I lied before, you can be cursed and you are! May they have mercy on your soul... And enjoy being relegated to playing Amber!!

(in 4e, go for powers that have a miss effect). Oh & don't ever touch anyone else's dice, especially mine!!! *scoops up dice*