I'm not really sure what my first one was. Prior to college, everything was probably pretty much by the book D&D, Shadowrun, Mechwarrior, or Rifts. I don't remember any of us even really playing with any house rules.
When I got to college, I probably threw my hat in with about a dozen different people who were all working on their own systems. I don't remember much about the mechanics for any of them other than that they were all insufferably bulky attempts at trying to describe the world with half understood real life mathematics in an attempt to have a role playing game that was "more realistic."
Within a couple of years, I had thoroughly burned myself out on that philosophy of gaming, as well as trying to run a solid weekly campaign with a bunch of flakey college students. I started putting together (albiet usually very railroady) stand alone games, each with their own system. The eventual idea was to publish them in a book entitled One Night Stands, but the idea eventually withered and died.
I think that the first game I made was a two part, sort of Twilight Zoney, science fiction, prison break story. In broad, and nowhere near as serious and gritty strokes as the game itself was presented, the world is embroiled in a war between the people who like technology (who are all mostly cyborgs), and the no good hippy psychics who are trying to ruin everything. The first half of the game, the players play a bunch of cyborg prison guards trying to track down and capture a bunch of escaped psychic POWs. After the players capture them, the tables are turned, and the players are informed that they're now playing the part of the prisoners, who just woke up from a prophetic dream where they are now able to foresee the coming events. The mechanics of the game centered around a deck of cards I made up that was basically a normal playing deck with Zener Card symbols (
http://en.wikipedia.org/wiki/Zener_cards) instead of the normal four suits.
Each character had a different ability that interacted with the cards in some way. If I remember correctly, the cyborgs abilities were all mechanical benefits tied to having a particular symbol show up when they were flipping cards. For the psychics, the abilities all involved predicting what the symbol of the next card was going to be. The deck was also never shuffled, and although there were abilities that would slightly affect the way that cards were discarded, the players could effectively count cards and try to memorize the order. I'm trying to remember the exact mechanics, but it's been almost 15 years. I think that I had a basic list of DCs, and that you'd flip over two (maybe three) cards to see if you beat the number. Combat, or any other head to head challenge was basically a game of War, but with two cards instead of one.