Looking at AC, to Hit bonus, and hit points and comparing it to the PCs stats it's one way to approach it. If attack and spell damage is enough to kill any of them instantly, it's probably not going to be a good choice. In the same way, if the AC and hp are so low it should be dead in one round, I know it's probably not going to be hard. The difficult part is to reverse engineer it and come up with a threat for a certain level in advance.
In general, a stock X HD monster is a reasonable threat for PC's for X level, where as an X+4 HD monster is serious challenge. Most monsters are going to do about 1d4 damage at 1 HD, and go up one average .5 to 1 damage per HD. So, 1d4, 1d6, 1d8, 2d4, 1d10, 1d12, 2d6, 3d4, 2d8, etc. is a fairly reasonable progression. AC is going to be about 8 and go down by 1 per HD you go up. In general, AC's rarely get much better than 0. After AC 0, start adding other sorts of defensive abilties - saves as higher HD monster, 10% blocks of spell resistance, an immunity, etc. You can trade back and forth a bit on damage versus protection, or break damage into multiple attacks. Feel free to play around a bit. Up until about 7HD, any extra ability like ranged attacks, surprise resistance or bonuses to hit and damage above the usual for the HD that deserves an extra XP award adds the equivalent of about 1 HD to the difficulty. Any really big ability like poison, gaze attack, or energy drain adds about 2 HD to the estimate. Above about 6HD monsters get about 1 extra ability for free (or one free HD), and above about 9HD they get two minor or one major one for free (or two free HD, so an 11HD monster with just good stats but not a lot of powrs is about a level 9 monster). Keep adding an addition ability or HD above the curve for each 3HD up. Power is scaling up fast at this point, and for characters above 10th level with the best sorts of equipment if you really want to bring the challenge you've practically got to cheat.
So, you want to design a really tough encounter for a party of 6 7th level characters, you need something like an 11HD monster with an attack doing 4d4 damage and a -2 AC and a powerful additional ability or a 9HD monster with several additional abilities. By the time you get to 12th level, you need like 16HD monsters with 3-4 powerful abilities. In general though, don't expect single encounters to overwhelm prepared PCs unless you cheat a bit with unfun save or die sort of stuff. I'm not a big fan of the glass cannon strategy for doing so that you see in 1e - dragons are really dangerous in 1e, but not because they are well designed. It's because their damage from their breath weapon is a bit absurdly out of scale with everything else they do.
Personally, when I'm awarding XP for monsters I like to award bonus XP for each additional ability that a monster has (see the list in the DMG) rather than just whether it has at least 1. That's because I liked to tweak down the percentage of XP coming from treasure acquisition just a tad, and it also seemed a fairer measurement to me.
