Subtract 1 FULL HD (d10) from the 3e Ranger's total, then find the ratio of actual hp to possisble hp on the remaining HD. Then take the same number of HD on d8s, multiply the total possible by that percentage, add 8 and round up. Ignore CON bonus for this process, then add it after you're done.
example:
11th level 3e Ranger: 90 HP before CON
1) Subtract the FULL HD gained at 1st level (10HP)
2) Find the percentage of 80/100 (the remaining 10HD.) This is 80% exactly.
3) Multiply 80 (max of 10d8) by .8 (80% of 80.) This is 64.
4) Add 8 (New full 1st level HD) to the total.
This gives the 11th level 3.5 Ranger 72HP prior to Con bonus.
Another option I've seen was to simply allow the character to have the same number of hp IF it wasn't more than the maximum available to him post-conversion. Not sure I like that, but it's an option if you don't want to do too much math, and more likely to appease those who don't like the HD change.
edit: Sorry for stating the obvious, but I wanted to clarify!!

Thanks to Jens for reminding me about the whole "full HD at first level". After playing too much hackmaster, I forget that kind of thing!
