a few things on indices
I work in publishing (non-gaming), and can't make excuses for this - I agree, indices are HIGHLY needed for many of these books. That said, here's some stuff from behind the curtain that may shed some illumination.
1) It's actually very expensive to do a "good" index - many companies even outsource this part of the production process. Auto-indexers are great, but if you want a strong, readily usable index, you don't want to see every page where "magic" was referenced, but the pages where it was referenced in regards to a new rule that you're going to want to check on the fly. Human decision making is needed to edit the results from what a program spits out as all of the uses of the word "magic" (for instance).
2) It's not the designer's/author's responsiblity to do an index - editors will change sequencing of pages, perform layout, and do other things that would throw off whatever preliminary index an author might write. Some houses have authors contribute to the process, but when it comes down to it, it's the actual generating of useful page numbers, not keywords, that is the labor intensive part.
3) Most Tables of Contents on WotC products are actually pretty good. That said, an index would be much more helpful. (e.g. Silver Marches - I can never find what I want in there, which is sad - it is SUCH a great book.)
4) The index is often one of the last things to be done. If a product is running late on a production schedule, it's no surprise that it gets scrapped.
I haven't seen Hero Games' more recent titles, but if they're doing it, I agree - it should be done.