Would it be worth keeping spiders out of /forums/ ?

[Plane Sailing]

I just checked your robots.txt file and I wonder if it would be worth adding the /forums/ directory to it, to prevent the forums being spidered by search engines such as google etc.

The thing is, there are so many links per page that a spidering could paralyse the web site with thousands of requests - I don't know if that is what I was seeing last night, but there were chunks of time when my poor little dial up connection couldn't get a look-in!

If you think that ENworld is best served by a policy of letting google etc. index everything then fair enough, but I hope you don't mind me raising it as a potential way of reducing load on the server without preventing current users from getting at stuff.

Cheers

 

[Piratecat]

Last night (and for the next few days) you'll see us reindexing the search index. It makes things much slower, but it will be done in another 4-5 days.

Interesting idea, though.

 

[Plane Sailing]

I guess a quick analysis of the server logs would easily indicate how much traffic spiders are generating and what proportion of the whole they are.

Cheers

 

[Morrus]

Search engine spiders are a good thing, though, are they not? We want people to end up here when they search for D&D stuff on Google.

 

[Plane Sailing]

Search engine spiders are a good thing, though, are they not?

It sorta depends on how often and how aggressively they search really; the fact that all the pages in the forums have lots of links may mean that each page generates 200 subsequent requests, and then each of those 200 requests more... although like I said, a quick look at the server logs would would reveal whether or not it is a problem.

The fact that re-indexing is taking place at the moment and slowing things down periodically probably means that is likely to be the culprit for the recent slowdowns I experienced anyway.

So on balance I guess it probably isn't worth keeping spiders out of /forums/ (although checking the logs for the degree of spider activity still might prove instructive)

Cheers