Inktomi

[Evil Chris]

It appears that Inktomi is spidering the hell out of xnations the past 24 hours or so.

I'm wondering... is this a good thing or a bad thing? Should I ban that IP or just let it go?

Anyone have any thoughts?

[wsjb78]

Our sites get often spidered by Inktomi but they never show up in any search results... but we still let it spider us!

[StuartD]

nah, they're the good guys... they used to power a ton of search engines... msn, yahoo, alltheweb.... a bunch of others I think.

But lately most of them have been switching to google.

I'm sure there's still enough search engines using inktomi to make it worth having them spider you.

[pornJester]

I wouldn't worry too much. Your site is run off the Vbulletin script and they probably would design it to be SE friendly.

[Cyndalie]

I got hit by MSIE Crawler about 1500 times last minth. Google 316 .. these numbers are normal.

A search engine spidering your site is a good thing - is you want to be indexed and it's not overloading your system. (On average a spider may use 5-50 Megs of Bandwidth or more). The number of revisits means that it is coming from pages that have linked to Xnations, or it could be repeating efforts because Xnations is so dynamic.

I wouldn't really worry about it.

[Cathedra]

No no i'd say it's a good thing.. let it ride :cool:

[Evil Chris]

It's freaking me out! LOL :D

[NetRodent]

If its not causing a problem let it spider. The only time we ever had a problem was years ago when we had about 200 domains with about 10,000 pages each in a single P2 400 mhz. One day T-Rex from Lycos started hitting the site so hard the machine froze up (swapping memory for run away apache processes). Ah, the good old days. :cool: