Help with SEO: Robots.txt

[Fyrflygrl]

I asked on Cozy and I'll ask here, too: I can find all kinds of information about Robots.txt files and how to exclude certain files from being indexed.... but I can't find anything that will tell me this:

Is it necessary to have a robots.txt file? What happens if you don't have one? (i.e. if there is no robots.txt file, will (potentially) every page of the site be indexed? or will NONE of the site be indexed?)



:confused:

Fyrflygrl

[wsjb78]

if you have no robots.txt every page will be attempted to be indexed...

[Cyndalie]

yep, robots.txt file is the first thing and engine will request when indexing or spidering your site, it tells it what NOT to index. Unless you want to keep spiders OUT, you dont need it.

[Fyrflygrl]

Thanks MUCH wsjb78 & Cyndalie!!

I really appreciate the help :)

[NetRodent]

Even on domains where I want everything indexed, I still include a blank robots.txt. Why? Because I hate to have my error logs cluttered up with unnecessary 404 entries.

Its also fun to use them to watch for malicious spiders. If you deny a directory in your robots.txt that nobody would have any legitimate reason to access, you can tell that any requests hitting that directory
are up to no good.

[Cyndalie]

This article goes a bit more in depth about Robots.txt and meta uses to direct spiders http://www.sitepronews.com/archives/2003/aug/29prt.html

[Taass]

Originally posted by NetRodent
Even on domains where I want everything indexed, I still include a blank robots.txt. Why? Because I hate to have my error logs cluttered up with unnecessary 404 entries.


Think i read somewhere that you need to put something in your robots.txt file or you risk not beeing indexed at all..
As i remember this, SE robots finding an empty robots.txt file can read this as meaning you don't wan't any robot spidering at all..
not 100 % sure of this though, but might be something you wan't to check up on :)