[ Home | About | Licence | About author | Contact ]

Written by Oleksandr Gavenko (AKA gavenkoa), compiled at 2015-10-05 from rev 764bb173578d.

Web site.

robots.txt.

To exclude all robots from the entire server:

User-agent: *
Disallow: /

To exclude all robots from part of the server:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

To allow a single robot:

User-agent: Google
Disallow:

User-agent: *
Disallow: /

To allow all robots complete access:

User-agent: *
Disallow:

See:

Sitemap.

Sitemaps protocol allows a webmaster to inform search engines about URLs on a website that are available for crawling.