2,457
edits
ROBOTS: Difference between revisions
no edit summary
Nicole Sharp (talk | contribs) No edit summary |
Nicole Sharp (talk | contribs) No edit summary |
||
| Line 46: | Line 46: | ||
== SITEMAP == | == SITEMAP == | ||
<u>[[SITEMAP]]</u> is an extension of the Robots Exclusion Protocol to allow listing a sitemap in the "<code>/robots.txt</code>" file. <ref><u><code>[[SITEMAP]]</code></u></ref> <ref><code>https://www.sitemaps.org/</code></ref> <ref><code>https://www.sitemaps.org/protocol.html</code></ref>  Having a precompiled list of links for the website available makes the bot's job to crawl and index the site a lot easier and more efficient.  Since the first thing a good bot does when accessing a website is to check for a "<code>/robots.txt</code>" file, it is best to have the link to the sitemap listed directly in the "<code>/robots.txt</code>" file so the bot doesn't have to guess whether or not the website has a sitemap available (which could be either "<code>/sitemap.txt</code>" or "<code>/sitemap.xml</code>").  The following "<code>/robots.txt</code>" provides an example of a public website with a sitemap.  Note that unlike the other ROBOTS instructions, the sitemap should be provided with a full URL (uniform resource locator) and not with a relative link. | <u>[[SITEMAP]]</u> is an extension of the Robots Exclusion Protocol to allow listing a [https://www.sitemaps.org/ sitemap] in the "<code>/robots.txt</code>" file. <ref><u><code>[[SITEMAP]]</code></u></ref> <ref><code>https://www.sitemaps.org/</code></ref> <ref><code>https://www.sitemaps.org/protocol.html</code></ref>  Having a precompiled list of links for the website available makes the bot's job to crawl and index the site a lot easier and more efficient.  Since the first thing a good bot does when accessing a website is to check for a "<code>/robots.txt</code>" file, it is best to have the link to the sitemap listed directly in the "<code>/robots.txt</code>" file so the bot doesn't have to guess whether or not the website has a sitemap available (which could be either "<code>/sitemap.txt</code>" or "<code>/sitemap.xml</code>").  The following "<code>/robots.txt</code>" provides an example of a public website with a sitemap.  Note that unlike the other ROBOTS instructions, the sitemap should be provided with a full URL (uniform resource locator) and not with a relative link. | ||
<code><highlight lang="robots"> | <code><highlight lang="robots"> | ||
| Line 87: | Line 87: | ||
* <u><code>https://www.nicolesharp.net/robots.txt</code></u> | * <u><code>https://www.nicolesharp.net/robots.txt</code></u> | ||
* <code>https://www.robotstxt.org/</code> | * <code>https://www.robotstxt.org/</code> | ||
* <code> | * <u><code>[[SITEMAP]]</code></u> | ||
* <code>https://www.securitytxt.org/</code> | * <code>https://www.securitytxt.org/</code> | ||
* <code>https://humanstxt.org/</code> | * <code>https://humanstxt.org/</code> | ||