2,457
edits
ROBOTS: Difference between revisions
no edit summary
Nicole Sharp (talk | contribs) No edit summary |
Nicole Sharp (talk | contribs) No edit summary |
||
| Line 6: | Line 6: | ||
"<code>robots.txt</code>" will only work from the root webdirectory ("<code>/</code>"). <ref><code>https://www.robotstxt.org/faq/shared.html</code></ref> | "<code>robots.txt</code>" will only work from the root webdirectory ("<code>/</code>"). <ref><code>https://www.robotstxt.org/faq/shared.html</code></ref> | ||
== documentation == | == documentation == | ||
| Line 17: | Line 13: | ||
* [https://developers.google.com/search/docs/crawling-indexing/robots/ Google Developers: Introduction to <code>robots.txt</code>] | * [https://developers.google.com/search/docs/crawling-indexing/robots/ Google Developers: Introduction to <code>robots.txt</code>] | ||
* [[wikipedia:robots.txt|<code>robots.txt</code> (Wikipedia)]] | * [[wikipedia:robots.txt|<code>robots.txt</code> (Wikipedia)]] | ||
== editor == | |||
As with all webtext files, you should use an advanced text editor such as [https://www.notepad-plus-plus.org/ Notepad-Plus-Plus] (not Microsoft Windows Notepad). <ref><code>https://www.notepad-plus-plus.org/</code></ref>  Files should be saved with [https://www.npp-user-manual.org/docs/preferences/#new-document Unix line endings and UTF-8 (Unicode Transformation Format Eight-Bit) character encoding]. | |||
== comments == | |||
Comments are added to the Robots Exclusion Protocol with a hash ("<code>#</code>") at the beginning of a new line. | |||
== examples == | == examples == | ||