2,457
edits
ROBOTS: Difference between revisions
no edit summary
Nicole Sharp (talk | contribs) No edit summary |
Nicole Sharp (talk | contribs) No edit summary |
||
| Line 8: | Line 8: | ||
Comments are added to the Robots Exclusion Protocol with a hash ("<code>#</code>") at the beginning of a new line. | Comments are added to the Robots Exclusion Protocol with a hash ("<code>#</code>") at the beginning of a new line. | ||
As with all webtext files, you should use an advanced text editor such as [https://www.notepad-plus-plus.org/ Notepad-Plus-Plus] that supports Unix line endings.  Do not use Microsoft Windows Notepad. | |||
For a public website, you want to allow access to all bots in order to get the site crawled and indexed by as many search engines as possible.  The following "<code>/robots.txt</code>" allows all bots access to all files and directories (minus anything excluded by "<code>/.htaccess</code>"). | For a public website, you want to allow access to all bots in order to get the site crawled and indexed by as many search engines as possible.  The following "<code>/robots.txt</code>" allows all bots access to all files and directories (minus anything excluded by "<code>/.htaccess</code>"). | ||