2,457
edits
ROBOTS: Difference between revisions
no edit summary
Nicole Sharp (talk | contribs) No edit summary |
Nicole Sharp (talk | contribs) No edit summary |
||
| Line 7: | Line 7: | ||
Comments are added to the Robots Exclusion Protocol with a hash ("<code>#</code>") at the beginning of a new line. | Comments are added to the Robots Exclusion Protocol with a hash ("<code>#</code>") at the beginning of a new line. | ||
For a public website, you want to allow access to all bots in order to get the site crawled and indexed by as many search engines as possible.  The following "<code>/robots.txt</code>" allows all bots access to all directories (minus anything excluded by "<code>/.htaccess</code>"). | For a public website, you want to allow access to all bots in order to get the site crawled and indexed by as many search engines as possible.  The following "<code>/robots.txt</code>" allows all bots access to all files and directories (minus anything excluded by "<code>/.htaccess</code>"). | ||
<code><highlight lang="robots"> | <code><highlight lang="robots"> | ||
| Line 23: | Line 23: | ||
</highlight></code> | </highlight></code> | ||
The following "<code>/robots.txt</code>" excludes two webdirectories ("<code>/sandbox/</code>" and "<code>/testbox/</code>") from crawling/indexing but permits access to all other | The following "<code>/robots.txt</code>" excludes two webdirectories ("<code>/sandbox/</code>" and "<code>/testbox/</code>") from crawling/indexing but permits access to all other files and directories on the site. | ||
<code><highlight lang="robots"> | <code><highlight lang="robots"> | ||