Meta Robots Tags, About Robots.txt and Search Indexing Robots
Entry | Meaning |
---|---|
User-agent: * | Because nothing is disallowed, everything is allowed for every robot. |
User-agent: mybot | mybot robot may not index anything, because the root path (/) is disallowed. |
User-agent: * Allow: / | For all user agents, allow. |
User-agent: BadBot | The BadBot robot can see the robot policy document, but nothing else.All other user-agents are by default allowed to see everything.This only protects a site if "BadBot" follows the directives in robots.txt |
User-agent: * | In this example, all robots can visit the whole site, with the exception of the two directories mentioned and any path that starts with private at the host root directory, including items in privatedir/mystuff and the file privateer.html |
User-agent: BadBot User-agent: * | The blank line indicates a new "record" - a new user agent command. All other robots can see everything except any subdirectory named "private" (using the wildcard character) |
User-agent: WeirdBot User-agent: * | This keeps the WeirdBot from visiting the listing page in the links directory, the tmp directory and the private directory. Allother robots can see everything except the temp directories or files,but should crawl files and directories named "temperature", and shouldnot crawl private directories. Note that the robots will use thelongest matching string, so temps and temporary will match the Disallow, while temperatures will match the Allow.
|
Bad Examples - Common Wrong Entries | |
use one of the robots.txt checkers to see if your file is malformed | |
User-agent: googlebot Disallow / | NO! This entry is missing the colon after the disallow. |
User-agent: sidewiner | NO! Robots will ignore misspelled User Agent names (it should be "sidewinder"). Check your server logs for User Agent name and the listings of User Agent names. |
User-agent: MSNbot Disallow: /PRIVATE | WARNING! Many robots and webservers are case-sensitive. So this path will not match any root-level folders named private or Private. |
User-agent: * User-agent: Weirdbot
| Robots generally read from top to bottom and stop when they reach something that applies to them. So Weirdbot would probably stop at the first record, *. Ifthere's a specific User Agent, robots don't check the * (all useragents) block, so any general directives should be repeated in thespecial blocks.
|
Read More...