I, Robot Part 2
Another way to prevent your Web pages from being indexed by roaming Web spiders is through the use of a tag. This approach offers three different options:
<meta name="robots" value="noindex">
This tag tells visiting robots not to index THIS page. They can, however, follow its links to other pages.
<meta name="robots" value="nofollow">
This tag tells robots that they may index this page, but they're not to follow any links on it to other pages.
<meta name="robots" value="none">
Finally, this tag tells robots neither to index this page nor to follow any of the links on it to other pages.
You'll need to remember to place this tag inside every page you wish to hide from visiting robots. As with the robots.txt file technique, this approach isn't guaranteed. Between the two, however, you'll probably have the obedience of 95 percent of the robots out on the Web.