meta tag robots
Each page can contain meta tags, and you can control the behaviour of some robots using the robots meta tag.
index vs. noindex
Section titled “index vs. noindex”If you use index, it indicates that the crawler should add the page into its potential pool or search results.
follow vs. nofollow
Section titled “follow vs. nofollow”follow refers to the crawler looking at the links on the current page and whether to follow those links.
Examples
Section titled “Examples”<META NAME="robots" CONTENT="noindex,follow">The page will not turn up on the search results, but pages it links to will still be processed.