Each page can contain meta tags, and you can control the behaviour of some robots using the robots meta tag.

index vs. noindex

If you use index, it indicates that the crawler should add the page into its potential pool or search results.

follow vs. nofollow

follow refers to the crawler looking at the links on the current page and whether to follow those links.

Examples

<META NAME="robots" CONTENT="noindex,follow">

The page will not turn up on the search results, but pages it links to will still be processed.