robots.txt is a file that excludes certain URLs or parts of your website from being read by search engine robots and other web crawlers (automated programs that surf and index the web). It works by placing robots.txt at the top level, next to favicon.ico. The robot will typically look at robots.txt first, and honour that file by not loading up the URLs for in that file.
It is only used for disallowing and excluding pages. It will not help search engines include or 'find' pages from your website
Sometimes it is unnecessary for pages such as the forgot password or feedback pages to be displayed in the search results of google.
Another reason for excluding pages is that it dilutes your ranking if you have unrelated or non optimised pages being indexed.
Exclude specific pages:
User-agent: * Disallow: /users/forgot_pass.php Disallow: /users/contact.php
User-agent: * Allow: /