Which of the following is NOT a function of the robots.txt file?

1. Show search engine bots all the pages on a site
2. Prevent search engines from crawling private pages
3. Keep search engine bots from overwhelming server resources
4. Specify the location of your sitemap

Correct Answer :

• Show search engine bots all the pages on a site