The purpose of a robots.txt file is to:

1. list the pages on a website to manage the content that’s shown to search engines
2. instruct search engines on how to handle duplicate content
3. instruct search engine bots on how to crawl the pages on a website
4. tell HubSpot the user agent of a visitor’s browser

Correct Answer:

• instruct search engine bots on how to crawl the pages on a website