Question: Which of the following is NOT a function of the robots.txt file?
- Show search engine bots all the pages on a site
- Prevent search engines from crawling private pages
- Keep search engine bots from overwhelming server resources
- Specify the location of your sitemap
Explanation
The robots.txt file does not list all pages on a site; it instructs search engines on which pages to allow or disallow for crawling. It can also help prevent bots from overloading server resources and indicate the location of the sitemap. Proper configuration of robots.txt ensures efficient crawl management and protection of sensitive content.
Why the other options are incorrect
B) Preventing crawling of private pages is a primary function of robots.txt.
C) Limiting bot activity helps reduce server strain, which robots.txt facilitates.
D) Robots.txt can specify the sitemap location to guide search engines.
Source for verification
HubSpot Academy: Technical SEO Audit
The answer(s) to the question is highlighted in the BOLD text above. You can also find more questions and answers related to the exams on the "HubSpot SEO II" page.