Question: When should you use a robots.txt file?
- When you have multiple versions of a page to indicate the preferred version
- When your website receives a penalty from Google
- When you have pages that you don't want search engines to crawl and index
- Whenever you feel like it
Explanation
A robots.txt file instructs search engines on which pages or sections of a website should not be crawled or indexed. It helps control crawler access to sensitive, duplicate, or non-public content. Proper use of robots.txt ensures that search engines focus on important pages and preserves crawl budget.
Why the other options are incorrect
When you have multiple versions of a page to indicate the preferred version is addressed with canonical tags, not robots.txt.
When your website receives a penalty from Google is unrelated to robots.txt functionality.
Whenever you feel like it is incorrect because robots.txt should be used deliberately, not arbitrarily.
Source for verification
https://knowledge.hubspot.com/seo/technical-seoturn3search4
The answer(s) to the question is highlighted in the BOLD text above. You can also find more questions and answers related to the exams on the "HubSpot SEO Certification Course" page.