Question: What effect will this line in robots.txt have on Google's crawling behavior? # Disallow: /
- It will disallow crawling of every page and resource on the site
- It will disallow crawling of the home page only
- It will allow crawling of the home page only
- It will not have any effect on crawling
Explanation
In robots.txt, a line that begins with # is treated as a comment, not as an active crawler directive. HubSpot’s documentation shows comment lines such as # Block all bots, followed separately by real crawl rules like Disallow entries. Because the slash rule here appears after #, Google would ignore it rather than apply it. Actual crawl blocking only happens when Disallow is written as an active directive under a matching user-agent. Developers HubSpot+1
Why the other options are incorrect
A) This would only happen with an active Disallow: / rule, but the leading # makes the line a comment instead. Developers HubSpot
B) The home page would not be singled out here, because no active crawl directive is being applied at all. Developers HubSpot
C) This line does not create an allow rule, and it does not function as an active directive because it is commented out. Developers HubSpot
Source for verification
HubSpot Developer Docs: Content Search Developers HubSpot
HubSpot Knowledge Base: Prevent content from appearing in search results HubSpot Knowledge Base
The answer(s) to the question is highlighted in the BOLD text above. You can also find more questions and answers related to the exams on the "HubSpot CMS for Developers II" page.