The purpose of a robots.txt file is to:

  • list the pages on a website to manage the content that’s shown to search engines
  • instruct search engines on how to handle duplicate content
  • instruct search engine bots on how to crawl the pages on a website
  • tell HubSpot the user agent of a visitor’s browser
Join our list

Subscribe to our mailing list and get interesting stuff and updates to your email inbox.

Thank you for subscribing.

Something went wrong.

Leave a Comment

Join our list

Subscribe to our mailing list and get interesting stuff and updates to your email inbox.

Thank you for subscribing.

Something went wrong.

Send this to a friend