The purpose of a robots.txt file is to:

Advertisement

  • list the pages on a website to manage the content that’s shown to search engines
  • instruct search engines on how to handle duplicate content
  • instruct search engine bots on how to crawl the pages on a website
  • tell HubSpot the user agent of a visitor’s browser
Join our list

Subscribe to our mailing list and get interesting stuff and updates to your email inbox.

Thank you for subscribing.

Something went wrong.

Leave a Comment


Share via
Join our list

Subscribe to our mailing list and get interesting stuff and updates to your email inbox.

Thank you for subscribing.

Something went wrong.

Send this to a friend