Skip to content
Pinhelm

Help

HelpSite Audit

Allow our crawler

robots.txt lets the Site Audit crawler read the public pages.

Site Audit fetches pages with its own user agent. It honors robots.txt.

Steps

  1. Open the site's robots.txt.
  2. Allow the user agent PinhelmBot, or allow all agents, for the paths you want checked.
  3. If you publish a sitemap, link it from robots.txt.
  4. Run the audit again from Site Audit.

Result

Pages that robots.txt allows can be scored. A blocked homepage is reported as a finding instead of a silent skip.

Tips

  • The user agent is PinhelmBot/1.0. The contact page for that agent is this article.
  • Allowing the crawler does not add the site to a public index. It only lets this audit read the pages.
  • See Run a site audit.