crawlerpeek

Check whether AI crawlers can access your site. For each bot (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot, Bingbot, CCBot and more) it tells you:

• whether robots.txt allows the URL's path for that bot (RFC 9309: user-agent groups, wildcards, $, longest match wins), and
• what the page actually returns when fetched with that bot's user-agent string: status, final URL after redirects, content type, size, title, and whether the response looks like a block or challenge page.

It also lints a robots.txt for mistakes, checks every URL in a sitemap against robots.txt, and generates a robots.txt that welcomes AI crawlers. A library, a CLI for CI, and an MCP server. Node 20 or newer, no runtime dependencies, ESM and CommonJS, TypeScript types included.

The same checks as a web tool: crawlerpeek.com. Documentation with examples for CI and MCP clients: crawlerpeek.com/cli.

Install

npm install crawlerpeek       # library
npx crawlerpeek --help        # CLI, no install needed

CLI

npx crawlerpeek check https://example.com

https://example.com
robots.txt: ok (HTTP 200) at https://example.com/robots.txt

Bot                 robots    HTTP
GPTBot              blocked   200  FAIL
ClaudeBot           allowed   403 (block page)  FAIL
Google-Extended     allowed   -

npx crawlerpeek check https://example.com/pricing --bots GPTBot,ClaudeBot --fail-on blocked
npx crawlerpeek check https://example.com --bots all --json
npx crawlerpeek check https://example.com --no-http          # robots.txt only
npx crawlerpeek see https://example.com --bot ClaudeBot      # one bot, full detail
npx crawlerpeek verify 66.249.66.1 --bot Googlebot          # is this IP really Googlebot?
npx crawlerpeek lint https://example.com                     # find mistakes in its robots.txt (or: lint ./robots.txt, lint -)
npx crawlerpeek scan https://example.com --limit 50          # robots.txt per bot for the URLs in its sitemap
npx crawlerpeek check https://example.com --markdown         # GitHub-flavoured Markdown table
npx crawlerpeek robots --disallow /admin/,/cart/ --sitemap https://example.com/sitemap.xml

Exit codes: 0 ok, 1 the --fail-on condition was met, 2 usage error (including an unknown option) or request error. see exits 0 when the page was fetched, whatever its HTTP status, and 2 when the request failed or was refused.

verify <ip> [--bot GPTBot] [--json] checks whether an address from your server logs really belongs to a crawler. It uses only what the operator documents: the published IP-range JSON (Google, OpenAI, Anthropic, Perplexity, Bing, Apple, Common Crawl, DuckDuckGo, Ahrefs) or reverse DNS with forward confirmation (Google, Bing, Apple, Common Crawl, Ahrefs). Without --bot every bot with such a method is checked. Each result has method (ip-list, reverse-dns or none), verified (true, false, or null when it cannot be decided, for example the list is unreachable or the operator documents no method) and a detail. Lists are cached for one hour, and nothing is ever sent to the IP. Exit codes: 0 verified for at least one bot, 1 not verified or undecidable, 2 usage error.

check --markdown prints the result as a GitHub-flavoured Markdown table (a heading with the URL, a ✅/❌ summary line and one row per bot) instead of the text table; it cannot be combined with --json. --summary-file <path> appends the same Markdown to a file in addition to the normal output, which is how the GitHub Action fills the job s

... View full README on GitHub