Crawl Readiness MCP Server

An MCP server that checks whether a website is set up for AI the way its owner intends: can GPTBot, ClaudeBot, PerplexityBot and 50+ other AI crawlers read it, and what does its access policy allow? Then it generates the files (llms.txt, an AI-crawler robots.txt, JSON-LD structured data), reads your Answer Analysis, and tells you whether a page's published signals permit a use (search, answers, training, agent action), inside Claude Desktop, Cursor, Cline, Windsurf and other MCP clients.

Ask Claude "can AI read my site?" and it will actually check, running a live audit against 50+ AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google's and more) and reading the access policy the site states. Then ask it to fix what is blocked, and it can generate a proper llms.txt, robots.txt or JSON-LD schema and write it straight into your project.

No more copy-paste between your editor and a browser.

What it does

Nine tools your AI assistant can call:

Audit (no signup):

• check_ai_readiness — Score a site 0–100 against 50+ AI crawlers, with a prioritized fix list; extras that don't affect the score are marked as such
• validate_schema — Audit JSON-LD structured data on any URL (20+ types, per-type rules)
• validate_robots — Line-by-line robots.txt audit
• check_content_parity — Compare what humans see vs what AI crawlers see

Permission (no signup):

• check_permission — Ask whether a page may be used for search, answers, training or agent action. Reads robots.txt, Content-Signal, Content-Usage, an RSL licence, the TDM reservation, ai.txt and the page's own directives: any reservation wins, and a source that could not be read makes the answer unknown, never allowed. Returns a dated receipt signed with Ed25519, with a hash of every file read. Documented at machinereadiness.com/permission

Generate (requires free API key):

• generate_llms_txt — Auto-crawl a site and produce a properly formatted llms.txt
• generate_robots_txt — AI-crawler-aware robots.txt with preset policies
• generate_schema — Organization / WebSite / Article JSON-LD ready to paste

Answer Analysis (read-only; needs an API key, and the analyses come with Monitor, Agency or the audit):

• get_monitor_trend — Read your Answer Analysis: for the questions your customers ask, whether live AI search (Perplexity, Google AI Overviews) named your brand or a competitor, and the change run over run (reads existing analyses; doesn't trigger runs)

Install

Claude Desktop, one click

Download crawl-readiness-mcp-<version>.mcpb from the latest release and double-click it (or drag it into the Claude Desktop window). Claude Desktop shows what the extension does and asks for the optional API key. The bundle carries the server and its dependencies, so nothing else is installed.

Claude Code, as a plugin

This repository is also a Claude plugin: the .mcp.json at its root starts the server (pinned to this version), and the ai-readiness skill tells Claude when to use which tool and how to read the results. Add it from the Claude directory once it is listed there. To try it before then, clone the repository and start Claude Code with it loaded:

claude --plugin-dir ./crawl-readiness-mcp

Claude Code asks for the optional API key when the plugin is enabled and keeps it in secure storage; the plugin's own files never contain it.

For Claude Desktop, by hand

Open your config file:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

Add this to the mcpServers block:

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": ["-y", "crawl-readiness-mcp"]
    }
  }
}

Restart Claude Desktop. You'll see "crawl-readiness" listed in the tools panel

... View full README on GitHub