io.github.sharozdawa/robotstxt-ai

Manage which AI bots can crawl your website — visually.

Toggle GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers on/off with a simple UI. Analyze any site's robots.txt instantly.

Features

• Visual Toggle UI — Block or allow AI bots with simple on/off switches
• 20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
• Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
• Generate robots.txt — Create a complete robots.txt with your chosen rules
• MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
• Check Bot Status — Verify if a specific bot is blocked on any website

MCP Tools

Installation

Web App

git clone https://github.com/sharozdawa/robotstxt-ai.git
cd robotstxt-ai
npm install
npm run dev

Open http://localhost:3000 in your browser.

MCP Server — Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": ["-y", "robotstxt-ai-mcp"]
    }
  }
}

MCP Server — Cursor

Add to .cursor/mcp.json:

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": ["-y", "robotstxt-ai-mcp"]
    }
  }
}

Tracked Bots

The server knows about 25+ bots including:

• AI Crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent
• AI Search: ChatGPT-User, OAI-SearchBot, PerplexityBot, YouBot
• Search Engines: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot

Why robotstxt.ai vs Manual Editing

More Open Source SEO Tools

License

MIT