io.github.Grubbomatic/crawl-readiness
Audit any site for 50+ AI crawlers, generate llms.txt, robots.txt and schema, track AI mentions.
753 servers in this category
Audit any site for 50+ AI crawlers, generate llms.txt, robots.txt and schema, track AI mentions.
All public upAPI operations as MCP tools: web scraping, search, screenshots, PDF, OCR and more.
Crawler diagnostics, AI visibility evidence and article drafts for review in Prerender Buddy.
Local browser automation for AI agents: real-browser fetch, capture, smoke tests, cross-engine QA.
Audit a site for AI readiness (llms.txt, schema, AI-crawler access) and generate the fixes.
Web Content Retrieval (full webpage, filtered content, or Markdown-converted), Custom User-Agent, Multi-HTTP Method Support (GET/POST/PUT/DELETE/PATCH), LLM-Controlled Request Headers, LLM-Accessible Response Headers, and more.
Give any AI a real computer. Open source MCP server with Ubuntu sandbox, browser automation, desktop control, and 30+ tools. Works with Claude Code, Cursor, Claude Desktop.
Spot pre-launch products before they trend. Search the web and tech sites, extract and parse pages…
MCP server for the Scrawly SEO crawler: query local audits and propose confirmation-gated fixes.
MCP server for technical & on-page SEO audits — crawlability, HTTPS, Core Web Vitals, sitemap, robots.txt, structured data, and AI-crawler access
Search, extract, crawl, map, research, scrape 16 platforms, browser automation, proxy — one API key.
MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl.
BYOK, fully local AI agent that tests your app/API and writes a real Playwright spec on success.
Verify what a web page really returns over HTTP and in a browser. Deterministic, local, no LLM.
Search and read the web pages and highlights you saved with the Markpin Chrome extension.
Live web access for agents: scrape, SERP search, crawl/map, 100+ collectors, datasets, proxies.
MCP server for AI agents: discover a web page's data sources and drive the page to reach them.
220 browser automation tools for Chrome, Edge, and Firefox via CDP + BiDi. 100% Python.
Self-hosted, Tavily-compatible web search, extract, crawl and map tools for AI agents.
MCP gateway for QA/SDET work: Playwright, files, GitHub, Jira, Postgres & more in one connection.