Screaming Frog SEO Spider MCP Server (headless)

A headless MCP (Model Context Protocol) server for Screaming Frog SEO Spider. It drives the SF command line and saved crawl database directly, so Claude (or any MCP-compatible client) can run crawls, export crawl data, and analyze the results with the Screaming Frog GUI closed: on your laptop, on a server, or inside scheduled audits and CI pipelines.

This is a community project, not affiliated with Screaming Frog. Since SEO Spider v24 there is also an official MCP built into the app. The two work differently and solve different problems.

How this differs from the official Screaming Frog MCP

Screaming Frog shipped an official MCP server in SEO Spider v24. It's substantial: around 29 tools covering crawl control (start, pause, resume, progress), reports and bulk exports with field selection, URL-level inspection, screenshots, embeddings exports, and optionally a Node.js script runner with npm and filesystem read/write tools. It runs in two modes, either a Streamable HTTP server inside the open app, or a STDIO mode where the MCP client launches the Spider itself, headless. Setup is documented for Claude Desktop and LM Studio.

If you want maximum capability in an interactive session (visualizations, crawl comparison, screenshots, scripted post-processing of exports), use the official MCP. It does far more, and it's maintained by the vendor.

This server makes a different trade: it's a small, deliberately limited wrapper around SF's CLI and the saved crawl database, built for runs where nobody is watching.

Locked-down by design. Nine read-and-export tools, nothing else. No script runner, no npm install, no filesystem write access. The official MCP offers all three, and its own docs note that enabling the Node runtime "allows the execution of arbitrary code on your system" and should only be granted to a fully trusted client. There's also an SF_ALLOWED_DOMAINS allowlist to restrict what an agent is able to crawl. When an agent runs unattended on a schedule, a tool surface this small is a feature.

Installs anywhere, plainly. A pip/uv-installable Python package with a one-line stdio config on any MCP client (Claude Code, Cursor, whatever) on macOS, Linux, or Windows. The official STDIO mode ships as a Claude Desktop extension (.mcpb); the HTTP mode means opening the app and starting the server from its settings.

Light process model. Screaming Frog only runs while a tool actually needs it. The official server is the Spider application running for the whole session, whichever mode you pick.

A few tools the official set doesn't have: aggregate_crawl_data for counts and distributions computed server-side (the official path to "how many 404s" is a full export, or a Node script), delete_crawl and storage_summary for cleaning up SF's crawl database (their sf_clear_crawl clears a paused crawl, it doesn't manage stored ones), regex filtering across any column of any export via read_crawl_data, and sf_check pre-flight diagnostics that catch license problems and GUI database locks before you waste a crawl.

What it feels like from chat. The official server is stateful: you ask it to load a crawl by ID, the Spider holds it in memory for the session, and follow-up questions answer in under a second. The cost is that the session owns SF's database the whole conversation, and exports come back as full inline dumps unless the model saves files and writes Node scripts to slice them (the approach their own docs recommend for staying inside the context window). This server is stateless: each export spawns the SF CLI fresh, so the first answer on a crawl takes longer, but you can just ask ("list my crawls, export the latest one") without managing IDs or sessions, reads ret

... View full README on GitHub