@pipeworx/commoncrawl

Common Crawl's public web archive — find every time a URL was crawled since 2008 and read back the exact page bytes that were captured.

Part of Pipeworx — an MCP gateway connecting AI agents to 1863+ live data sources. This is an independent, unofficial integration — not affiliated with, endorsed by, or published by the upstream provider.

Tools

• commoncrawl_crawls(limit?, contains_date?) — the monthly crawl collections (CC-MAIN-2026-34 and friends) with the capture window each one covers. contains_date answers "which snapshot would have seen my page on that day".
• commoncrawl_index_search(url, crawl?, match_type?, limit?, page?, filter?, from?, to?) — CDX index lookup for a URL, host or whole domain. Returns capture timestamp, HTTP status, MIME type, detected language, content digest, and the WARC filename + offset + length that the next tool needs.
• commoncrawl_fetch_record(filename, offset, length, max_body_bytes?) — reads one archived record back by byte range: WARC headers, the captured HTTP response headers, and the page body as crawled.

Auth

Keyless.

Data sources

• https://index.commoncrawl.org/collinfo.json — the crawl collection list.
• https://index.commoncrawl.org/{crawl}-index — the CDX index, JSON-lines.
• <https://data.commoncrawl.org/{warc path}> — WARC records, HTTP range reads.

Things the next person would otherwise rediscover

• The CDX host 502s sporadically under load. Measured 2026-09-17: the same url=example.com query alternated between 200 and nginx 502 inside a minute, so it is load, not the query. ccFetch retries a 5xx twice with a short backoff. Without that a caller reads transient nginx noise as "Common Crawl has no record of this URL", which is a different and wrong answer.
• "No captures" is a 404 with an English sentence, not JSON. Handled as an empty result set with an explicit note, so an empty captures array is never silently indistinguishable from an upstream failure.
• Each indexed record is its own gzip member, so a byte-range read of offset..offset+length-1 decompresses standalone — no need to stream the whole 1 GB WARC. DecompressionStream('gzip') handles it in the Workers runtime.
• new TextDecoder('utf-8', { fatal: false }) does not typecheck against @cloudflare/workers-types: its TextDecoderConstructorOptions requires ignoreBOM too. Pass no options; non-fatal is the default anyway.
• A broad match_type: "domain" query over a large domain is expensive upstream and is the first thing to 502. Narrow with match_type: "host" plus a filter such as =status:200.

Related packs

crawlgraph is our own link graph over sites we crawl. This pack reads the Common Crawl Foundation's corpus — different data, different questions.

Quick Start

Add to your MCP client (Claude Desktop, Cursor, Windsurf, etc.):

{
  "mcpServers": {
    "commoncrawl": {
      "url": "https://gateway.pipeworx.io/commoncrawl/mcp"
    }
  }
}

What this endpoint actually serves

tools/list at https://gateway.pipeworx.io/commoncrawl/mcp returns the tools in the table above plus the shared Pipeworx meta-tools — ask_pipeworx, discover_tools, search_within, remember/recall and the rest of the gateway-wide set. So the tool count you see is larger than this table: a single-pack endpoint currently lists roughly 30 shared tools alongside the pack's own. The connection's initialize response states its exact scope, and is the authoritative answer for a given day.

This is deliberate, not multiplexing by accident. The meta-tools are what let a scoped connection answer a question this pack does not cover — via ask_pipeworx, which routes across the whole catalog — without you adding a second MCP server. There is currently no way to mount a pack endpoint without them; if the extra schemas cost you more context than the rou

... View full README on GitHub