crawl-census-client

Ask before you fetch. A drop-in client that stops your crawler spending requests on doors that are shut, and stops it routing around content someone is trying to sell.

Reading robots.txt answers one question and hides two others. Measured across 23,482 domains by Crawl Census:

• 2,874 domains permit AI agents in robots.txt and then refuse those same agents at the network edge. A parser sees permission; the fetch returns 403. You pay for the round trip and get nothing.
• 208 domains answer an AI user agent with HTTP 402 Payment Required. That is a price, not a refusal. Treating it as a block walks away from content the operator wants to sell you. Retrying around it takes something they are charging for.

No dependencies. No key required.

MCP server

The same measurement is exposed as a remote MCP server, so an agent can ask before it fetches rather than after it fails. Listed in the official MCP registry as io.github.taylorsmithgg/crawl-census.

{ "mcpServers": { "crawl-census": { "url": "https://crawlcensus.com/mcp" } } }

No authentication for read tools. Streamable HTTP.

Install

npm i github:taylorsmithgg/crawl-census-client
pip install git+https://github.com/taylorsmithgg/crawl-census-client

Use

import { politeFetch } from "crawl-census-client";

const r = await politeFetch("https://example.com/", { agent: "gptbot" });
if (r.skipped) console.log(r.verdict, r.reason);   // disallow | refuse | pay
else           process(await r.response.text());

from crawl_census import polite_fetch

r = polite_fetch("https://example.com/", agent="gptbot")
if r.skipped:
    print(r.verdict, r.reason)
else:
    process(r.body)

Skipping is returned, not raised. It is the normal outcome for a large share of the web, and a crawl loop should be able to count skips without a try/except around every URL.

Split a queue before crawling it

One call per 1,000 domains instead of one per host:

const { crawl, skip, pay, unknown } = await partition(urls, { agent: "gptbot" });

p = partition(urls, agent="gptbot")
p.crawl, p.skip, p.pay, p.unknown

Or just take the file

For a fetcher that only needs a deny list in memory, skip the per-domain calls entirely:

curl https://crawlcensus.com/agents/gptbot/blocklist.txt   # one domain per line, commented header

const sync = await syncBlocklist("gptbot");   // full list once
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000);  // then deltas only, a few hundred bytes

sync = BlocklistSync("gptbot")
if host in sync: skip()
sync.refresh()          # {'added': 3, 'removed': 1, 'size': 3310, 'cursor': ...}

The delta feed is https://crawlcensus.com/agents/<agent>/changes.json?since=<unix> and each response carries next_since, so a long-running crawler stays current on a few hundred bytes an hour instead of re-downloading the list.

That file covers robots.txt only. Edge refusal and HTTP 402 are per-request behaviours and still need preflight or politeFetch.

What a crawl costs the census

Measured, not asserted. Twenty hosts fetched concurrently used to cost twenty preflight calls carrying one domain each; the same host requested three times at once cost three, because the cache only helps after the first lookup resolves. The anonymous allowance is 240 calls an hour, so a crawler hit its ceili

... View full README on GitHub