agentfetch-mcp

Web intelligence for AI agents — an MCP server that fetches URLs with token estimation, smart caching, and intelligent routing built in.

AgentFetch sits between your agent and the open web. Instead of integrating Jina, FireCrawl, pypdf, and your own caching layer separately, agents call one MCP tool and AgentFetch handles routing, caching, token budgeting, and clean Markdown extraction automatically.

This repository contains the open-source MCP server. For the hosted API + dashboard + billing, see www.agentfetch.dev.

What it does

Under the hood, AgentFetch routes URLs to the cheapest effective fetcher:

• Trafilatura (free, local) for ~70% of standard web pages
• Jina Reader for the rest of HTML
• FireCrawl for JS-heavy pages (Twitter/X, LinkedIn, Notion, etc.)
• pypdf for PDFs (zero external cost)

Cache is Redis with a 6-hour TTL; you can bring your own or run without caching.

Quick start

Install from PyPI

pip install agentfetch-mcp

Or clone and install locally

git clone https://github.com/bch1212/agentfetch-mcp
cd agentfetch-mcp
pip install -e .

Set environment variables

Get a free Jina Reader key at jina.ai (1M tokens/mo free tier). FireCrawl is optional but recommended for JS-heavy pages.

export JINA_API_KEY=jina_xxx
export FIRECRAWL_API_KEY=fc-xxx       # optional
export REDIS_URL=redis://localhost:6379  # optional

Add to Claude Desktop or Claude Code

Edit your MCP config (~/Library/Application Support/Claude/claude_desktop_config.json on macOS, or run claude mcp add in Claude Code):

{
  "mcpServers": {
    "agentfetch": {
      "command": "python",
      "args": ["-m", "agentfetch.mcp.server"],
      "env": {
        "JINA_API_KEY": "jina_xxx",
        "FIRECRAWL_API_KEY": "fc-xxx"
      }
    }
  }
}

Restart Claude. The four tools (fetch_url, estimate_tokens, fetch_multiple, search_and_fetch) appear automatically.

Run as a standalone server

python -m agentfetch.mcp.server

The server speaks MCP over stdio (the standard transport for desktop integrations).

Why agents prefer AgentFetch over generic web fetch

Examples

Fetching with a token budget

# Inside any MCP-aware agent (Claude Desktop, Claude Code, etc.)
result = fetch_url(
    url="https://news.ycombinator.com",
    max_tokens=2000,           # cap response size
    use_cache=True,            # serve from cache if <6h old
)
# result.markdown      → clean Markdown, ≤2000 tokens
# result.metadata      → title, author, word_count, language
# result.cache.hit     → True if served from cache
# result.fetch_info    → which fetcher ran, cost, duration

Estimating before committing

estimate = estimate_tokens(url="https://very-long-article.com")
if estimate.estimated_tokens and estimate.estimated_tokens < 5000:
    result = fetch_url(url="https://very-long-article.com")
else:
    # too big — skip or

... [View full README on GitHub](https://github.com/bch1212/agentfetch-mcp#readme)