Web scraping + research toolkit for AI agents: scrape, crawl, monitor URLs with anti-bot bypass.
MCPpedia last refreshed this data
io.github.SanggonBoy/PyreCrawl is an MCP server that web scraping + research toolkit for AI agents: scrape, crawl, monitor URLs with anti-bot bypass. Its tool list has not been published yet over stdio and http, requires no API key, and scores 83/100 on MCPpedia's security, maintenance and efficiency rubric.
Config is the same across clients - only the file and path differ.
{
"mcpServers": {
"pyrecrawl": {
"args": [
"--from",
"pyrecrawl",
"pyrecrawl",
"serve"
],
"command": "uvx"
}
}
}Are you the author?
Add this badge to your README to show your security score and help users find safe servers.
Web scraping + research toolkit for AI agents: scrape, crawl, monitor URLs with anti-bot bypass.
This server supports HTTP transport. Be the first to test it - help the community know if it works.
Five weighted categories - click any category to see the underlying evidence.
No known CVEs.
Checked pyrecrawl against OSV.dev.
Click any tool to inspect its schema.
Be the first to review
Have you used this server?
Share your experience - it helps other developers decide.
Sign in to write a review.
Others in browser / ai-ml
Open-source web scraper for AI agents with scrape, crawl, and map tools
A cross-platform desktop application for managing AI Agents and MCP Servers. 一款用于管理AI智能体与MCP服务器的跨平台桌面应用程序。
Search-first MCP server for JavaScript reverse engineering: 735 tools across 36 domains.
An open-source AI agent that brings the power of Gemini directly into your terminal.
MCP Security Weekly
Get CVE alerts and security updates for io.github.SanggonBoy/PyreCrawl and similar servers.
Start a conversation
Ask a question, share a tip, or report an issue.
Sign in to join the discussion.
One command gives any AI agent the whole web. Scrape, extract, crawl, map, and search — self-hosted, no API keys, no rate limits, no subscription.
PyreCrawl speaks MCP (Model Context Protocol), the standard tool interface for Claude, Cursor, VS Code, Codex, OpenCode, Hermes, and any MCP-compatible agent.
A smart auto-fallback ladder always picks the cheapest method that succeeds:
fast HTTP
│ (403/503/Cloudflare challenge or empty body)
▼
stealth browser (real Chromium + Cloudflare solver)
│ (still blocked, or the page needs full JS rendering)
▼
deep processing (LLM-ready markdown, citations, structured extraction)
| Tool | What it does |
|---|---|
scrape(url, prefer="auto") | Single URL → LLM-ready markdown |
extract(url, schema) | Scrape + structured extraction (JsonCss schema) |
map_site(root, include_pattern=None, limit=200) | Enumerate all internal URLs |
crawl(root, max_pages=5, prefer="auto", include_paths=None, exclude_paths=None, max_depth=0) | Multi-page crawl with path filters + true BFS depth |
document(url) | PDF/DOCX/PPTX → markdown (no browser, optional [docs] extras) |
search(query, limit=10) | Web search via DuckDuckGo HTML (no API key) |
search_papers(query, limit=8, source="arxiv", category=None) | Academic search via arXiv + Crossref (no API key) — feed pdf_url into document |
batch_scrape(urls[], ...) | Many URLs in ONE call — parallel, deduped, cache-aware |
deep_research(query, limit=5, scrape_top=3) | Search → evidence pack with [n] citations (no LLM synthesis — your agent does that) |
monitor(url, action, css_selector=None) | Change detection with persisted snapshots + unified diff |
session(session, action, ...) | Persistent browser session (cookies kept) — login walls, multi-step flows, screenshots |
cache(action) | Inspect/clear/enable/disable the HTTP response cache |
health() | Versions + import sanity check |
MCP Resources (read-only state without a tool call):
pyrecrawl://cache/stats · pyrecrawl://sessions · pyrecrawl://monitors
MCP Prompts (ready-made playbooks): research(topic) · rag_ingest(site) · watch_page(url)
| Variable | Default | Effect |
|---|---|---|
PYRECRAWL_CACHE | off | 1 = in-memory LRU (128 pages), or a directory path (reserved for disk mode) |
PYRECRAWL_CACHE_TTL | 900 | Cache entry lifetime in seconds |
PYRECRAWL_MONITOR_DIR | ~/.pyrecrawl/monitors | Where monitor snapshots persist |
PYRECRAWL_NO_TELEMETRY | off | 1 = disable the anonymous startup ping (also honors DO_NOT_TRACK=1) |
prefer options: "auto" (default ladder) · "fast" (HTTP only) · "stealth" (CF bypass) · "llm" (deep processing).
UV is a fast Python package manager that handles Python itself — no need to install Python separatel