WebScrape MCP Server

English · Español

English

MCP server that lets AI agents search the web and extract clean Markdown content — no ads, no clutter, just the text your LLM needs.

What's New in v2.0.0

New features:

• JS rendering — Render JavaScript-heavy sites with Playwright (Chromium headless). Auto-fallback when httpx gets 403 or empty content
• Structured data extraction — Extract JSON-LD, Open Graph, Twitter Cards, meta tags, canonical URLs, and hreflang links with extract_schema=True
• Screenshots — New webscrape_screenshot tool captures page screenshots with configurable viewport, full-page mode, and PNG/JPEG format
• Multi-engine search — DuckDuckGo primary, automatic fallback to Google and Bing if DDGS is unavailable
• Smart truncation — Content is truncated at paragraph/sentence boundaries instead of mid-word

Improvements:

• Enhanced cache: 500 entries with 15-minute TTL (was 200, no expiry)
• Better error handling with specific messages for 403, 404, 429, timeouts
• Updated Dockerfile with Chromium dependencies for Playwright

Tools

Features

• PDF support: URLs ending in .pdf or with application/pdf content-type are auto-detected and text is extracted page by page
• Readability mode: Pass use_readability=True to webscrape_fetch_url for cleaner article extraction using Mozilla Readability (removes nav, sidebars, ads, comments)
• JS rendering: Pass js_render=True to render JavaScript-heavy sites with Playwright (headless Chromium). Auto-fallback when httpx gets 403 or empty content
• Structured data extraction: Pass extract_schema=True to extract JSON-LD, Open Graph, Twitter Cards, meta tags, canonical URLs, and hreflang links
• Multi-engine search: DuckDuckGo primary, automatic fallback to Google and Bing if DDGS is unavailable
• Screenshots: Capture page screenshots with configurable viewport, full-page mode, and PNG/JPEG format
• Built-in cache: 500-entry cache with TTL-based eviction (15 min) for repeated URLs
• Batch fetching: Up to 5 URLs in parallel
• Smart truncation: Content is truncated at paragraph/sentence boundaries instead of mid-word

How to use

Option 1 — MCPize (recommended)

1. Go to https://mcpize.com/marketplace
2. Search Web Scrape and click Start Free
3. You'll get an API key
4. Configure in your AI client:

{
  "mcpServers": {
    "webscrape": {
      "url": "https://webscrape.mcpize.run",
      "headers": {
        "Authorization": "Bearer your-api-key"
      }
    }
  }
}

Option 2 — Render (dev)

{
  "mcpServers": {
    "webscrape": {
      "url": "https://webscrape-mcp.onrender.com"
    }
  }
}

Option 3 — Local

git clone https://github.com/carrasquelalex1/webscrape-mcp.git
cd webscrape-mcp
pip install -r requirements.txt
playwright install chromium
python webscrape_mcp.py

Official Registry

io.github.carrasquelalex1/webscrape-mcp

Dependencies

mcp, httpx, beautifulsoup4, markdownify, pydantic, ddgs, readability-lxml, PyMuPDF, playwright

License

MIT

Español

Servidor MCP que permite a agentes de IA buscar en la web y extraer contenido limpio en Markdown — sin anuncios, sin navegación, solo el texto que tu LLM necesita.

Novedades en v2.0.0

Nuevas características:

• **Rendering JS

... View full README on GitHub