SearXNG + Crawl4AI MCP Server

A self-hosted MCP (Model Context Protocol) server providing fast search and reliable web scraping using SearXNG + Crawl4AI stack.

🚀 Why This Solution?

This project evolved from limitations found in self-hosted Firecrawl:

• ❌ Firecrawl's search API doesn't work in self-hosted mode
• ❌ Missing Fire-engine features in self-hosted version
• ❌ Authentication issues and poor documentation

Our solution provides:

• ✅ Truly self-hosted search via SearXNG (aggregates 70+ search engines)
• ✅ Superior scraping via Crawl4AI (50k+ GitHub stars)
• ✅ 3x faster than Claude Code native search tools
• ✅ 100% reliable vs failing native WebFetch
• ✅ Complete privacy - no external API dependencies

🏗️ Architecture

┌─────────────┐    ┌──────────────┐    ┌─────────────┐
│             │    │              │    │             │
│  SearXNG    │    │  Crawl4AI    │    │   Redis     │
│  (Search)   │    │  (Scraping)  │    │  (Cache)    │
│             │    │              │    │             │
│  Port 8081  │    │  Port 8001   │    │ Port 6380   │
└─────────────┘    └──────────────┘    └─────────────┘
        │                   │                   │
        └───────────────────┼───────────────────┘
                           │
                  ┌──────────────┐
                  │              │
                  │ MCP Server   │
                  │ (TypeScript) │
                  │              │
                  └──────────────┘
                           │
                    ┌─────────────┐
                    │             │
                    │ Claude Code │
                    │             │
                    └─────────────┘

📦 Features

• 🔍 Fast Search: SearXNG aggregates 70+ search engines (Google, Bing, DuckDuckGo, etc.)
• 🕷️ Advanced Scraping: Crawl4AI with Playwright for JavaScript-heavy sites
• ⚡ High Performance: Sub-second search, reliable scraping
• 🐳 Docker Ready: Complete Docker Compose orchestration
• 🔄 Proxy Support: Built-in rotating IP proxy integration
• 📊 MCP Integration: 3 powerful tools for Claude Code
• 🛡️ Privacy First: All processing happens locally

🚀 Quick Start

1. Clone and Setup

git clone https://github.com/yourusername/searxng-crawl4ai-mcp
cd searxng-crawl4ai-mcp
npm install
npm run build

2. Start Docker Services

# Start all services (SearXNG, Crawl4AI, Redis)
docker compose up -d

# Verify services are running
curl http://localhost:8081/search?q=test&format=json  # SearXNG
curl http://localhost:8001/health                      # Crawl4AI

3. Configure Claude Code MCP

Simple Configuration (No Proxy):

{
  "mcpServers": {
    "searxng-crawl4ai": {
      "command": "node",
      "args": ["fixed-mcp-server.js"],
      "cwd": "/absolute/path/to/your/project"
    }
  }
}

With Proxy Configuration:

{
  "mcpServers": {
    "searxng-crawl4ai": {
      "command": "node",
      "args": ["fixed-mcp-server.js"],
      "cwd": "/absolute/path/to/your/project",
      "env": {
        "PROXY_URL": "http://username:password@your-proxy-server.com:10000"
      }
    }
  }
}

4. Increase Token Limits (Recommended)

Create .claude/settings.json:

{
  "environmentVariables": {
    "MAX_MCP_OUTPUT_TOKENS": "100000"
  }
}

🛠️ Available MCP Tools

1. search_web - Lightning Fast Search

{
  "query": "latest AI developments 2025",
  "maxResults": 10
}

Returns: 30+ search results in <1 second from multiple engines

2. crawl4ai_scrape - Advanced Web Scraping

{
  "url": "https://finance.yahoo.com/quote/BTC-USD/",
  "formats": ["markdown"]
}

Returns: Full page content with metadata (title, word count, clean markdown)

3. search_and_scrape - Combined Power Workflow

{
  "query": "Bitcoin technical analysis September 2025",
  "maxResults": 2
}

Returns: Search

... View full README on GitHub