🐳 Crawl4AI+SearXNG MCP Server

Web Crawling, Search and RAG Capabilities for AI Agents and AI Coding Assistants

(FORKED FROM https://github.com/coleam00/mcp-crawl4ai-rag). Added SearXNG integration and batch scrape and processing capabilities.

A self-contained Docker solution that combines the Model Context Protocol (MCP), Crawl4AI, SearXNG, and Supabase to provide AI agents and coding assistants with complete web search, crawling, and RAG capabilities.

🚀 Complete Stack in One Command: Deploy everything with docker compose up -d - no Python setup, no dependencies, no external services required.

🎯 Smart RAG vs Traditional Scraping

Unlike traditional scraping (such as Firecrawl) that dumps raw content and overwhelms LLM context windows, this solution uses intelligent RAG (Retrieval Augmented Generation) to:

• 🔍 Extract only relevant content using semantic similarity search
• ⚡ Prevent context overflow by returning focused, pertinent information
• 🧠 Enhance AI responses with precisely targeted knowledge
• 📊 Maintain context efficiency for better LLM performance

Flexible Output Options:

• RAG Mode (default): Returns semantically relevant chunks with similarity scores
• Raw Markdown Mode: Full content extraction when complete context is needed
• Hybrid Search: Combines semantic and keyword search for comprehensive results

💡 Key Benefits

• 🔧 Zero Configuration: Pre-configured SearXNG instance included
• 🐳 Docker-Only: No Python environment setup required
• 🔍 Integrated Search: Built-in SearXNG for private, fast search
• ⚡ Production Ready: HTTPS, security, and monitoring included
• 🎯 AI-Optimized: RAG strategies built for coding assistants

Overview

This Docker-based MCP server provides a complete web intelligence stack that enables AI agents to:

• Search the web using the integrated SearXNG instance
• Crawl and scrape websites with advanced content extraction
• Store content in vector databases with intelligent chunking
• Perform RAG queries with multiple enhancement strategies

Advanced RAG Strategies Available:

• Contextual Embeddings for enriched semantic understanding
• Hybrid Search combining vector and keyword search
• Agentic RAG for specialized code example extraction
• Reranking for improved result relevance using cross-encoder models
• Knowledge Graph for AI hallucination detection and repository code analysis

See the Configuration section below for details on how to enable and configure these strategies.

Features

• Smart URL Detection: Automatically detects and handles different URL types (regular webpages, sitemaps, text files)
• Recursive Crawling: Follows internal links to discover content
• Parallel Processing: Efficiently crawls multiple pages simultaneously
• Content Chunking: Intelligently splits content by headers and size for better processing
• Vector Search: Performs RAG over crawled content, optionally filtering by data source for precision
• Source Retrieval: Retrieve sources available for filtering to guide the RAG process

Tools

The server provides essential web crawling and search tools:

Core Tools (Always Available)

1. scrape_urls: Scrape one or more URLs and store their content in the vector database. Supports both single URLs and lists of URLs for batch processing.
2. smart_crawl_url: Intelligently crawl a full website based on the type of URL provided (sitemap, llms-full.txt, or a regular webpage that needs to be crawled recursively)
3. get_available_sources: Get a list of all available sources (domains) in the database
4. perform_rag_query: Search for relevant content usin

... View full README on GitHub