Documentation Crawler & MCP Server

This project provides a toolset to crawl websites, generate Markdown documentation, and make that documentation searchable via a Model Context Protocol (MCP) server, designed for integration with tools like Cursor.

Features

• Web Crawler (crawler_cli):
  • Crawls websites starting from a given URL using crawl4ai.
  • Configurable crawl depth, URL patterns (include/exclude), content types, etc.
  • Optional cleaning of HTML before Markdown conversion (removes nav links, headers, footers).
  • Generates a single, consolidated Markdown file from crawled content.
  • Saves output to ./storage/ by default.
• MCP Server (mcp_server):
  • Loads Markdown files from the ./storage/ directory.
  • Parses Markdown into semantic chunks based on headings.
  • Generates vector embeddings for each chunk using sentence-transformers (multi-qa-mpnet-base-dot-v1).
  • Caching: Utilizes a cache file (storage/document_chunks_cache.pkl) to store processed chunks and embeddings.
    • First Run: The initial server startup after crawling new documents may take some time as it needs to parse, chunk, and generate embeddings for all content.
    • Subsequent Runs: If the cache file exists and the modification times of the source .md files in ./storage/ haven't changed, the server loads directly from the cache, resulting in much faster startup times.
    • Cache Invalidation: The cache is automatically invalidated and regenerated if any .md file in ./storage/ is modified, added, or removed since the cache was last created.
  • Exposes MCP tools via fastmcp for clients like Cursor:
    • list_documents: Lists available crawled documents.
    • get_document_headings: Retrieves the heading structure for a document.
    • search_documentation: Performs semantic search over document chunks using vector similarity.
• Cursor Integration: Designed to run the MCP server via stdio transport for use within Cursor.

Workflow

1. Crawl: Use the crawler_cli tool to crawl a website and generate a .md file in ./storage/.
2. Run Server: Configure and run the mcp_server (typically managed by an MCP client like Cursor).
3. Load & Embed: The server automatically loads, chunks, and embeds the content from the .md files in ./storage/.
4. Query: Use the MCP client (e.g., Cursor Agent) to interact with the server's tools (list_documents, search_documentation, etc.) to query the crawled content.

Setup

This project uses uv for dependency management and execution.

1. Install uv: Follow the instructions on the uv website.

2. Clone the repository:

   git clone https://github.com/alizdavoodi/MCPDocSearch.git
   cd MCPDocSearch
   

3. Install dependencies:

   uv sync
   

   This command creates a virtual environment (usually .venv) and installs all dependencies listed in pyproject.toml.

Usage

1. Crawling Documentation

Run the crawler using the crawl.py script or directly via uv run.

Basic Example:

uv run python crawl.py https://docs.example.com

This will crawl https://docs.example.com with default settings and save the output to ./storage/docs.example.com.md.

Example with Options:

uv run python crawl.py https://docs.another.site --output ./storage/custom_name.md --max-depth 2 --keyword "API" --keyword "Reference" --exclude-pattern "*blog*"

View all options:

uv run python crawl.py --help

Key options include:

• --output/-o: Specify output file path.
• --max-depth/-d: Set crawl depth (must be between 1 and 5).
• --include-pattern/--exclude-pattern: Filter URLs to crawl.
• --keyword/-k: Keywords for relevance scoring during crawl.
• --remove-links/--keep-links: Control HTML cleaning.
• --cache-mode: Control crawl4ai caching (`DEF

... View full README on GitHub