Siftdog — open-source, self-hosted Tavily alternative

Website · Benchmark · Blog · Hosted waitlist

Siftdog is an open-source, self-hosted web search API for AI agents and LLM apps, and a drop-in replacement for the Tavily API. It serves the same /search, /extract, /crawl and /map endpoints with the same request and response shapes, so code written for Tavily (including the official Python SDK and the LangChain integration) works against your own server by changing only the base URL. It needs no search API key.

• Search through a SearXNG metasearch instance, with no search API keys
• Extraction of clean markdown or text from web pages with trafilatura
• Ranking: BM25 relevance blended with the upstream engine's order; advanced depth fetches each page and returns its most relevant chunks
• Answers (include_answer) written by Claude from the retrieved results
• Crawling and site maps with depth, breadth, limit and regex path/domain filters
• MCP server for Claude Code, Claude Desktop, Cursor and other MCP clients, over HTTP at /mcp or stdio with siftdog mcp
• SSRF protection: private and internal addresses are blocked, including via redirects and DNS rebinding
• MIT licensed

Use it as a drop-in Tavily replacement

Point the official Tavily clients at your Siftdog server with api_base_url. The key can be any string when auth is disabled, or one of your API_KEYS.

from tavily import TavilyClient

client = TavilyClient(api_key="your-siftdog-key", api_base_url="http://localhost:8000")
results = client.search("latest python release", search_depth="advanced", max_results=5)
pages = client.extract(urls=["https://en.wikipedia.org/wiki/Okapi_BM25"])

LangChain, through langchain-tavily:

from langchain_tavily import TavilySearch

search = TavilySearch(
    max_results=5, tavily_api_key="your-siftdog-key", api_base_url="http://localhost:8000"
)
search.invoke({"query": "what is BM25 ranking"})

Tested with tavily-python 0.8.4 (search, extract, crawl, map; sync and async) and langchain-tavily 0.2.18 (search, extract).

Quick start

Docker Compose (includes SearXNG)

The full stack, with a SearXNG instance for /search, using the published image:

git clone https://github.com/khsarvar/siftdog && cd siftdog
cp .env.example .env          # optional: set API_KEYS and ANTHROPIC_API_KEY
docker compose up

Try a search:

curl -s localhost:8000/search -H "Content-Type: application/json" \
  -d '{"query": "latest python release", "search_depth": "advanced"}'

If you set API_KEYS, add -H "Authorization: Bearer <key>". OpenAPI docs are served at http://localhost:8000/docs.

Docker image only

docker run -p 8000:8000 ghcr.io/khsarvar/siftdog

/extract, /crawl and /map work on their own. For /search, point it at a SearXNG instance with the JSON format enabled: -e SEARXNG_URL=http://your-searxng:8080. Images are published for linux/amd64 and linux/arm64, tagged latest and by version (0.1.0, 0.1).

pip

pip install siftdog
SEARXNG_URL=http://your-searxng:8080 siftdog --port 8000

Configuration is read from environment variables or a .env file (see Configuration).

Us

... View full README on GitHub