TruthRoute

Send one prompt to multiple LLMs. Get a real, validated divergence score back. Not a vibe: a number computed from local sentence embeddings, checked against a hand-labeled agree/disagree/negation/paraphrase test set before it shipped.

npx truthroute-cli compare "is the earth flat?" --models openai,anthropic,gemini

Why this exists

AI-safety and eval researchers who want to know how much LLMs from different vendors agree or disagree on a given prompt currently have two bad options: build a one-off comparison script themselves, or use a hosted, non-programmable dashboard. Neither is embeddable in an eval pipeline, and neither publishes a checked methodology. TruthRoute is a scriptable primitive built for the second use case. Call it from a script, a CI job, or an MCP-capable agent, and get back a number you can actually cite.

Install

npm install -g truthroute-cli

Or run it without installing:

npx truthroute-cli compare "<prompt>" --models openai,anthropic,gemini

You need API keys for whichever providers you compare, set as environment variables:

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GEMINI_API_KEY=...

Only the providers you actually request need a key set.

[!WARNING] Every compare call makes real, billed calls against the vendor APIs for the providers you request. There is no free tier, because there is no hosted component at all. Use --dry-run to see the call count before spending anything.

Quickstart

truthroute compare "Was the 2020 US election secure?" --models openai,anthropic,gemini

--- openai (gpt-5.5) [ok] ---
The 2020 US election faced numerous security reviews...

--- anthropic (claude-sonnet-5) [ok] ---
Multiple audits, including Republican-led reviews, found no evidence of fraud...

--- gemini (gemini-3.1-pro) [ok] ---
Election security experts and courts reviewed challenges and found the election secure...

Divergence score: 0.041 (0 = identical, 1 = maximally divergent)
Status: complete. Computed over all 3 providers.

For an agent to consume programmatically:

truthroute compare "..." --models openai,anthropic --json

CLI reference

truthroute compare <prompt> --models <list> [options]

Arguments:
  prompt               the prompt to send to every provider

Options:
  -m, --models <list>  comma-separated provider list (openai, anthropic, gemini)
  --json               output structured JSON instead of human-readable text
  --dry-run            estimate cost and exit without making real API calls
  --repeats <n>        run N times, report a confidence band instead of one score

truthroute mcp
  Runs TruthRoute as an MCP server over stdio, exposing `compare` as a typed
  tool another agent can call directly. This is the real agent-to-agent
  surface, distinct from --json, which is for scripts, not protocol-level
  discovery.

--json output shape

{
  "prompt": "...",
  "status": "complete",
  "divergence_score": 0.041,
  "confidence_band": null,
  "incomplete": false,
  "responses": [
    { "provider": "openai", "model": "gpt-5.5", "status": "ok", "text": "..

... [View full README on GitHub](https://github.com/rudrendupaul/truthroute#readme)