TruthRoute

Send one prompt to multiple LLMs. Get a real, validated divergence score back. Not a vibe: a number computed from local sentence embeddings, checked against a hand-labeled agree/disagree/negation/paraphrase test set before it shipped.

npx truthroute-cli compare "is the earth flat?" --models openai,anthropic,gemini

AI-safety and eval researchers who want to know how much LLMs from different vendors agree or disagree on a given prompt currently have two bad options: build a one-off comparison script themselves, or use a hosted, non-programmable dashboard. Neither is embeddable in an eval pipeline, and neither publishes a checked methodology. TruthRoute is a scriptable primitive built for the second use case. Call it from a script, a CI job, or an MCP-capable agent, and get back a number you can actually cite.

npx truthroute-cli compare "<prompt>" --models openai,anthropic,gemini

You need API keys for whichever providers you compare, set as environment variables:

export OPENAI_API_KEY=sk-... export ANTHROPIC_API_KEY=sk-ant-... export GEMINI_API_KEY=...

Only the providers you actually request need a key set.

[!WARNING] Everycomparecall makes real, billed calls against the vendor APIs for the providers you request. There is no free tier, because there is no hosted component at all. Use--dry-runto see the call count before spending anything.

truthroute compare "Was the 2020 US election secure?" --models openai,anthropic,gemini

--- openai (gpt-5.5) [ok] --- The 2020 US election faced numerous security reviews... --- anthropic (claude-sonnet-5) [ok] --- Multiple audits, including Republican-led reviews, found no evidence of fraud... --- gemini (gemini-3.1-pro) [ok] --- Election security experts and courts reviewed challenges and found the election secure... Divergence score: 0.041 (0 = identical, 1 = maximally divergent) Status: complete. Computed over all 3 providers.

For an agent to consume programmatically:

truthroute compare "..." --models openai,anthropic --json

truthroute compare <prompt> --models <list> [options] Arguments: prompt the prompt to send to every provider Options: -m, --models <list> comma-separated provider list (openai, anthropic, gemini) --json output structured JSON instead of human-readable text --dry-run estimate cost and exit without making real API calls --repeats <n> run N times, report a confidence band instead of one score truthroute mcp Runs TruthRoute as an MCP server over stdio, exposing compare as a typed tool another agent can call directly. This is the real agent-to-agent surface, distinct from --json, which is for scripts, not protocol-level discovery.

{ "prompt": "...", "status": "complete", "divergence_score": 0.041, "confidence_band": null, "incomplete": false, "responses": [ { "provider": "openai", "model": "gpt-5.5", "status": "ok", "text": "...", "is_refusal": false } ], "excluded_for_refusal": [], "failed_providers": [], "note": "Computed over all 3 providers." }

statusis one ofcomplete(all providers succeeded),partial(at least 2 usable responses, but not all providers succeeded, or one was excluded for refusal), orfailed(fewer than 2 usable responses, sodivergence_scoreisnull; divergence has no meaning against a single data point).

- Scoring:local sentence embeddings (fastembed, modelBGESmallENV15). No paid API for scoring, only the 3 providers being compared. Divergence is1 - average pairwise cosine similarityacross all response pairs, in[0.0, 1.0].
- Validated, not assumed.The model was checked against a hand-labeled test set (test/fixtures/validation-set.json) covering agreement, paraphrase, negation, and clear disagreement before shipping. A smaller embedding model (MiniLM-L6) was tried first and rejected during that check: it scored negation pairs aslessdivergent than paraphrases, the opposite of correct.BGESmallENV15was chosen because it passes that check.
- Refusals are excluded from scoring, not just flagged. A refusal's text distance from a real answer is not factual disagreement, and would otherwise dominate the score.
- Responses are normalized before scoring(markdown and formatting stripped) so verbosity differences between providers aren't measured as semantic divergence.
- Determinism, stated plainly:all provider calls usetemperature=0, which reduces but does not eliminate run-to-run variance. Vendor-side inference infrastructure (GPU batching, floating-point non-associativity) can still cause drift independent of anything this tool controls. Use--repeats Nto get a confidence band instead of trusting a single score as exactly reproducible.

- A compressed score range is expected, not a bug.Cosine-similarity scores between two responses to the same topically-related prompt naturally compress into a smaller range than a naive 0-to-1 intuition suggests. The signal that matters is relative ordering (agreement scores lower than disagreement), which is what the validation set actually checks.

duhis a full multi-model consensus platform: a propose/challenge/revise/commit debate protocol across 5 providers plus local models, with a web UI, REST API, WebSocket streaming, persistent SQLite/Postgres storage, auth, cost tracking, and PDF export. It is more mature and far more feature-complete than TruthRoute. TruthRoute is not trying to be a smaller version of it. TruthRoute does one narrow thing: score how much N providers' responses to the same prompt diverge, as a stateless CLI/MCP primitive with no server, no database, and no accounts to set up. If you want debate, dissent-tracking, and a full decision-audit platform, useduh. If you want a scriptable divergence number to drop into an existing eval pipeline or CI job with nothing to host, that is what TruthRoute is for.

TruthRoute is not an LLM gateway or router (seeLiteLLMandPortkey). It does no routing, failover, or cost optimization. If you need those, use one of those tools. TruthRoute measures disagreement between providers; it doesn't route between them.

Anchor Browser (https://anchorbrowser.io) is secure infrastructure for computer-use agents — stealth cloud browsers, authentication, captcha bypass, and a hosted MCP server for Cursor, Claude, and Windsurf.