TokenTrust
About
Verifies token/cost savings claimed by AI context-reduction proxies via MCP tools.
Details
- Author
- rudrendupaul
- Categories
- Developer Tools
Jump to
Setup
Install TokenTrust in your MCP client (Claude Desktop, Cursor, Windsurf, and others).
Repository: https://github.com/rudrendupaul/TokenTrust
Follow the installation instructions in the repository README, then restart your MCP client.
What is TokenTrust, and why does it exist
TokenTrust is a command-line tool that measures whether an AI-coding-agent context-reduction proxy's advertised token and cost savings hold up against a real, labeled task corpus, run with a local tokenizer instead of a spreadsheet estimate. It exists because compression proxies currently self-report their own savings numbers, and there is no independent, repeatable, CI-native way to check one before adopting it. TokenTrust is not a proxy itself and does not compress anything. It verifies proxies that do.
TokenTrust's own validation work has already fed back into a real, independently tracked GitHub issue:](https://github.com/RudrenduPaul/TokenTrust-CLI/commit/e42246c)rtk-ai/rtk#1313(filed by @ChrisEdwards, asking rtk for a lossless-only mode and an honest account of the silent failures truncation causes in agent contexts) was originally verified as only partially addressed by rtk's existing mechanism, because TokenTrust's own fixtures didn't yet carry the quality markers needed to prove it either way. Extending three of TokenTrust'spipe --filterfixtures with real, verified quality markers closed that gap in TokenTrust's own instrumentation, not in rtk, and let the tool confirm, against the real rtk 0.43.0 binary and not a claim, that rtk's existing never-worse guard mechanism already does what the issue asked for. The issue's verdict moved from partial to a genuine, re-verified pass as a direct result. TokenTrust never touched rtk's own repository; it got sharp enough to prove what was already true there.
pip install tokentrust-cliinstalls the sametokentrustCLI as a genuine Python port, not a wrapper around the Node binary: real Python source underpython/src/tokentrust/, its own pytest suite, and the identical bundled 23-task corpus, copied verbatim into the wheel. Both distributions run the samecl100k_basetokenizer encoding, verified to produce identical token counts on real sample text, and both are maintained together going forward, includingtokentrust mcp: the Python port exposes the sameverify_proxy_savingsMCP tool, with a byte-identical wire schema, as the npm package (seepython/README.md's "Agent-native / MCP" section). Seepython/README.mdfor install instructions,python/docs/getting-started.mdfor a walkthrough, andpython/docs/concepts.mdfor the verification methodology shared by both packages.
What is TokenTrust, and how is it different from a context-reduction proxy like rtk or headroom?TokenTrust is not a proxy itself and does not compress anything. It is a vendor-neutral verification layer: it runs a proxy likertkorheadroomas a real subprocess against a fixed, labeled 23-task corpus, measures the actual token and dollar savings with a local tokenizer, and prints that measured number next to the number the proxy's own README claims. The differentiator is independence: TokenTrust has no stake in whether a proxy's claimed number holds up, so it never averages the gap away.
Which platforms does TokenTrust run on, and are the npm and PyPI packages the same tool?Both are genuine, separately maintained ports of the same tool, not one wrapping the other.npm install -g tokentrust-cli(ornpx tokentrust-cli) installs the Node.js build;pip install tokentrust-cliinstalls a real Python port underpython/src/tokentrust/, with its own pytest suite. Both expose the sametokentrustcommand, the same TT01-TT05 categories, the same bundled task corpus, and the samecl100k_basetokenizer encoding.
Does TokenTrust work with AI agents directly, not just from a shell?Yes.tokentrust mcp(ornpx tokentrust-cli mcp) starts an MCP (Model Context Protocol) server over stdio that exposes one tool,verify_proxy_savings, backed by the samerunVerify()engine the CLI uses. Any MCP-compatible client, including Claude Code and Claude Desktop, can call that tool and get back the same structured JSON report--format jsonproduces on the command line.
How does TokenTrust compare to tokbench, the other independent proxy benchmark?tokbenchis real prior art and deserves credit: a rigorous, disclosed pilot with raw transcripts and a pre-registered protocol. Its current scope is narrower than a first read suggests, one repository, one task, N=1 per arm, with replication in progress. TokenTrust instead runs a 23-task corpus continuously, in your own CI, on your own repo, every time a proxy version bumps, rather than as a single published pilot report.
Thetokentrust verify --helptext on the npm package says the default task corpus has 15 tasks. Which is correct, 15 or 23?23 is correct. The bundled corpus was expanded from 15 to 23 tasks incommit e42246c, and every real run (both npm and PyPI) reports "Task corpus: 23 labeled tasks" at the top of its output, matching the actualfixtures/tasks.ymlfile. The npm package's--tasksflag help text is a leftover string from before that expansion and hasn't been updated to say 23; the PyPI package's help text already says 23 correctly. This affects only what the--helptext displays, not what tasks the CLI actually runs.
What ifpip install tokentrust-clifails on my Python version?The PyPI package declaresrequires-python = ">=3.10"in itspyproject.toml, sopipwill refuse to install it on Python 3.9 or older. Upgrade to Python 3.10, 3.11, 3.12, or 3.13 (the versions the package is tested against), or use the npm package instead, which only requires Node.js 18 or newer.
Can I use TokenTrust commercially, and do I need to attribute it?Yes. TokenTrust is licensed Apache-2.0 (seeLICENSE), which permits commercial use, modification, and distribution, including inside closed-source products, as long as you keep the license and copyright notice and note any changes you made to the source itself.
Does TokenTrust modify my code or my proxy's compressed output?No. It runs the proxy as a real subprocess against fixture tasks, captures the output, and measures it. Nothing in your repo or the proxy's configuration is changed.
Can TokenTrust verify a proxy's live, provider-billed cost instead of an estimate?Yes, with--live --confirm-cost, capped at 5 tasks by default via--live-max-tasks. It uses your own API key and never runs a real charge without printing the estimated spend first.
Is TokenTrust importable as a library, or is it CLI-only?CLI-only today. Neither the npm package nor the PyPI package ships a documented, importable public API: the npmpackage.jsonlists amainentry that points at a file the published package doesn't actually contain, and the PyPI package's top-level module only exports a version string. Use thetokentrustcommand or theverify_proxy_savingsMCP tool; there is no supported way toimport/requirethis package's verification logic directly yet.
This is a web browser that enables your coding agent, such as Claude Code, to visit websites on your behalf and assist you in identifying bugs or creating UI test cases.
Create crafted UI components inspired by the best 21st.dev design engineers.
Bring agent evaluations, observability, and synthetic test set generation directly into your IDE for free with Galileo's new MCP server
An MCP server to help AI assistants to answer questions and generate AccelByte Extend SDK code more effectively .
MCP server for AI Diagram Maker — generate beautiful software engineering diagrams directly inside Cursor, Claude Desktop, Claude Code, or any MCP-compatible AI agent
ALAPI MCP Tools,Call hundreds of API interfaces via MCP
AI-powered SVG animation generator that transforms static files into animated SVG components using the Allyson platform
MCP server that gives AI assistants on-demand access to 1,500+ amCharts docs, ~300 code examples, and 1000+ class API references.
APIMatic MCP Server is used to validate OpenAPI specifications using APIMatic. The server processes OpenAPI files and returns validation summaries by leveraging APIMatic’s API.
One shared context layer for AI agents and humans — live API specs, DB schemas, and versioned contracts across repos so every agent and teammate works from the same source of truth.
Build and deploy full-stack Next.js apps with 98 tools for React, AWS, and MongoDB
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
