tokentoll

Prevent LLM cost regressions before production.

tokentoll is a CI gate for LLM cost. It statically analyzes Python, JavaScript, and TypeScript for LLM API calls, scores every pull request against a policy you control, and posts a PASS/WARN/FAIL verdict directly on the PR. Optionally, it fails the workflow when the policy is violated, so cost regressions cannot be merged.

Jwrede/tokentoll-demois a small polyglot LLM app (Python + TypeScript) wired up to the tokentoll cost gate. Two PRs are already open against it:

- PR #1: Add Anthropic Haiku translation helper. New call site, well within budget. Verdict: PASS, workflow green.
- PR #2: switch supportbot to gpt-4o. A model swap that trips two policy rules. Verdict: FAIL, workflow red.

## tokentoll verdict: FAIL Blocking findings (2): - src/agent.py:42 - per-call cost grew 15.0x (threshold 5x) - total monthly delta +$812.00 exceeds budget $250.00 > Required action: revert the regression, raise the threshold in .tokentoll.yml, or add an exemption.

name: tokentoll on: pull_request: paths: - ".py" - ".ts" - ".tsx" - ".js" - ".jsx" permissions: contents: read pull-requests: write jobs: cost-gate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - uses: Jwrede/tokentoll@v0.7.0 with: fail-on-policy-violation: true

budgets: max_monthly_delta_usd: 250 max_callsite_monthly_usd: 100 max_relative_increase: 5.0 policies: block_unknown_models: true fail_on_policy_violation: true

pip install tokentoll # Scan current directory for LLM API calls and their costs tokentoll scan . # Show cost impact of your last commit tokentoll diff HEAD~1 # Compare two refs and fail on policy violation tokentoll diff main..HEAD --fail-on-policy-violation

tokentoll scan [PATH...] [--format table|json|markdown] [--calls-per-month N] [--config PATH] tokentoll diff [REF] [--base REF] [--head REF] [--format table|json|markdown|github-comment] [--config PATH] [--fail-on-policy-violation] tokentoll update # refresh bundled pricing data from LiteLLM

# Per-SDK defaults for dynamic (runtime-resolved) model names default_models: openai: gpt-4o-mini anthropic: claude-haiku-3-20240307 # Assumed monthly call volume per call site (used for dollar estimates) calls_per_month: 5000 # Skip cost estimation for dynamic models entirely. # Default false: dynamic calls are priced against the per-SDK default. skip_dynamic_models: false # Default excludes (tests/, examples/, docs/, cookbook/, benchmarks/, evals/, # scripts/, notebooks/) are applied automatically. Opt out with: use_default_excludes: false # Additional excludes (prefix or glob) exclude: - "*_test.py" - vendor/ # Per-path overrides (longest prefix match) overrides: - path: src/agents/ default_model: gpt-4o calls_per_month: 10000 - path: src/azure/ skip_dynamic_models: true

pip install tokentoll[mcp] claude mcp add --transport stdio tokentoll -- tokentoll-mcp

Source code (.py, .ts, .tsx, .js, .jsx) | v +----------------+ +------------------+ | AST scanners |-->| SDK detectors | | ast (Python) + | | OpenAI, Anthropic| | tree-sitter | | Google, LiteLLM, | | (JS/TS) | | LangChain, Zhipu,| +----------------+ | Vercel AI SDK | +------------------+ | v +------------------+ | Pricing engine | | 2200+ models | +------------------+ | v +------------------+ | Diff engine | | (old vs new) | +------------------+ | v +------------------+ | Policy evaluator | | PASS/WARN/FAIL | +------------------+ | v +------------------+ | PR comment / CLI | | output | +------------------+

A multi-pass constant propagation engine resolves model names through variable assignments,os.getenv()/process.env.Xfallbacks, function defaults, class attributes, constructor arguments, dict and object literals,kwargsunpacking, and Vercel AI SDK provider wrappers (openai("gpt-4o")), so real-world code with indirection still produces useful estimates.

Pricing is bundled and works offline. To refresh from LiteLLM:

Coverage: 300+ models across OpenAI, Anthropic, Google, AWS Bedrock, Azure, and more, plus 2200+ entries from LiteLLM's combined catalog.

- Static analysis only. Models loaded from databases or remote config cannot be resolved; tokentoll falls back to the configured per-SDK default and marks the call site as(default).
- Token estimates use a characters/4 heuristic unlesstiktokenis installed (pip install tokentoll[tiktoken]).
- Monthly estimates assume uniform call volume per call site. Override per-project withcalls_per_monthor per-path withoverrides.
- JS/TS resolution is same-file only. Importing a model name from another module produces a dynamic call site rather than a resolved value.

- v0.9: Public demo repo with a known-failing PR, gpt-researcher case study, expanded adoption section
- Future: Context-aware call frequency inference (FastAPI routes versus scripts versus loops); cross-file import resolution for JS/TS

This is a web browser that enables your coding agent, such as Claude Code, to visit websites on your behalf and assist you in identifying bugs or creating UI test cases.

Drop-in MCP proxy that losslessly re-encodes JSON tool responses as GCF. 71% fewer tokens, 100% comprehension across 1,700+ LLM evaluations. Zero code changes.

a complete and intuitive SDK for building MCP Servers, MCP Agents, and LLM integrations (OpenAI, Claude, Gemini) with minimal effort. It abstracts all the complexity of the MCP protocol, provides an intelligent agent with automatic model routing, and includes a universal client for external APIs all through a single, simple, and powerful interface. Perfect for chatbots, enterprise automation, internal system integrations, and rapid development of MCP-based ecosystems.

A lightweight, LLM-agnostic RESTful proxy that unifies multiple MCP servers under a single API.

A lightweight, LLM-agnostic RESTful proxy that unifies multiple MCP servers under a single API.

An open-source library to connect any LLM to any MCP server, enabling the creation of custom agents with tool access.

One shared context layer for AI agents and humans — live API specs, DB schemas, and versioned contracts across repos so every agent and teammate works from the same source of truth.

The MCP server for Bitrix24 provides AI assistants with structured access to the Bitrix24 API. It delivers up-to-date method descriptions, parameters, and valid values, allowing assistants to work with precise data instead of guesswork. This reduces code errors and accelerates Bitrix24 integration development.

Local-first knowledge graph for developers. Watches project files, extracts entities and relationships via LLMs, and lets you query across projects with natural language and source citations.

Tool platform by IBM to build, test and deploy tools for any data source