tokensave
About
Supercharge your Agent with Semantic Code Intelligence and save π° in the process!
Details
- Author
- aovestdipaperino
- Categories
- Developer Tools, AI
Jump to
Setup
Install tokensave in your MCP client (Claude Desktop, Cursor, Windsurf, and others).
Repository: https://github.com/aovestdipaperino/tokensave
Follow the installation instructions in the repository README, then restart your MCP client.
-
Semantic code searchβ Ask for symbols by meaning, e.g. "find authentication code", andtokensave_searchreturns matches likelogin,validateToken, andAuthServicewithout grepping files.
Impact analysisβ Before editing a function, asktokensave_callersortokensave_impactto trace every caller and callee so you know exactly what breaks.
Context buildingβ Usetokensave_contextto get entry points, related symbols, and code snippets for a task in one call instead of reading multiple files.
Cross-branch comparisonβ Querytokensave_branch_diffto see symbols added, removed, or changed between two git branches without switching checkouts.
Session memoryβ Save design decisions withtokensave_record_decisionand recall them later viatokensave_session_recallso the agent doesn't re-explain architecture choices.
Atomic code editsβ Apply unique-anchor replacements withtokensave_str_replaceor multi-file edits viatokensave_multi_str_replace, avoiding regex and shell-quoting hazards.
Semantic Code Intelligence for AI Coding Agents
Fewer tokens β’ Fewer tool calls β’ 100% local
AI coding agents waste tokens exploring codebases. Every grep, glob, and file read costs money. On complex tasks, agents spawn multiple Explore sub-agents that scan hundreds of files just to build context.
tokensave gives agents a pre-indexed semantic knowledge graph.Instead of scanning files, the agent queries the graph and gets instant, structured answers -- the right symbols, their relationships, and source code, in one call.
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β AI Coding Agent (Claude Code, Codex, Gemini, Cursor, ...) β β β β "Implement user authentication" β β β β β βΌ β β βββββββββββββββββββ βββββββββββββββββββ β β β Sub-agent β βββββ β Sub-agent β β β ββββββββββ¬βββββββββ βββββββββββ¬ββββββββ β βββββββββββββΌβββββββββββββββββββββββββββΌββββββββββββββββββββββββ β β βΌ βΌ ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β tokensave MCP Server β β βββββββββββββββ βββββββββββββββ βββββββββββββββ β β β Search β β Callers β β Context β β β β "auth" β β "login()" β β for task β β β ββββββββ¬βββββββ ββββββββ¬βββββββ ββββββββ¬βββββββ β β ββββββββββββββββββΌβββββββββββββββββ β β βΌ β β βββββββββββββββββββββββββ β β β libSQL Graph DB β β β β β’ Instant lookups β β β β β’ FTS5 search β β β βββββββββββββββββββββββββ β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Without tokensave:Agents usegrep,glob, andReadto scan files -- many API calls, high token usage.
With tokensave:Agents query the graph via MCP tools -- instant results, local processing, fewer tokens.
brew install aovestdipaperino/tap/tokensave
scoop bucket add tokensave https://github.com/aovestdipaperino/scoop-bucket scoop install tokensave
cargo install tokensave # full (50+ languages, default) cargo install tokensave --features medium # medium tier cargo install tokensave --no-default-features # lite (smallest binary)
Prebuilt binaries (Linux, Windows, macOS):
Download from thelatest releaseand place the binary in yourPATH.
tokensave install # auto-detects installed agents tokensave install --agent antigravity # Google Antigravity (formerly Windsurf) tokensave install --agent auggie # AugmentCode tokensave install --agent claude # Claude Code tokensave install --agent cline # Cline tokensave install --agent codex # OpenAI Codex CLI tokensave install --agent copilot # GitHub Copilot tokensave install --agent cursor # Cursor tokensave install --agent droid # Factory Droid tokensave install --agent gemini # Gemini CLI tokensave install --agent kilo # Kilo CLI tokensave install --agent kiro # AWS Kiro tokensave install --agent kimi # Moonshot Kimi CLI tokensave install --agent opencode # OpenCode tokensave install --agent pi # Pi (pi.dev) tokensave install --agent plank # Plank (macOS only) tokensave install --agent qwen # Qwen Code tokensave install --agent roo-code # Roo Code tokensave install --agent vibe # Mistral Vibe tokensave install --agent zed # Zed tokensave install --agent grok # Grok Build (xAI) tokensave install --git-hook yes # auto-install the global post-commit and post-checkout hooks (no prompt) tokensave install --git-hook no # skip the post-commit and post-checkout hooks (no prompt)
Each agent gets its MCP server registered in the native config format. Claude Code additionally gets a PreToolUse hook (blocks wasteful Explore agents), a UserPromptSubmit hook, a Stop hook, prompt rules in CLAUDE.md, and auto-allowed tool permissions. Kiro gets global MCP config,tokensave.mdsteering loaded as a resource, and a tokensave-managed default agent with permissive built-in/tokensave tool approval, delegation guardrail hooks, and post-write sync; user-managed Kiro agents are preserved.
All changes are idempotent -- safe to run again after upgrading. After agent setup, you'll be offered global git post-commit and post-checkout hooks.
By defaulttokensave installregisters the MCP server in yourglobalagent config (e.g.~/.claude.json). To register tokensave for just the current project instead, add--local:
tokensave install --local --agent claude
This writes project-scoped config you can commit and share with your team. For Claude that's./.mcp.json,./.claude/settings.json, and./CLAUDE.md. Supported agents:claude, cursor, droid, gemini, zed, opencode, roo-code, kiro, auggie, plank(each writes its own project file, e.g..cursor/mcp.json,.factory/mcp.json,.gemini/settings.json,.zed/settings.json,opencode.json,.roo/mcp.json,.kiro/settings/mcp.json,.augment/settings.json,.mcp.jsonfor plank). Other agents have no project-scoped config and report an error with--local.
Remove a project-local install withtokensave uninstall --local.
This creates a.tokensave/directory with the knowledge graph database. Initialization and sync are separate commands:initis a one-time opt-in per project, whilesynconly updates projects that were already initialized. This prevents the global git hooks from silently creating databases in repos you never intended to index. Afterinit, usetokensave syncto incrementally update -- only changed files are re-indexed.
{ "mcpServers": { "tokensave": { "command": "/path/to/tokensave", "args": ["serve"] } } }
The hook runstokensave hook-pre-tool-use-- a native Rust command (no bash or jq required). It intercepts Agent, Grep, Glob, and Bash tool calls: Explore agents are blocked outright, symbol-shaped grep/rg/ag invocations (plain identifiers, alternations,\b-wrapped names) are redirected to the matching tokensave MCP tool, and path-shaped discovery (Glob,find -name,fd --extension) over code extensions is redirected totokensave_files. Regex patterns,git grep, piped commands, non-code extensions, search roots outside the index, andfindpredicates that change what the command does (-exec,-delete,-mtime) all pass through untouched; setTOKENSAVE_DISABLE_GREP_HOOK=1to opt out per shell.
Filters are read most-specific-first: an explicittypeis authoritative, then an explicit file glob, then the search path. A documentation search such aspath: "."withglob: "/.md"therefore passes through rather than being treated as a code search on the broad path, while a code-only glob (/.rs) still redirects even under a non-code path. Mixed globs (/.{rs,md}) pass through, since they can return documentation.
Headless / subagent dispatch (claude -p).Child processes dispatched by an orchestrating session inherit its~/.claude/settings.json, including this hook. To let a child run raw searches, setTOKENSAVE_DISABLE_GREP_HOOK=1in the child's environment -- the native binary honors it and passes every path (Grep, Glob, Bash, Agent) through, so there is no need for the blunt--settings '{"hooks": {}}'that stripsallhooks. The guardrail is stateless: it never consults citation history, so it only ever redirects the symbol-shaped searches described above and steers untyped research fan-out; ordinary commands are unaffected whether the session is interactive or headless.
Appends instructions to~/.claude/CLAUDE.mdthat tell Claude to use tokensave tools before reaching for Explore agents or raw file reads.
Tree-sitter grammars are compiled C/C++ code. They occasionally hit an internal assertion or otherwise terminate the process by paths that Rust panic handling cannot intercept. As of v4.3.0, every file is parsed inside a short-lived worker subprocess: if a grammar segfaults, callsabort(), or hits a stack overflow, only the worker dies. The pool respawns it, the offending file is logged and skipped, andsynckeeps going.
The worker is a hiddenextract-workersubcommand authenticated against the parent via a 256-bit per-spawn token, required as both aTOKENSAVE_WORKER_TOKENenv var and as the first 32 bytes received on stdin. Direct invocation by users fails. Defaults toavailable_parallelism()workers; opt out withTOKENSAVE_DISABLE_SUBPROCESS=1.
Edit primitives (tokensave_str_replace,tokensave_insert_at, etc.) still run in-process: they target one file at a time where subprocess overhead would dominate, and an extractor crash there is immediately visible to the agent.
tokensave can optionally maintain a separate code graph per git branch. When enabled, switching branches never gives you stale results and never re-indexes files you already parsed on another branch. Multi-branch tracking is opt-in -- without it, tokensave uses a single database for all branches.
When you track a branch, tokensave copies the nearest ancestor DB and syncs only the files that differ. This means tracking a feature branch offmainis nearly instant -- it only parses the files you've changed.
tokensave branch add # track the current branch tokensave branch list # see tracked branches and DB sizes tokensave branch remove <name> # stop tracking a branch tokensave branch removeall # remove all tracked branches except default tokensave branch gc # clean up branches deleted from git
Three MCP tools enable cross-branch queries without switching your checkout:
- tokensave_branch_search-- search symbols in another branch's graph
- tokensave_branch_diff-- compare code graphs between two branches: symbols added, removed, and changed (signature differs). Supports file and kind filters.
- tokensave_branch_list-- list tracked branches with DB sizes, parent branch, and sync times
When the MCP server can't find a database for the current branch, it serves from the nearest ancestor branch's DB and includes a warning in every tool response suggesting you runtokensave branch add.
Once multi-branch mode is bootstrapped (a first manualtokensave branch addcreated the branch metadata), new branches can be tracked automatically instead of falling back to the ancestor DB. Two independent mechanisms cover this; projects in single-DB mode are never affected, and neither mechanism ever touches the default branch's database.
Git hook (on branch checkout).Thepost-checkouthook thattokensave installsets up recognizes abranchcheckout (as opposed to a file checkout) and runstokensave branch addin the background. That command is a no-op when the branch is already tracked or is the default branch, so ordinary switching between known branches costs nothing. The initial checkout of a freshgit cloneand of a newgit worktree addis a branch checkout too, and it can land on a branch that is not the default one (git clone -b feature,git worktree add -b feature); there the hook runstokensave initfirst andtokensave branch addafter it, in that order. A hook written by an earlier version keeps the body it was installed with β the installer never rewrites an existing one β so on those installs a fresh worktree still needsauto_trackbelow, or a manualtokensave branch add.
Open-time auto-track (opt-in).WhenTokenSave::openruns β CLI command or MCP server start β and the active branch is untracked, tokensave can track it on the spot by copying the nearest tracked ancestor's DB and recording it in the branch metadata. This is gated by theauto_trackconfig field (defaultfalse) or theTOKENSAVE_AUTO_TRACKenvironment variable, which overrides the config per-run (any value enables it except0,false,no,off, or empty). The copy is the same near-instant ancestor-DB copy a manualbranch addperforms; no sync runs at that moment β thepost-commithook keeps the new branch DB fresh as you commit, or runtokensave syncto refresh immediately. Auto-tracking is strictly best-effort: any failure is reported as a warning andopen()proceeds with the usual ancestor fallback, so it can never break a tool call.
In short: with the hook installed, checking out a new feature branch β including the branch a fresh clone or worktree starts on β transparently gives it its own per-branch graph; withauto_trackenabled, even a branch created outside a checkout is picked up the first time tokensave opens the project on it.
Seedocs/BRANCHING-USER-GUIDE.mdfor the full guide.
Three MCP tools persist decisions and code-area context across sessions, stored in the per-project.tokensave/tokensave.db.
Use these so the agent doesn't have to re-explain architecture choices session-to-session.
Every MCP call writes an append-only row to~/.tokensave/global.db(savings_ledgertable). Inspect withtokensave gain:
tokensave gain # current project, last 30 days tokensave gain --all # all projects tokensave gain --history --range 7d tokensave gain --json
Dollar estimates use the existing pricing module (Sonnet input pricing, refreshed daily via LiteLLM).
tokensave benchruns a fixed query set throughtokensave_contextand reports retrieval savings vs a full-file baseline (mirrors the CCE methodology):
tokensave bench # ships with 10 default queries tokensave bench --queries my-queries.toml --json tokensave bench --max-nodes 5
Measured against this repo (tokensaveitself) using the shipped generic query set:
Aggregate:88% mean retrieval savings (142.8k β 5.5k tokens across 10 queries).
The default query set targets patterns present in most application codebases (CLIs, daemons, services). Run it on your own project withtokensave benchto see your numbers, or write a tailored query file (--queries my.toml) for tighter recall.
Criterion bench against large real-world repos
Repos and pinned refs(defined inbenches/repos.rs):
Each repo is shallow-cloned (git init+git fetch --progress --depth 1 origin <ref>+checkout FETCH_HEAD) on first use and cached locally; subsequent runs reuse the checkout. Git output is streamed to the terminal so the multi-GB fetch shows real-time progress.
Tools covered (5 queries each).Read tools βsearch,context,callers,callees,node,by_qualified_name,signature,impact,body,files,complexity,doc_coverage,largest,hotspots,god_class,module_api,derives,dead_code,rank,coupling,circular. Write tools βstr_replace,multi_str_replace,insert_at, and (ifast-grepis onPATH)ast_grep_rewrite.
Force-sync on every run.Before any benchmark fires, the harness runs the equivalent oftokensave sync --forceon each repo (index_all()regardless of.tokensave/freshness) so timings always reflect the pinned source.
Write benches and cleanup.Write tools mutate files. To keep the "match must be unique" precondition holding, the harness uses criterion'siter_batchedβ a small scratch file under<repo>/.tokensave-bench-scratch/is rewritten with known contentbefore every timed iteration, then the edit tool runs against it. After all benchmarks finish, the harness runsgit stash --include-untracked && git stash dropinside every prepared repo so the working tree returns to the pinned ref.
Criterion configuration.The bench overrides criterion's defaults tosample_size = 10andmeasurement_time = 30s(vs the stock 100 / 5s), which gives each per-query timing ~30 seconds of measurement β enough that slow tools liketokensave_contexton polkadot-sdk produce stable numbers.
# Required: a writable cache directory for the cloned repos + their indexes. <p align="center"> <a href="https://ai.enzolombardi.net/">
</a> </p> # Expect several GB of disk and a long first run (shallow clone + full index of each repo). export TOKENSAVE_BENCH_REPOS_DIR=~/tokensave-bench-cache cargo bench --bench large_repos
IfTOKENSAVE_BENCH_REPOS_DIRis unset the bench prints a notice and registers zero benchmarks (socargo bench --allstays cheap on contributors' machines).
Configuration (all optional, via environment):
Filtering benchmarksuses the standard criterion CLI β for example, only thesearchtool onscipy:
cargo bench --bench large_repos -- 'scipy/tokensave_search'
Reports (HTML + raw samples) land undertarget/criterion/.
To change the pinned refs (e.g. to a newer release or a specific SHA), editREPOSinbenches/repos.rsand delete the corresponding$TOKENSAVE_BENCH_REPOS_DIR/<repo>/.bench-refmarker so the next run re-fetches. If you skip the post-run cleanup (e.g. youCtrl-Cmid-bench), runninggit stash --include-untracked && git stash dropinside each repo dir restores it manually.
MCP test-matrix probe (scripts/mcp_probe)
scripts/mcp_probe/is a Python harness that drivestokensave serveover stdio against a configurable set of real repos and exercisesevery read-only MCP tool with 5 query variants per language, producing a per-tool / per-repo status table. Same harness serves two purposes:
- Regression sweep.New language support, new tool, or a refactor β re-run the matrix and any cell that newly errors, times out, or returns empty results stands out as a π©.
- Perf probe.Per-call timings are logged in TSV; the same fixed corpus of repos doubles as a coarse cross-version comparison. The currenttokensave_inheritance_depthcycle bug was found by this harness when a single tool on polkadot-sdk timed out at >60 s.
Layout*βprobe.pyis the driver (id-matched JSON-RPC so a slow tool can't poison subsequent calls),isolated.pyre-runs a single tool with a fresh server per call (escapes server queueing),build_matrix.pyreads the TSV and emits markdown,tools/<lang>.pymodules contribute per-language query sets (Rust shipped; add Python/Go/β¦ by dropping a new module),repos.tomllists target repos (override via$TOKENSAVE_PROBE_REPOS).
cargo build --release --bin tokensave python3 scripts/mcp_probe/probe.py python3 scripts/mcp_probe/build_matrix.py > matrix.md
Output cells areβ 5/5(clean),π e/N(errors),β± N/N(timeouts),β
E/N(empty),π’ ok/slow(>10 s calls). Any cell carrying an error or timeout earns a π© in the rightmost column. Per-call detail with the first 100 chars of each error landS in the TSV log for follow-up.
Different from the criterion bench above: criterion measures per-iteration latency for a focused tool set on pinned refs and produces statistical reports undertarget/criterion/;mcp_probeexercises every tool with a broader query set on whatever repos you point it at, optimising for breadth of coverage rather than measurement precision.
The server exposes more than 80 tools (one fewer when the optionalast-grepbinary is not onPATH); the tables below group the most commonly used ones by category. Most are read-only, safe to call in parallel, and annotated withreadOnlyHint. The edit primitives are scoped to single files and re-index in place; session baseline and memory-recording tools also mutate local.tokensavestate and are annotated as non-read-only. The three core tools (tokensave_context,tokensave_search,tokensave_status) are markedanthropic/alwaysLoadso they bypass the client's tool-search round-trip.
Semantic read tools can query an explicitly selected local graph without restarting the MCP server:
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.




