ken
About
Fast hybrid code search for agents - it's pure Go, single static binary, 5 lexical + Model2Vec semantic embeddings + RRF fusion + a code-aware reranker, with the retrieval algorithm ported verbatim from semble
Details
- Author
- townsendmerino
- Categories
- Developer Tools, Search, Knowledge Base
Jump to
Setup
Install ken in your MCP client (Claude Desktop, Cursor, Windsurf, and others).
Repository: https://github.com/townsendmerino/ken
Follow the installation instructions in the repository README, then restart your MCP client.
Fast hybrid code search for agents.Pure Go, single static binary, drop-in MCP-compatible withMinishLab/semble— same tool schemas, same output format, install steps swapped to a Go binary.
ken is a Go port ofsemble: BM25 lexical + Model2Vec semantic embeddings + RRF fusion + a code-aware reranker, with the retrieval algorithm portedverbatimfrom semble'ssearch.py+ranking/.py.
- ~97% recall@10 in the default (hybrid) mode—0.967 NL / 0.995 symbolon semble's 1,251-query benchmark, vs grep's ~99.9% — while costing an agent~46× fewer tokensthangrep + Read(4,120 vs 189,773 median tokens on NL queries — measured in the same default hybrid mode). For "find the chunk that answers this," that's a 1–2 order-of-magnitude token win at near-parity recall. (Reproduce:docs/BENCH.md.)
- Single static binary.Pure Go, no cgo, no Python interpreter on cold start, no GIL on indexing. Cross-compiles to Linux / macOS / Windows (amd64/arm64) for free.
- Drop-in for semble.Samesearch/find_relatedMCP tool schemas and the same markdown-string wire format — swap thecommand:path and existing agents work unchanged.
- Local, CPU-only.Embedding inference, BM25, and fusion all run on the CPU. No API keys, no GPU, no vector DB, air-gapped friendly.
One knob controls recall.The82–91%figure in the token-budget tables is theBM25-only fallbackken runs in when no embedding model is installed.ken-mcp fetches the model automatically on first run(~60 MB, pure-Go, no Python — serves bm25 until it lands, then upgrades to the~97%hybrid path;KEN_MCP_AUTO_FETCH=0to disable). For the CLI, runken download-modelonce. Exhaustive enumeration (refactors, pre-rename audits) still belongs to grep; ken is for "find the chunk that answers this."
- ARCHITECTURE.md— current-state map: module layout, runtime/concurrency model, data flow, invariants. Start here for the code.
- docs/USERS.md— agent users. Install ken-mcp, point your agent at it, use the nine tools. 5-minute on-ramp.
- docs/DEVELOPERS.md— SDK authors and tuners. Themcp.Runembedded-corpus library, prebuilt indices,fs.FSindexing, custom chunkers, tuning rerank, performance expectations.
- docs/DESIGN.md+docs/internal/DECISIONS.md— algorithm spec + every architectural decision (ADRs).
- docs/BENCH.md— benchmark reproduction (NDCG, token-budget recall, the hybrid-vs-BM25 decomposition).
# macOS / Linux (Homebrew) — installs both ken and ken-mcp: brew install --cask townsendmerino/tap/ken
# Windows (Scoop): scoop bucket add townsendmerino https://github.com/townsendmerino/scoop-bucket scoop install ken
# Install both binaries (Go 1.26+). go install github.com/townsendmerino/ken/cmd/ken@latest go install github.com/townsendmerino/ken/cmd/ken-mcp@latest # Download the default Model2Vec model (~60 MB, one-time). Pure Go, no Python. # (ken-mcp auto-fetches this on first run; the CLI needs it explicitly.) # This is the single biggest retrieval-quality lever — it puts you on the ~97% path. ken download-model # Search any local repo from the CLI. ken search /path/to/myrepo "save model to disk" --model ~/.ken/model
Or skip the model and use lexical-only mode (BM25-only costs ~14 pp recall@10 vs the hybrid default — seedocs/BENCH.md):
ken search /path/to/myrepo "validateToken" --mode bm25
Not sure your setup is right?ken doctorchecks model availability, rerank-cache warmth, enrichment, token-savings tracking, and ken-mcp config, and prints prioritized recommendations (e.g. "no model — runken download-model").
Pre-built binaries formacOS, Linux, and Windows(amd64/arm64) are attached to eachrelease—.tar.gzfor macOS/Linux,.zipfor Windows.
As of v0.3,ken index <path>defaults towatch mode— it stays alive and re-indexes on change (2 s debounce);--no-watchrestores build-once-and-exit.ken-mcpalways watches, so an agent editing the repo mid-session sees its own changes without a restart. ken also respectsnested.gitignorefiles (per-directory, matching git).
ken-mcpspeaks JSON-RPC over stdio and serves the same two core tools (search,find_related) semble does, with the same arg shapes and markdown output.
# Claude Code claude mcp add ken -s user -- /absolute/path/to/ken-mcp
// ~/.cursor/mcp.json (or .cursor/mcp.json) — also .vscode/mcp.json with "servers" { "mcpServers": { "ken": { "command": "/absolute/path/to/ken-mcp" } } }
# ~/.codex/config.toml [mcp_servers.ken] command = "/absolute/path/to/ken-mcp"
// ~/.opencode/config.json { "mcp": { "ken": { "type": "local", "command": ["/absolute/path/to/ken-mcp"] } } }
The default above runs ken-mcp as alocal subprocess— the right choice for most setups (the OS-user boundary is the auth boundary; nothing is network-exposed). For acentralized dev box, staging server, or team-shared instance— one ken-mcp feeding many agents, or a remote IDE not co-resident with the code — ken-mcp also speaks MCP overStreamable HTTP(KEN_MCP_TRANSPORT=http, ADR-041). Same tools, same wire format; agents already trained on the stdio server work unchanged.
This exposes ken-mcp to the network, so auth is mandatory and several guards fail loud at startup:
# On the server (front with a TLS-terminating reverse proxy — ken-mcp does NOT do TLS): export KEN_MCP_TRANSPORT=http export KEN_MCP_ADDR=:8080 # default export KEN_MCP_AUTH_TOKEN_FILE=/etc/ken/token # preferred (keeps the secret out of the environment) export KEN_MCP_DEFAULT_REPO=/srv/code export KEN_MCP_RATE_LIMIT=100 # req/min per client IP (0 disables); default 100 ken-mcp # exits non-zero if no token is set
Point an agent at it with a bearer token (Claude Code shown; any Streamable-HTTP MCP client works):
claude mcp add --transport http ken https://ken.example.com \ --header "Authorization: Bearer $TOKEN"
// Cursor / VS Code mcp.json — remote form (check your editor's remote-MCP docs) { "mcpServers": { "ken": { "url": "https://ken.example.com", "headers": { "Authorization": "Bearer <token>" } } } }
Guards: HTTP moderefuses to start without a token(no insecure default, no localhost exception);KEN_DB_SAMPLE_ROWS>0ishard-rejectedin HTTP mode (sampled DB values would be network-searchable); requests are rate-limited per client IP. TLS is out of scope by design — terminate it at a reverse proxy in front of ken-mcp. stdio stays the default and is completely unaffected.
The full env reference — including theKEN_DB_database variables — is indocs/USERS.mdanddocs/db-indexing.md. For agents that should route between ken and grep deliberately (rather than ken's default "prefer ken" instruction), see the routing snippet in[docs/USERS.md.
…
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
