memtrust

Agent memory backends each publish their own benchmark numbers, on different tests, measured different ways. memtrust runs the same evals against all four and publishes the raw logs. Run against the vendors, not by them.

pip install memtrust-cli memtrust run --backends mempalace,mem0,zep,openviking --eval all

(For contributing to this repo instead of just running it, seeDevelopment--pip install -e ".[dev]"from a clone.)

Contents:Why this exists·What it does·Commands·How this differs·Contradiction detection·Compression fidelity·Temporal-KG boundary·The landscape·Benchmarks·GitHub Actions usage·Self-host·Install·Hosted layer·Backend coverage·Development·FAQ·License·Success stories

If you've compared agent-memory backends recently, you've probably noticed each one leads with a different accuracy number, on a different benchmark, measured a different way. MemPalace's own community already flagged the problem in public. Issue#27on the MemPalace repository, opened April 7, 2026 and still open, documents that a headline 100% LongMemEval figure, measured with Haiku reranking, wasn't reproducible from the repository's own benchmark scripts and was pulled from the README as unverifiable. A separate 96.6% figure people cite everywhere turns out to be mostly ChromaDB's default embeddings doing the work in raw mode, not MemPalace's own architecture. A "lossless" compression claim (the "AAAK" mode) drops the same LongMemEval score from 96.6% to 84.2% in practice, a 12.4 percentage point gap. Two internal pull requests attempting to fix the reporting problem, #433 and #729, were both closed without merging on April 12, 2026 -- #729 within seven minutes of being opened. As of this writing, the issue has 232 thumbs-up reactions and 39 comments.

Where this stands right now, in one place:live benchmark resultsfor one backend (mem0_direct, self-hostedmem0ai), including a real bug this project's own attempt to get those numbers surfaced in mem0's default configuration; and197 real GitHub issues and PRsfiled against MemPalace, Mem0, Zep/Graphiti, and OpenViking independently root-caused against this codebase -- 55 (28%) PASS, 16 (8%) PARTIAL, 42 (21%) a genuine capability gap, 84 (43%) not applicable, every verdict re-verified by a reviewer independent of whoever built the fix.

$ memtrust run --backends mempalace,mem0,zep,openviking --eval all memtrust 0.3.4 -- run_id=mt_2026-08-04T061759Z Backends: mempalace, mem0, zep, openviking Evals: longmemeval, locomo, contradiction, resource_sync_safety, compression, ranking_quality, scale_stress, embedding_drift, crash_recovery, extraction_quality, migration_rollback, filter_injection, lock_contention, stats_accuracy, orphan_cleanup, result_consistency, temporal_kg_boundary mempalace: SKIPPED (not configured) -- mempalace is not configured: environment variable MEMPALACE_STORAGE_PATH is not set. Skipping this backend. See docs/methodology.md for setup instructions. mem0: SKIPPED (not configured) -- mem0 is not configured: environment variable MEM0_API_KEY is not set. Skipping this backend. See docs/methodology.md for setup instructions. zep: SKIPPED (not configured) -- zep is not configured: environment variable ZEP_API_KEY is not set. Skipping this backend. See docs/methodology.md for setup instructions. openviking: SKIPPED (not configured) -- openviking is not configured: environment variable OPENVIKING_API_KEY is not set. Skipping this backend. See docs/methodology.md for setup instructions. Cost: $0.00 (no LLM-judged evals ran -- structural evals only, or judge not configured) Full report: memtrust-report-2026-08-03.json

$ pytest --cov=memtrust --cov-report=term-missing ... (33 module rows total; the 11 most relevant to this README are shown below) Name Stmts Miss Cover ------------------------------------------------------------------------------------- src/memtrust/adapters/base.py 290 1 99% src/memtrust/adapters/mempalace_adapter.py 265 15 94% src/memtrust/adapters/mem0_adapter.py 140 12 91% src/memtrust/adapters/mem0_direct_adapter.py 281 34 88% src/memtrust/adapters/openviking_adapter.py 178 18 90% src/memtrust/adapters/zep_graphiti_adapter.py 63 3 95% src/memtrust/adapters/zep_graphiti_selfhosted_adapter.py 165 24 85% src/memtrust/evals/contradiction.py 127 2 98% src/memtrust/evals/compression.py 86 1 99% src/memtrust/evals/temporal_kg_boundary.py 90 3 97% src/memtrust/receipt.py 118 10 92% ------------------------------------------------------------------------------------- TOTAL 4167 265 94% 590 passed, 8 skipped in 5.92s

590 passing tests across 33 source modules, 94% overall statement coverage, 98% on the contradiction-detection eval, 99% on compression/round-trip fidelity, 97% on the temporal-KG boundary eval, 85-99% across the adapter layer. The 8 skips are live-mempalace-package tests that only run with the optionalmempalace-directextra installed (pip install -e '.[dev,mempalace-direct]'). Every test mocks its HTTP or wire layer, or uses an in-memory fake backend -- none of them touch a real network, though a meaningful share of the adapter tests now import and exercisereal installed vendor classesdirectly (mem0ai==2.0.12's embedder and vector-store modules, and -- gated behind the optionalmempalace-directextra -- the realmempalace.mcp_serverfunctions), mocking only the outermost network or wire-client boundary rather than the whole library.graphiti-coreis not installed in this environment, so its self-hosted adapter's tests still run against a hand-written Protocol double built to match the real package's confirmed method signatures, not the real classes -- seedocs/methodology.md's adapter confidence table for exactly which claim rests on which kind of verification.

$ memtrust --help Usage: memtrust [OPTIONS] COMMAND [ARGS]... memtrust: an independent, reproducible benchmark harness for agent-memory backends. Options: --version Show the version and exit. --help Show this message and exit. Commands: keygen Generate a new Ed25519 keypair for signing memtrust run... report Read a prior memtrust run JSON report and print a formatted... run Run the eval suite against the requested backends. verify Verify a signed receipt produced by memtrust run --sign.

Every line above came straight from runningmemtrust --help,memtrust run --help,memtrust report --help,memtrust keygen --help, andmemtrust verify --helpagainst this repo. Nothing here is invented.

memtrust ships aModel Context Protocolserver so an AI agent (Claude, Cursor, or any MCP-compatible client) can run a memory-backend benchmark directly, without a human invoking the CLI by hand.

pip install "memtrust-cli[mcp]"

Add it to your MCP client's config (for Claude Desktop,claude_desktop_config.json):

{ "mcpServers": { "memtrust": { "command": "uvx", "args": ["--from", "memtrust-cli", "memtrust-mcp"] } } }

The server exposes one tool,run, that shells out tomemtrust runwith the given arguments (memtrust has no--jsonflag, so the wrapper writes to a private temp--outputfile and reads it back) and returns the parsed JSON report:

run(["--backends", "mempalace", "--eval", "stats_accuracy"])

Transport is stdio, so there is nothing to host: the MCP client spawns the server as a local subprocess. Source:src/memtrust/mcp_server.py.

Honest scope, stated the same way this project states it for every other eval: the realmempalacePyPI package is not installed in this build environment, and PR#1914's fix had not shipped in a releasedmempalaceversion as of this adapter's live-verified 3.5.0 build -- it lands under the package's[Unreleased]changelog section.tests/test_temporal_kg_boundary.pyproves theclassification logicis correct against two hand-written fake implementations that reproduce the confirmed pre-#1914 (closed-interval) and post-#1914 (half-open-interval) SQL comparison exactly.This has not been run against a live MemPalace instance.It is wired intomemtrust run --eval temporal_kg_boundary(see "Commands" above); against any backend other thanmempalace, it reportsnot_applicablerather than an error.

Live results: mem0_direct (self-hosted), July 2026.MemPalace, Zep, and OpenViking are still not yet measured against a live backend -- see "Backend coverage" below for the confidence level on each adapter. Mem0 has one real result, produced against the actualmem0aiOSS library running self-hosted -- in-process, viaMem0DirectAdapter, backed by a local Qdrant instance and the OpenAI API for embeddings and extraction.

[!NOTE] This result is from the self-hostedmem0aiOSS library, not Mem0's hosted Platform API. Don't read it as a claim about the hosted product.

$ export MEM0_DIRECT_EMBEDDER_PROVIDER=openai $ export MEM0_DIRECT_VECTOR_STORE_PROVIDER=qdrant $ export MEM0_DIRECT_VECTOR_STORE_URL=http://localhost:6333 $ memtrust run --backends mem0_direct --eval contradiction,compression,extraction_quality memtrust 0.3.2 -- run_id=mt_2026-07-20T210918Z Backends: mem0_direct Evals: contradiction, compression, extraction_quality mem0_direct: configured, running evals... Running Contradiction-Detection against mem0_direct... flagged: 0.0% silent-overwrite: 100.0% served-stale: 0.0% empty-or-lost: 0.0% Running Compression/Round-Trip-Fidelity against mem0_direct... fidelity by mode -- default: 31.6% Running Extraction-Quality against mem0_direct... junk-retained: 0.0% valid-lost: 100.0% feedback-loop-duplicate: 0.0% Cost: $0.00 (no LLM-judged evals ran -- structural evals only, or judge not configured)

The full raw report is committed atleaderboard/mem0_direct-2026-07-20.json, andleaderboard/data.jsoncarries the contradiction numbers into the static leaderboard site (mempalace/mem0/zep/openvikingstill shownot_measuredthere;mem0_directis the one real row).

What that means case by case, not just the percentage:

- Contradiction detection, 7/7 cases: every contradicting fact silently overwrote the old one.0% were flagged as a conflict, 0% served stale, 0% empty-or-lost. Tell it your meeting moved from 2pm to 3pm and it stores the new fact with no signal that anything changed -- this is exactly the question LongMemEval and LoCoMo don't test, and exactly what "Why this exists" above is about.
- Compression/round-trip fidelity, 5 cases: 31.6% mean literal character-level reconstruction.This is expected, not a defect -- mem0's design goal is semantic fact extraction, not verbatim storage, so a literal-reconstruction score was never going to be high. It quantifies what "not built for lossless storage" concretely means for this backend: ask it what you said and you get the gist back, not your words.
- Extraction quality, 15 cases (12 deliberately junk, 3 deliberately valid): 0% junk retained, 100% of the valid cases lost.All 12 junk inputs (boot-file restating, cron heartbeat noise, system dumps, hallucinated-profile bait) were correctly rejected. All 3 valid-content cases were also dropped -- stored but never came back on retrieval. The valid-side sample is small (n=3); treat this as a signal worth digging into further, not a settled number.

[!WARNING] A real bug this run surfaced in mem0ai itself, not in memtrust: a freshmem0ai==2.0.12install with nothing butOPENAI_API_KEYset fails every single LLM-based extraction call, out of the box, for anyone.

Getting any of the numbers above required a fix first. mem0's own default model (mem0/llms/openai.py:self.config.model = "gpt-5-mini") is a reasoning-tier model that only accepts the API's default temperature, but mem0's own reasoning-model detection (mem0/llms/base.py'sreasoning_modelsset) checks for the string"gpt-5o-mini", not"gpt-5-mini"-- two different strings, so the check never fires, and mem0 sendstemperature=0.1on every call regardless. The result is a400 Unsupported value: 'temperature' does not support 0.1 with this modelerror on every extraction call, silently caught by mem0 and reported by memtrust asN/A (no scoreable cases)rather than a real result.Mem0DirectAdapternow works around it by passingis_reasoning_model=Trueexplicitly -- mem0's own documented override for exactly this situation -- seesrc/memtrust/adapters/mem0_direct_adapter.py's "Default LLM extraction is broken out of the box" section for the full citation. No upstream mem0ai issue filed for this as of this writing.

To reproduce this or measure the remaining three backends:

export MEM0_API_KEY=... # and/or export ZEP_API_KEY=... export OPENVIKING_API_KEY=... export MEMPALACE_STORAGE_PATH=... export MEMTRUST_JUDGE_API_KEY=... # needed for LongMemEval/LoCoMo grading; contradiction-detection doesn't need it memtrust run --backends mempalace,mem0,zep,openviking --eval all memtrust report memtrust-report-<date>.json

The command prints per-backend accuracy and contradiction-handling rates, writes a full JSON report, and prints an estimated cost for any LLM-judged evals that ran.MemPalaceAdapter's drawer and knowledge-graph calls are now live-verified against a real installed instance (see "Backend coverage" below), but OpenViking's memory-write/query paths, and parts of the self-hosted Mem0 and Zep/Graphiti adapters, are still built against best-effort interpretations of documented or source-read product concepts rather than a live-confirmed API -- see the confidence table indocs/methodology.mdbefore treating any adapter's output as authoritative, and consider that table's gaps a standing invitation to contribute a fix.

Labeling requirement for any futureaccuracyfigure published here.LongMemEval and LoCoMoaccuracygrades the LLM judge's verdict on raw retrieved-record content directly -- there is no answer-generation step in either eval runner. This is not the same measurement as the official LongMemEval/LoCoMo leaderboards' generate-then-judge QA-accuracy scores. Anyaccuracynumber this project publishes for those two evals must be labeled "retrieval-graded accuracy," not bare "accuracy," and must not be directly compared to leaderboard figures without that caveat. Seedocs/methodology.md's "Retrieval-graded accuracy vs. generated-answer accuracy" section.

Run the suite on a schedule and publish results to the leaderboard:

name: memtrust-leaderboard on: schedule: - cron: "0 9   1" # weekly workflow_dispatch: {} jobs: benchmark: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" - run: pip install memtrust-cli - run: memtrust run --backends mempalace,mem0,zep,openviking --eval all --output leaderboard/data.json env: MEM0_API_KEY: ${{ secrets.MEM0_API_KEY }} ZEP_API_KEY: ${{ secrets.ZEP_API_KEY }} OPENVIKING_API_KEY: ${{ secrets.OPENVIKING_API_KEY }} MEMTRUST_JUDGE_API_KEY: ${{ secrets.MEMTRUST_JUDGE_API_KEY }} - run: git add leaderboard/data.json && git commit -m "Update leaderboard" && git push

This repo's own CI (.github/workflows/ci.yml) runs lint, type-check, test, and a dependency security audit on every push and pull request -- no vendor credentials required, since every test runs fully offline.

git clone https://github.com/RudrenduPaul/memtrust cd memtrust pip install -e ".[dev]" export MEM0_API_KEY=... memtrust run --backends mem0 --eval all

Point an adapter at your own backend, or run the suite against your own conversation data instead of the bundled synthetic fixtures (seedocs/methodology.md's note on swapping in the real LongMemEval/LoCoMo datasets). Nothing leaves your machine unless you choose to publish it.

pip install memtrust-cliis the verified, working install path -- confirmed against a clean virtualenv as of this writing.pip show memtrust-cliandmemtrust --versionboth report0.3.4.

npx (currently broken -- tracked, not hidden)

The npm package (memtrust-cli) is live and no longer 404s, and its source onmain(npm/memtrust-cli/bin/memtrust.js) correctly runsuv tool run --from memtrust-cli==<version> memtrust <args>. The published0.3.4npm tarball, however, still ships the earlier, broken build of that same file, which runsuv tool run --from memtrust==<version> memtrust <args>instead -- pointed at a PyPI project namedmemtrustthat has never existed (pypi.org/pypi/memtrust/jsonreturns 404, same as the FAQ below already documents). The source fix landed onmain; the npm publish that would ship it has not gone out yet. Confirmed live, today, by downloading the actual published tarball (npm pack memtrust-cli@0.3.4) and inspectingbin/memtrust.jsdirectly, not by reading source and assuming it matches what's published:

$ npx -y memtrust-cli --version npm error could not determine executable to run × No solution found when resolving tool dependencies: ╰─▶ Because memtrust was not found in the package registry and you require memtrust==0.3.4, we can conclude that your requirements are unsatisfiable.

Until a new npm version ships with the fixed wrapper, usepip install memtrust-cli(above) -- it is unaffected, since the bug is only in the npm wrapper script, not the PyPI package it bootstraps. For CI and agent runners that have Node.js but not Python: hold off onnpx memtrust-cliuntil this section no longer carries this notice, or provision Python and usepip install memtrust-clidirectly.

The npm package is namedmemtrust-cliso it is unambiguous as a CLI tool at a glance (and so it doesn't collide with any futurememtrustJS library package).npxalways resolves the package name to its matchingbinentry automatically, sonpx memtrust-cli ...is the intended zero-install path once the fixed build ships. Once installed, the package also exposes the shortermemtrustcommand as a secondbinalias -- matching the underlying Python CLI's own command name -- so you are not stuck typingmemtrust-clifor every subsequent invocation.

This was never meant to be a zero-dependency install:npx memtrust-clistill fetchesmemtrust-clifrom PyPI on first use. What it removes is a Python toolchain to provision by hand -- it bootstraps the interpreter and package fetch for you via a bundled, verified copy of Astral'suv. Each platform package bundles a genuine, SHA-256-verified copy ofuv's own GitHub release binary (fetched at npm package-publish time, never at end-user install time). The npm package is pinned to its own version -- bumpnpm/memtrust-cli/package.json's version and republish when a new PyPI release ships, and every subsequent install resolves to that exact release, not whatever happens to be newest at run time. That republish is exactly the step still outstanding here.

pip install -e ".[dev]" ruff check . && ruff format --check . mypy --strict src/memtrust pytest --cov=memtrust --cov-report=term-missing --cov-fail-under=80 pip-audit

.pre-commit-config.yamlwires ruff and mypy intopre-commitif you'd rather run these on every commit than remember to run them by hand.

What is memtrust, and what actually makes it different from reading a vendor's own benchmark page?It's a CLI harness that runs the same evals (LongMemEval, LoCoMo, and 15 others registered in--eval, including a contradiction-detection eval none of the four tracked backends publish a number for) against MemPalace, Mem0, Zep/Graphiti, and OpenViking, and prints the raw output rather than a curated summary. The differentiator isn't a proprietary scoring model; it's that nobody outside the vendor had previously run the same test, the same way, against every option, with the full methodology published alongside the code that produced it (docs/methodology.md). See "Why this exists" above for the MemPalace LongMemEval overclaim (mempalace/mempalace#27) that motivated the project.

…