iwe memory bench

Measures whether adding the IWE MCP server makes claude -p (Claude Code headless) better at agent-memory tasks, on the LOCOMO dataset, judged Mem0-style.

Every model interaction — answering and judging — runs through claude -p. There are no direct API calls; runs bill the claude.ai subscription of the bench profile. The judge prompt and grading criteria are ported verbatim from the legacy Mem0 LOCOMO evaluation (binary CORRECT/WRONG "J" metric, categories 1–4, adversarial category 5 excluded).

Arms

All arms answer the same questions over identical markdown session transcripts; only the tool surface differs.

Arm	Tools	Measures
fs	Grep, Glob, Read	filesystem baseline (Letta-style)
iwe	iwe_find, iwe_retrieve, iwe_tree, iwe_squash, iwe_stats via MCP	IWE retrieval in isolation
fs-iwe	both	is IWE additive? (headline)
full-context	none; transcript inline	reference ceiling, judge calibration

Dataset

The LOCOMO dataset is not distributed with this repo; cargo xtask download fetches locomo10.json from snap-research/locomo (CC BY-NC 4.0; Maharana et al., Evaluating Very Long-Term Conversational Memory of LLM Agents, ACL 2024) into the gitignored data/ directory.

Setup

One-time: create the isolated bench profile with a claude.ai subscription login.

CLAUDE_CONFIG_DIR=$PWD/.claude-profile claude /login

The iwe and iwec binaries must be on PATH (or set IWE_BIN/IWEC_BIN).

Usage

cargo xtask download
cargo xtask prepare
cargo xtask doctor

cargo xtask answer --run results/fs-pilot --arm fs --conversations conv-26 --limit 5
cargo xtask judge  --run results/fs-pilot
cargo xtask report --run results/fs-pilot

Runs are resumable: rerunning answer or judge with the same --run skips completed items. answer aborts after 5 consecutive failures (usage limits); rerun to resume.

Useful flags: --model / --judge-model (default sonnet), --categories 1,2,3,4, --conversations <id,...>, --limit N, --workers N, --max-budget-usd, --timeout-secs.

Isolation

Verified against claude 2.1.201; cargo xtask doctor re-verifies before every sweep:

• runs use a dedicated CLAUDE_CONFIG_DIR (.claude-profile/, gitignored) with no CLAUDE.md, settings, hooks, skills, or plugins;
• ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN are stripped from the subprocess env (a set key silently outranks the claude.ai login);
• --setting-sources "" blocks project and ancestor CLAUDE.md discovery; --strict-mcp-config admits only the workspace .mcp.json;
• doctor plants sentinel CLAUDE.md files and asserts the probe answers NONE, and confirms subscription auth.

Metrics

summary.json per run, overall and per category: j (judge accuracy — headline), f1, exact_match, bleu1 (clipped unigram precision × brevity penalty; BLEU-2..4 from the legacy harness are dropped), cost, turns, duration percentiles, token totals.

Since subscription runs are quota-billed, cost_usd is informational; tokens and turns are the primary efficiency metrics.

Deviations from the legacy Mem0 harness

• The answering "system" is Claude Code with tools, not a search+prompt pipeline; the judge runs as a one-shot claude -p call with tools disabled.
• BLEU reduced to BLEU-1 with a simple tokenizer.
• No temperature control exists for claude -p; headline runs use 3 repetitions, reported mean ± std.