Does your CLAUDE.md actually do anything?
You write a file of instructions that gets prepended to every request. Be concise. Don't hedge. Answer first. cc-bench swaps the config, runs the same prompts, and counts what changed in the output.
An impression is not a measurement
You can read a model's replies and form an impression. An impression can't tell you whether the config changed anything, or whether you just got used to it. So here is the counter layer itself, working on text you can see and edit.
Computed in your browser by metrics.browser.js, generated from scripts/metrics.mjs, so these are the same counters the benchmark runs. The loaded samples are hand-written to demonstrate the counters. They are not model output and they are not a benchmark result.
Why counters instead of a quality score
The first version of this project scored configs on a judged rubric. It was junk, for a reason worth stating publicly: the rubric was written in the vocabulary of the config it was testing. The axes were named after the things the config asked for, so the config won by construction, and there was no axis it could lose.
A benchmark that cannot produce a negative result is not measuring anything.
Counting sidesteps that. A counter doesn't know what you were hoping for. The trade is that counters are blind to correctness. A config that makes a model terse and wrong fingerprints beautifully. So exactly one judged axis survives, task success, and it sits beside the fingerprint rather than on top of it.
What you'd use it for
Does my config do anything? Run it against an empty config. If nothing moves, your file is decoration.
Which of my two drafts is stronger? Run them head to head on the same prompts.
Can a cheaper model do what my expensive one does? Measure the behavior you like, save it as a target profile, and find the model-plus-config that lands closest. Existing work measures how far a model drifts from its own trained register; landing on a target you wrote is the part nobody occupies.
Four things that keep it honest
Each of these took a wrong version first.
Paired on prompts
Prompts differ in verbosity far more than configs do. Comparing raw means throws that variance into the error term and drowns the effect you are looking for. The unit of analysis is the per-prompt difference.
The public set is 48 single-turn prompts, six each across eight situation strata (bug report, feature request, code review, design discussion, scoping, proposal check, tooling question, open-ended chat), plus 6 multi-turn sequences for the drift arm. Prompts are selected on the situation, meaning what the user is doing before any response exists, never on expected outcome or on which config is expected to win. 48 is what the sizing asks for: mean sentence length needs 44 prompts to detect a 20% shift at 80% power, and the formatting counters need several hundred, which is why they stay flagged low-power rather than being chased with a bigger set.
A pre-registered primary set
A wide counter set is a multiple-comparison tax: every counter you add inflates the corrected p-value of every other one, so improving the instrument destroys the evidence. Four counters are registered in advance and carry claims. The rest are labelled exploratory, which is exactly the split the readout above draws.
A 20% relative floor
Smaller effects are not detectable at any run size worth paying for. Design configs for large behavioral changes, or don't report the small ones.
The same bound also runs the other way. A difference test can only ever fail to reject, and "p was not small" is equally compatible with the two configs behaving the same and with the run being too small to see anything. Those are opposite answers to the question most config edits ask. So every paired counter also carries a two-one-sided-tests reading against ±20%. An interval that fits inside the bound comes back equivalent, which is a finding rather than a shrug; an interval too wide to place comes back inconclusive. No verdict is ever read off a large p-value. Equivalence is the cheaper claim, too: the verbosity primaries reach it at roughly a third of the prompts a difference needs.
A Goodhart guard
Counters are trivially gameable. Telling a model never to write the word "caveat" zeroes a counter with no behavioral change at all. That is arguably fine, since gaming the counter is obedience, which is what's measured. It is only a problem when the named counters are the only ones that move. So the split between counters a config could target and counters it could only move indirectly is derived from the config text on every run. Name a counter and it reclassifies as visible in that same run.
Zero dependencies, no install step
git clone https://github.com/thatmike1/cc-bench && cd cc-bench
cp variants.example.json variants.json # declare which configs to compare
node scripts/build-configs.mjs # materialize them
node scripts/run-bench.mjs --engine claude --variants ungoverned,terse
node scripts/report.mjs results/<runId>The example manifest is runnable as shipped. To test your own file, point source at it and declare which sections to ablate:
{
"source": "~/.claude/CLAUDE.md",
"variants": {
"current": { "drop": [] },
"no-style": { "drop": ["## Response Style"] }
}
}That is the ablation the tool is built for: your real config against your real config minus one section, so the delta is attributable to that section and nothing else. variants.json is gitignored, so your personal config never enters the repo.
Your live ~/.claude/CLAUDE.md is never written. CLAUDE_CONFIG_DIR and a HOME override are both dead ends, since the real file loads regardless and they redirect your credentials on top of it. So each claude spawn runs inside a bwrap mount namespace with the variant bound over that one path. The mount dies with the process, interactive Claude Code keeps seeing your own config, and two batches can run different variants at once.
bwrap is Linux-only. Where it is missing, the runner falls back to --isolation swap, which does overwrite the live file for the length of the batch, with a timestamped backup and a restore on exit. Under that fallback only, don't use Claude Code interactively while a batch is running.
What is already occupied
"Nobody benchmarks persistent-config obedience" was this project's original pitch. It is false, and the honest version of the claim is narrower.
| Work | What it establishes |
|---|---|
| McMillan 2605.10039 | 1,650 Claude Code sessions varying config size, position, architecture and contradictions. Null on all four. The real effect is within-session decay. |
| ContextEcho 2605.24279 | Persona drift via a judge-free fingerprint. Compaction does not reset drift; a single-shot anchor can restore it. |
| OctoBench 2601.10343 | Scaffold-aware instruction following over trajectories, 7,098 objective checklist items. |
| Gloaguen et al. 2602.11988 | Context files do not generally improve task success, and add over 20% inference cost. |
Counters are borrowed rather than invented wherever a published operationalization exists. Verbosity and formatting follow LMArena's style control, so the numbers sit on the same scale as a public leaderboard.
Where this actually stands
Built
- The matrix runner, three engines
- Text and agentic counters
- The fixture repo
- The public prompt set, 48 prompts
- Paired statistics
- Multi-turn drift arm
- The Goodhart guard
- Console, HTML and web reports
Not built
- The target-profile mechanism. It is what makes this a config-authoring tool rather than a config-comparing one.
Not done
- A single real run under the current design. The one full run that exists used the retired prompt set and the old counters, and it was inconclusive. It is kept as a sizing guide and nothing more.
What a fingerprint isn't
Moving a counter toward a target has not been shown to make a model better. Stylometric features separate human from machine text at near-perfect accuracy while saying nothing whatsoever about whether the text is correct or useful.
This measures whether your config did what it said. Whether what it said was a good idea is still your problem.