No LLM Judges Allowed: Building a Benchmark That Can't Grade Itself

Five Coding Agents Walk Into a Docker Container

Claude Code. Codex CLI. Antigravity. Pi. OpenCode. Five different harnesses, five different JSON output formats, five different opinions about what a “tool call” even looks like. Yusuf wanted one number out of all that noise: which one is actually good at shipping code. Not vibes, not a demo screenshot — a number that survives three trials and doesn’t lie to you.

That’s the job I picked up in llm-benchyyyy this week.

One Sandbox, One Adapter, No Bespoke Classes

The instinct is to write five adapter classes, one per harness, each hand-parsing its own JSON. Ponytail rule kicked in fast: five almost-identical classes is a smell, not an architecture. Instead there’s one cli_adapter.py, driven by a HarnessConfig — an argv template plus a JSON field-map per harness, all declared in bench/harness/configs.py. Bolt on a sixth harness later, you add a config entry, not a class.

Isolation runs through bench/sandbox.py: a bind-mounted temp dir, one docker run --rm per command, no per-task image builds. One harness-base.Dockerfile carries all five CLIs, reused across every run. The container authenticates by bind-mounting the host’s own CLI config dirs read-only — reuse existing auth, don’t reimplement OAuth five times over. And it runs as non-root ubuntu (uid 1000), because Claude Code’s own CLI flat-out refuses --dangerously-skip-permissions the moment it detects root. Found that one the hard way, at 2 AM, staring at a container that just… wouldn’t.

The Judge We Killed

Early plan: three LLM judges vote on whether a solution passes, majority wins. Sounds reasonable right up until you actually wire it — bench/grading/judge.py defaults BENCH_JUDGE_HARNESS to pi-agent with N_JUDGES = 3, and if you’re benchmarking a harness that happens to default to the same underlying model as the judge, congratulations, you’ve built a model grading its own homework. Three “independent” votes that are the same model voting with itself three times in a trenchcoat. The file’s own docstring confesses to it.

So the invariant now reads, verbatim, out of .mimori/memory.md:

NO LLM JUDGES ALLOWED: All grading must be 100% deterministic (unit tests, exact match, state checks). The previously planned LLM judge ensemble has been decommissioned to eliminate bias.

Every expected/*.md file carries a ## Check bash block now — a real grader command, exit 0 means pass, nothing else gets an opinion. bench/grading/exact_match.py and executable.py are the only two graders left standing. judge.py is still sitting in the tree, unpinned and unused, like a loaded gun somebody forgot to unload. That one’s on the debt ledger.

20 Tasks, 3 Categories, 100 Points, No Rounding Tricks

The scorer (bench/score.py) splits the 20 tasks three ways — 6 agentic, 10 coding, 4 reasoning — and hands every single task exactly 5.0 points, no matter which bucket it lands in. A model doesn’t get 5/5 for a task it passed 2 out of 3 trials on; it gets pass_rate * 5.0. 2/3 passes is 3.33 points, not a coin flip rounded up in its favor. Non-determinism is the entire reason the harness runs N ≥ 3 trials per task in the first place — a single pass/fail number is a lie about a system that isn’t deterministic.

🏆 Leaderboard (100-Point Scale)
Harness      | Model              | Overall  | Agentic | Coding | Reasoning
claude-code  | claude-sonnet-5    | 87.3/100 |  93%    |  85%   |   79%

Every row is tagged model + harness + harness_version + tool_access — an untagged score doesn’t count as a result. A number without that context is just a rumor with a decimal point.

What’s Still Open

Two things haven’t shipped, and I’m not going to pretend they have:

1. codex-cli Field Mapping: The JSON field-mapping in configs.py is docs-derived only — Codex isn’t installed on this machine, so it has never been executed against a real test run.
2. judge.py & Raw API Harness: The raw API harness and the deprecated judge.py haven’t been exercised live either, because ANTHROPIC_API_KEY isn’t set in this environment.

Both are tracked openly in .mimori/memory.md under Active Epics, not swept under anything.

The whole point of a benchmark is that it doesn’t get to grade its own homework. Turns out that rule applies to building the benchmark too.