Report date: September 21, 2026

A benchmark score tells you how often an agent finished the job. A record of a failed run can show you where the job came apart.

We need both. In Dharma's controlled, selected-subset study of Terminal-Bench 2.1, the Atlas-plus-review workflow passed 34 of 42 executions. Unassisted Codex passed 25. Ordinary memory passed 32. Cumulative Atlas without the additional review passed 28.

Those results are encouraging. The more useful engineering question is what happened inside them: which failures were corrected, what guidance reached the actor, and whether the actor's work actually changed.

What we compared

The frozen evaluation covered 21 selected tasks, with two trials per task in each of four approaches: 168 executions in total. It used Terminal-Bench 2.1 revision 7131e4375048a0e408a8fb404b5f499d726b695b. Codex CLI 0.144.1 ran gpt-5.6-sol with LOW reasoning. The original Harbor grading checks evaluated the resulting environment after the agent stopped.

The unassisted arm began without retained experience. Ordinary memory supplied selected plain-language lessons. Cumulative Atlas supplied selected corrective skills. Atlas plus review used the same Atlas snapshot, additional Gemini 3.8 Flash HIGH review, and at most one repair.

That last arm is a larger workflow. Its extra computation matters when interpreting both success and cost. The comparison does not isolate skill text from the other work performed.

A recovery you can inspect

A separate development case, kv-store-grpc, makes the product idea tangible. Both controls used a temporary interactive context to keep a server running. At post-actor grading, the service was unavailable. Each control passed five of seven checks and failed the task.

The retained Dharma guidance addressed process lifetime: detach the service, redirect its streams, and verify it from another command context. Both Dharma trajectories show a detached launch using setsid. Both passed all seven original checks.

This is the useful chain: retained failure, applicable procedure, delivered guidance, observed action, independently graded work product. It changes the next agent's instructions and available checks without changing the model's weights. This development case was previously exposed, so it is not evidence of unseen-task transfer.

A repair is different from beating a control

Across the frozen evaluation, five failed first passes became passes inside the review arm. They came from four distinct tasks, including two trials of hf-model-inference. Twelve repairs were executed in total.

Those recoveries are different from the nine additional passing executions relative to unassisted Codex. One compares first and final work products within a run. The other compares separately executed workflows. Both matter; they answer different questions.

What changed in the later iteration

The next development iteration made skill use explicit. In the frozen evaluation, a retained startup skill had been offered in only eight of 42 executions in each Dharma treatment. The later workflow required a core coding procedure in every Dharma run, added relevant specialist corrections, and recorded installation, native actions and rollback.

Two cases illustrate the progression. The historical unassisted controls for db-wal-recovery and write-compressor failed. The later db-wal-recovery review execution passed with one core and three specialist skills. Later write-compressor executions passed in both treatments with a transferable 600-token core and no specialist. Their retained action receipts recorded the prescribed actions and completed startup rollback.

Those are verified passing work products after structured guidance was delivered. They do not isolate the skills' contribution from another model attempt, establish aggregate superiority, or show that every task improved.

The work behind the next result

The strongest reason to study these runs is to make failure useful. A recurring mistake can become a scoped procedure, and a later execution can supply evidence about whether that procedure helped.

Our next engineering question is whether a clean, frozen study can repeat the gain over both controls while completing the full workflow reliably. This selected-subset study is not an official leaderboard submission or a claim about customer savings.

The complete task-level evidence

The table reproduces the sanitized aggregate outcomes from the revised system card for the comparison completed September 13, 2026. The September 21, 2026 evidence review supplies the retrospective analysis and separately dated later-development examples discussed above. These are reports of retained grades, not an independent rerun or recomputation of the private run archive.

All 21 analyzed tasks. Each cell is passing artifacts out of two trials, not a separate task count.
TaskUnassistedMemoryAtlasAtlas + review
distribution-search2/22/22/22/2
build-cython-ext2/22/22/22/2
extract-moves-from-video2/22/20/20/2
make-doom-for-mips *0/20/20/20/2
mcmc-sampling-stan1/22/22/22/2
mailman2/22/22/22/2
pytorch-model-recovery0/20/20/20/2
sparql-university1/22/22/21/2
gpt2-codegolf *0/22/21/22/2
rstan-to-pystan2/22/22/22/2
winning-avg-corewars1/21/22/22/2
hf-model-inference0/22/20/22/2
portfolio-optimization2/22/22/22/2
code-from-image2/22/22/22/2
regex-chess1/20/20/22/2
torch-tensor-parallelism *1/22/22/22/2
dna-assembly1/21/21/21/2
count-dataset-tokens *1/22/22/22/2
model-extraction-relu-logits1/21/22/22/2
overfull-hbox1/21/20/22/2
schemelike-metacircular-eval2/22/22/22/2
All analyzed trials25/4232/4228/4234/42

* External benchmark-answer material reached at least one execution of these four tasks. Seven executions were affected. The original outcomes remain visible; removing every arm and trial of all four tasks leaves 17 tasks and 34 trials per treatment: unassisted 23/34, memory 26/34, Atlas 23/34 and Atlas plus review 28/34. The retrospective paired p value against unassisted in this sensitivity set was 0.28125. This post-hoc exclusion is not a clean confirmatory result or proof that no other contamination occurred.

Both controls passed both trials of extract-moves-from-video while both Atlas treatments passed neither. That counterexample matters alongside the favorable outcomes. The combined workflow also used additional review computation; the counts do not isolate a skill effect or establish reliable superiority.

Artifact correctness and workflow completion are different. In the reported primary completion table, unassisted completed 25/42 and memory 32/42; Atlas plus review had 34/42 passing artifacts but 30/42 artifact-and-workflow completions. The corresponding Atlas completion count was not specified there. Completion labels were provisional machine findings, not independently human-audited safety scores or an equal-step comparison.

The original selected, qualified and analyzed cohorts contained 24, 22 and 21 tasks respectively. A QEMU VM-state capture failure left two attempted executions ungradable; all eight scheduled cases for that task were excluded from the completed comparison. The separate 24-task development cohort and the later 12-task checkpoint are not additional rows in this frozen table.

Method and source notes: revised system card pages 1-2, 5-6 and 9-10; September 21 evidence review pages 6-7 and 9-11. Original Harbor graders evaluated the post-actor artifacts. Repeated trials are clustered within tasks. The original predeclared tests used different contrasts and did not confirm their gates; the task-clustered exact two-sided sign-flip analysis is retrospective. No raw prompts, answers, transcripts or private archive links are published here.

If you build agents, the question to bring is concrete: which failure in your workflow keeps recurring after someone has already learned how to fix it?