An agent can answer fluently while accepting an assumption it should challenge. In consequential workflows, the useful question is whether the agent obtains the evidence it needs, records the boundary of that evidence, and reaches a supported answer.
In an early Dharma experiment, a workflow that records evidence as the task proceeds reduced false-premise acceptance from 9 of 100 runs to 5 of 100. Runs passing all required workflow checks increased from 70 of 100 to 75 of 100. The test used ten scenarios repeated ten times for each approach on Azure’s dharma-gpt-oss-120b deployment. Results were recorded on July 1, 2026 (UTC).
Those numbers are a bounded result, not a claim that we solved sycophancy. Five false-premise acceptances remained, and 25 of the 100 stateful runs still failed at least one release condition. Two earlier prompt-gate experiments also matter: a strong result on two selected cases did not reproduce in a broader cross-model test.
Together, these experiments show why a validation process needs to preserve failures as carefully as improvements.
Make the evidence requirement observable
The stateful experiment compared a direct-answer baseline with a workflow that exposed three environment tools: retrieve_ground_truth, write_evidence_boundary, and final_answer.
The workflow made retrieval and an evidence-boundary record observable parts of the task. Its checks covered retrieval, the boundary record, final-answer grounding, false-premise resistance, and tool discipline. The primary score came from deterministic environment checks, including lexical correction matching, rather than an LLM judge.
Both arms used the same ten scenarios with ten repetitions per scenario. That is 100 runs per arm, not 100 different problems.
| Measure | Direct baseline | Stateful workflow |
|---|---|---|
| False premise accepted | 9/100 | 5/100 |
| Protocol passed | 70/100 | 75/100 |
| Final correction supplied | 70/100 | 78/100 |
| Retrieval state created | Not applicable | 100/100 |
| Evidence-boundary state written | Not applicable | 92/100 |
The false-premise change was four percentage points, approximately 44% relative to the baseline rate. The absolute counts matter more than a headline percentage: this was a small repeated-scenario study, and the remaining failures were meaningful.
This comparison tests a workflow with tools and required state against direct answering. It does not isolate a model improvement, establish that the gain transfers to other tasks, or demonstrate production readiness. Lexical matching can also miss valid paraphrases or reward superficial matches. A human audit of the matcher and broader, independently selected scenarios would be needed for stronger conclusions.
The prompt-gate replication did not reproduce the gain
H9 expanded to ten scenarios and changed the model environment to Azure dharma-gpt-oss-120b. The completed-run update was recorded on July 1, 2026 (UTC), before the separate stateful-tools results. It planned 15 repetitions per scenario per arm: 450 trajectories, of which 446 were scored and four were provider-blocked.
The final corrected archive counts, retained in the September technical review, were:
| H9 treatment | Judged flips | Provider blocks |
|---|---|---|
| Baseline | 25/150, 16.7% | 0 |
| Experimental generic gate | 26/150, 17.3% | 0 |
| Corrective gate | 45/146, 30.8% | 4 |
The later stateful-tools test used a different intervention and a different scoring method. Its 9/100 to 5/100 finding does not erase H9, and the two studies should not be pooled. We have not verified a later replication of that same stateful comparison.
What this means for agent validation
For a team deciding whether an agent can act, a useful evaluation should make the required evidence and failure conditions explicit. It should distinguish a supported answer from a plausible answer, preserve provider blocks and failed checks, and compare the proposed workflow with an appropriate baseline.
These experiments formed an early validation foundation for the subsequent product delivery work. They are historical research results, not evidence that a later deployment inherited the same performance.
The stateful test offers a concrete example of that foundation: retrieval and evidence-boundary records became inspectable, and a bounded false-premise metric improved. The remaining failures and the negative prompt-gate replication explain why release decisions need more than a favorable chart.
Evidence and method boundaries
The aggregate tables in this article are reproduced from the September 9, 2026 technical review of the June 28-July 1 experiments. They are a sanitized account of retained results, not a new raw-data recomputation or experiment. The H8 and H9 figures use automated-judge flip metrics; the stateful workflow uses separate deterministic environment checks. Their cohorts and interventions are not pooled.
Study notes
H8 executions: June 28–29, 2026 (UTC). H9 and stateful-tools results: recorded July 1, 2026 (UTC). Technical review: September 9, 2026. This article is a new retrospective, not a republication of an earlier blog post or a fresh experiment.
The source task family derives from SYCON-Bench. These experiments are not official full-benchmark SYCON scores or official tau-bench results. The stateful test’s deterministic checks are not interchangeable with H8/H9’s automated-judge flip metric.