Valid answers still need proof of improvement.

METHOD BEFORE LEADERBOARD

Preserve what the model actually saw.

The source corpus contained 100 labeled learning garments with 377 photographs, plus 20 separately reserved validation garments with 78 photographs. Customer expert estimates supplied the learning references. These are not independently verified transaction prices. The small pilots used subsets of the learning data; they did not execute a new blind validation study.

We retained the original v190 appraisal policy, evidence order, and image hashes. OpenAI native-image trials attached original image bytes as image input. Jev’s hybrid condition used retained Gemini visual extractions; its separate Jev-only condition used deterministic image features. Text features—even base64 pasted as ordinary text—must not be relabeled as demonstrated native-image understanding.

Execution receipts bind requests, configurations, raw responses, and outcomes. A claim is persisted before dispatch; uncertain calls are not silently replayed. Completed work is reused only when its full request identity matches. Missing answers and abstentions stay in operational accuracy denominators. Usage-based cost estimates remain estimates until billing reconciliation.

RETAINED JEV TRIALS

Execution is evidence. It is not yet success.

Earlier exploration included 200 attempts across derived-evidence conditions. Those attempts remain in the history; they are not 200 successful prices. A later custody audit of Jev 1.13.0 retained 92 validated hybrid outputs against 100 planned learning targets: 23 returned prices, 69 abstained, and eight lacked validated outputs. None of those 23 prices was within 20% of its expert reference.

Retrospective scoring with the regular operational price metric gives 9.64% across all 100 planned Jev hybrid targets. All 69 abstentions and eight missing outputs contribute zero; the denominator is not restricted to the 23 returned prices.

The deterministic-feature condition retained 100 outputs and 100 abstentions. That distinction matters: the API ran, but this representation did not yield a usable valuation. We must investigate evidence sufficiency, question design, abstention policy, and task suitability before assigning a cause.

These Jev conditions and the native-image OpenAI pilots differ in representation, cohort, and execution history. They are useful diagnostic trials, not a controlled proof that one vendor is intrinsically better. A defensible head-to-head would freeze equivalent evidence and cohorts, preserve each model’s supported contract, and score failures consistently.

THE POSITIVE RESULT

An Atlas-informed procedure improved a learning score.

In the earlier same-model appraisal study, Dharma reviewed expert corrections across 100 learning garments. The Failure Atlas grouped recurring mistakes: unsupported scarcity premiums, excessive missing-tag discounts, confusion between original and later issues, and scrap-floor estimates for wearable garments.

Those patterns informed a procedural skill, not a table of target prices. Both arms kept Gemini 3.8 Flash and the original request settings. The provisional operational price score rose from 70.18% to 75.79%, a gain of 5.61 percentage points. Usable prices increased from 94/100 to 100/100.

This score rewards closeness in log-price space; it is not within-20% accuracy. Recovering six missing prices contributed 3.65 points of the gain. On the 94 jointly priced items, the average paired score gain was 2.08 points. Across all 100, 48 improved, 22 tied, and 30 regressed.

The result supports a bounded claim: expert-informed procedural changes improved this provisional learning score. The same labels helped generate and select candidates, so selection optimism remains. Independent blind pricing improvement is unverified. That is a reason to test transfer, not to discard the observed learning result—or advertise it as a production gain.

A SCORE GAIN, NOT A PASSED GATE

The Atlas candidate we did not promote.

A later Atlas organized OpenAI diagnostic events into families such as repeated reference misses, selective failure to price, and comparable-evidence limitations. Fixed-template guidance proposed attribute-first magnitude estimation and a clearer distinction between a qualified estimate and verified sales evidence. These were hypotheses, not proven causes.

The live test scheduled 80 attempts: 20 garments, two arms, and two repetitions. Ten garments were development replays; ten were fresh to the prior OpenAI request history. Both arms retained six questions, the 180-price grid, full policy text, and original images. The candidate changed only price instructions. There were 76 accepted executions and four uncertain or nonaccepted attempts.

Using the same regular price score as our previous Atlas article, the fresh-cohort control scored 27.46% and Decisions with Dharma Failure Atlas scored 29.45%: an observed gain of 1.99 percentage points. We recomputed this metric offline from the saved outputs; no new inference was run. The score is the mean of exp(−ln(predicted / expert)² / (2 × 0.9²)), with every scheduled failure or abstention contributing zero. With only ten garments, this exploratory difference is not proof of generalization.

On the fresh learning cohort, both arms achieved 0/20 within-20% accuracy. The unchanged control returned 19/20 prices; Atlas guidance returned 17/20. Priced-only mean absolute percentage error worsened from 89.1% to 98.2%. Twenty scheduled observations here represent ten garments repeated—not twenty independent cases.

A DIFFERENT ESTIMATOR

Better prices, but not enough evidence to expand.

We then froze 20 previously unattempted learning garments, balanced across five price bands, with 73 original images. Each API arm ran once per garment, plus one repeat on five frozen garments: 50 scheduled inference calls, zero automatic retries, 47 accepted executions, and three uncertain outcomes.

The control used native-image Decisions with gpt-6-luna. The explicit alternative used gpt-6.1-sol through Responses, high reasoning effort, and strict structured JSON. It retained the same policy and original images. Continuous estimates were mapped locally to the existing price grid by nearest log-price distance; raw estimates were retained.

This is an estimator-plus-protocol comparison, not a Responses fallback disguised as Decisions. The Atlas cases recorded after these outputs did not cause Sol’s improvement.

DHARMA’S METHODOLOGY

Cognitive Integrity makes improvement answerable to evidence.

For this workflow, Cognitive Integrity means preserving evidence, distinguishing observations from hypotheses, recording failures, testing frozen changes, and rejecting changes that fail acceptance criteria. The Failure Atlas is the diagnostic layer: evidence-linked recurring patterns that can inform procedural candidates. It is not an automatic accuracy guarantee.

FOR TEAMS BUILDING WITH DECISION MODELS

Start with failures you can verify.

Bring retained trajectories, representative evidence, expert references, the existing answer contract, and a business-relevant scoring rule. Separate learning from validation before tuning. Choose your promotion gate in advance, including output coverage and costly failure modes—not just accuracy on answers the model chose to give.

Then use the Atlas to propose the smallest justified change. Keep native-image and extracted-evidence conditions distinct. Compare a stronger estimator and simple baselines when the failure may reflect missing capability rather than missing instructions. A structured API can simplify execution; it cannot decide which intervention your evidence supports.

Our trials show a learning-score gain, an Atlas candidate worth rejecting, and an estimator with lower error but insufficient expansion evidence. The value is not a claim that Dharma makes every model better. It is a disciplined way to find out what improves, what regresses, and what should remain out of production.