Changing the question helped Jev return far more usable prices. It did not close the appraisal gap.
Jev is TypeSafe’s model for decisions among defined answers, including decisions from images. Gemini supplied the open-ended appraisals in our regular pilot. We evaluate both with the same operational pricing score, rather than treating a valid JSON response as a correct valuation.
Failure Atlas is our method for turning recurring, evidence-linked failures into testable changes. Here it exposed an awkward pattern: Jev often chose “insufficient evidence” even when its other outputs supported making an estimate. We tested a new decision structure instead of declaring those photographs unusable.
THE GAIN, AND THE REMAINING GAP
More prices. Better score. Still behind Gemini.
| Study / condition | Regular score | Prices | Within 20% |
|---|---|---|---|
| Original study: native Jev control | 20.90 | 36/100 | 6/100 |
| Follow-up: separated flat Jev | 46.05 | 74/100 | 14/100 |
| Follow-up: two-range Jev beam | 50.20 | 91/100 | 17/100 |
| Historical regular Gemini baseline | 70.18 | 94/100 | 18/100 |
| Historical regular Gemini + Atlas v3 | 75.79 | 100/100 | 23/100 |
Our regular score is 100 × mean(exp(−ln(predicted/expert)² / (2 × 0.9²))). It gives partial credit for log-price closeness. Failures and abstentions contribute zero; the denominator stays at 100. The within-20% column asks a stricter, different question: how often was the estimate close to the expert reference?
The two-range follow-up gained +29.30 score points over original native Jev. Its adjusted 97.5% paired-bootstrap interval was +19.54 to +39.03. Against historical Gemini + Atlas, it remained −25.59 points behind, with an adjusted interval of −35.45 to −15.75.
Those are substantial observed differences on this reused learning cohort. They do not turn an adaptive follow-up into independent confirmation. The practical verdict is narrower: the revised Jev system is much better than our earlier Jev protocol, but it has not beaten the regular Gemini estimator here.
DIAGNOSE THE NONPRICE BEFORE CHANGING THE PROMPT
One broad abstention competed with 180 exact prices.
The original price question offered 180 numeric choices, an insufficient-evidence answer and an outside-range option. The broad insufficient-evidence answer won 64/100 times. None of the 36 numeric selections were subsequently discarded by our local evidence-sufficiency gate.
In all 64 abstentions, numeric choices collectively held more than half the returned probability mass—83% on average. A separate evidence question judged 62 of those 64 garments sufficient for a qualified estimate. One broad abstention could outrank every individual price while the many numeric alternatives collectively dominated.
This identifies a problem at the decision interface, not Jev’s hidden reasoning. The returned probabilities are not verified calibration, and a sufficiency answer does not prove that the photographs reveal market value. But the receipts do not support interpreting all 64 abstentions as genuinely unusable images.
That distinction matters for remediation. Adding more policy text to the same overloaded choice might leave the failure intact. Separating “can this be estimated?” from “which price is appropriate?” gives each judgment a clearer job.
A SEPARATE FOLLOW-UP, NOT A REPLACEMENT RESULT
Estimate eligibility, then magnitude, then price.
We tested two new native-image conditions. The flat condition separated eligibility from a conditional 180-price question. The beam condition first considered ten price ranges, retained the two most plausible ranges, and then evaluated 18 prices within each in a second image-bearing request. Keeping two paths avoids committing everything to one early range choice.
- Eligibility and magnitude: assess whether an estimate is usable; retain two of ten ranges.
- Conditional price: ask an 18-price question for each retained range. First-stage visual judgments remain unverified observations.
- Local decoding: combine returned numeric probabilities and choose the grid price with the highest expected regular-score utility. Preserve explicit unusable and outside-grid outcomes.
This follows the broad decomposition pattern in TypeSafe’s hierarchical-classification cookbook; our fixed-depth price decoder is application-specific. It is not simply the model’s highest-probability categorical answer. We retain that alternative too: categorical-only decoding scored 28.80 for flat and 48.82 for beam, compared with 46.05 and 50.20 under the primary decoder.
Question structure, eligibility handling and probability decoding all changed. We therefore cannot attribute the whole gain to better visual understanding or hierarchy alone. Nor is 50.20 a result of appending the old Atlas v3 procedure: the follow-up used the control policy, not that addendum. Atlas guided the diagnosis and intervention test; its earlier transfer result stays separate below.
All 91 completed first-pass beam decisions produced prices. The other nine were execution failures with uncertain transport outcomes—not model abstentions—and still scored zero. Flat had three execution failures and 23 outside-grid selections. Returning more prices improved coverage, but did not make those prices accurate by definition.
THE REMAINING ERROR IS LARGELY VALUATION
Jev still loses the order of magnitude.
The expert’s nearest-grid price branch survived in the two retained ranges for only 43/91 completed beam decisions. Keeping two paths helped relative to greedy selection, which retained it in 30/91, but too many estimates still started in the wrong part of the price scale.
The expensive garments were especially difficult. Among 17 garments with expert references of at least $2,000, the median estimate-to-expert ratio was only 5.12%. Sixteen were underestimated by more than 20%. All 17 completed, so this weakness cannot be explained by service failures.
Even assigning a perfect score to all nine failed beam decisions would raise its overall score only to 59.20, holding the completed outputs fixed. That is an upper bound, not a predicted recovery result, and it still falls below Gemini’s 70.18 baseline. Mapping Gemini’s continuous estimates onto Jev’s same grid also leaves its scores essentially unchanged: 70.22 baseline and 75.80 with Atlas.
Jev did beat Gemini + Atlas on 29 individual garments, while losing on 71. That does not establish an aggregate advantage or a deployable routing rule. We have evidence that this Jev configuration trails the retained Gemini estimator—not proof that no future Jev system could ever do better.
ORIGINAL STUDY / OCTOBER 16 / UNCHANGED RESULTS
Vision and Atlas transfer were different tests.
The original four-arm study compared native photographs with retained visual extractions, each with and without the successful regular pilot’s frozen Atlas v3 procedure. Native arms covered 100 garments; extracted arms covered the 94 with complete retained extractions. The missing six were not recreated.
| Original contrast | Control → comparison score | Difference | Adjusted 97.5% interval |
|---|---|---|---|
| Extracted → native control, same 94 garments | 14.86 → 20.22 | +5.36 | −0.99 to +12.29 |
| Native control → native + Atlas v3, all 100 | 20.90 → 20.83 | −0.07 | −2.52 to +2.02 |
Neither original contrast passed the frozen improvement rule: an adjusted interval excluding zero, at least five score points of gain, and no decline in pricing coverage or within-20% accuracy. Native vision had a positive point estimate, but its interval crossed zero. Atlas transfer was near zero; both native arms priced 36/100 and reached 6/100 within 20%.
The fourth arm, extracted evidence + Atlas v3, scored 16.04 against extracted control’s 14.86. Both priced 29/94 garments; within-20% results were 2/94 with Atlas and 3/94 without it. These remain original-study outcomes, not results from the decomposition follow-up.
Our successful regular appraisal pilot remains a third, distinct result: its provisional learning score increased from 70.18 to 75.79. The procedure was developed using those learning labels. Its success with Gemini did not guarantee transfer to Jev, and neither study supplies independent blind pricing improvement.
Failing a relative improvement rule is not the same as failing an absolute readiness test. Conversely, the follow-up’s large relative gain does not establish production readiness. The original study and the optimization answer different questions; both remain in the record.
METHODS AND EVIDENCE BOUNDARY
Same learning garments. Explicitly different protocols.
Both Jev studies used jev-1.14.0, the complete retained v190 policy, the existing 180-price grid from $5 to $30,000, and native input from 377 photographs. We preserved 360 files unchanged and proportionally resized 17 oversized photographs across eight garments. The follow-up reused exactly those submitted image bytes. No crops, contact sheets or dropped images were introduced.
Images were sent as referenced multipart assets, not base64 characters embedded in ordinary prose. TypeSafe’s October 14 release and image documentation describe native images and opt-in long context as experimental, not production GA. We fixed both feature headers across the tested Jev conditions.
Expert prices are appraisal references, not observed transaction prices. Neither targets nor old appraisal estimates were sent to Jev. Historical Gemini results came from retained gemini-3.8-flash appraisals; they were not concurrent randomized vendor arms. Gemini received original photographs, so overlapping garment identities do not mean identical submitted bytes for the eight resized garments. On the 92 unchanged-image garments, beam still scored 50.43 against Gemini + Atlas’s 77.02.
Each Jev protocol was internally frozen before its inference calls—not publicly preregistered. Twenty frozen garments were repeated per arm in each study; repeats are not additional independent garments and do not enter the primary first-pass table. The follow-up reused already-observed learning outcomes. It is exploratory optimization, not held-out confirmation.
Intervals use 20,000 paired garment-bootstrap draws, seed 190, and Bonferroni-adjusted 97.5% intervals for the two prespecified contrasts within each study. Those intervals quantify sampling variation on this cohort; they do not correct away adaptive learning-data reuse. Private receipts preserve requests, raw responses, image hashes and failed attempts without automatic retries.
A second Python implementation by the same agent reproduced scoring and checked image-byte custody and usage arithmetic. That is a cross-check, not an external audit; the bootstrap draws were not separately reproduced. Accepted usage implies approximately $0.51 for the original study and $0.40 for the follow-up at the recorded rate. These are usage-rated estimates, not verified settled bills.
WHAT FAILURE ATLAS MAKES USEFUL
Keep the improvement. Keep the failure visible.
The useful loop was concrete: inspect the abstentions, distinguish model choices from local filtering, propose a smaller sequence of judgments, freeze it, then score every scheduled target. It produced a better Jev estimator and located the remaining bottleneck in price magnitude, rather than letting API completion stand in for appraisal quality.
That is Cognitive Integrity in practice: preserve observations, label hypotheses, keep failures in the denominator, and refuse a stronger claim than the test supports. The result does not justify replacing the broad Gemini valuation workflow. It also does not erase the progress from 36 to 91 usable prices.
For the next Jev trial, we would start with a bounded visual decision—category, readable-label presence or visible defects—against independently supplied human labels. These are proposed experiments, not measured wins. A learned price head would need a separate development set and genuine held-out confirmation before expansion.
Sources and related studies
- TypeSafe: Jev 1.14 release, October 14, 2026
- TypeSafe: native image input and experimental limits
- TypeSafe: hierarchical classification and beam search
- Dharma AI: the regular appraisal pilot and its learning gain
- Dharma AI: earlier Jev and OpenAI Decisions trials, under different protocols
This update adds the decomposition follow-up without replacing the original study’s outcomes. Only anonymized aggregates are public. Customer photographs, raw trajectories and private receipts are not included. No blind-validation result or production qualification is claimed.