Evaluations and Failure Atlas

Dharma evaluates actual organization trajectories rather than treating a provider connection as proof of quality. Evaluation scopes can cover the organization, selected logical agents, or selected endpoints across local, managed, and BYOK execution.

Product boundary

Evaluation Runs and Agent Fabric are separate product workflows.

Evaluation Runs tests a production-like agent through versioned tasks, controlled execution arms, scorer evidence, Failure Atlas analysis, remediation candidates, held-out backtests, and reports.
Agent Fabric connects repository-scoped coding agents and devices for signed tasks, trajectory capture, governed skills, and rollout receipts.

An Evaluation Runs workflow must not invoke Agent Fabric enrollment, relay, workspace, signed-task, or skill-rollout commands merely because a repository contains an Agent Fabric skill. A deliberate cross-product handoff may send a validated remediation into Agent Fabric, but the products do not share setup or completion criteria.

Every Evaluation Runs campaign reports productScope: evaluation_runs and one integration mode:

observe_existing_runtime is the default. The customer production provider, model, credentials, dependencies, deployment, and synchronous response path remain unchanged. Evaluation runs through a separate customer-hosted GCP BYOK or Dharma-managed endpoint.
migrate_to_dharma_managed is an explicit architecture change. Preserve the original provider behind a tested feature flag until an isolated smoke test, canary, and rollback drill pass. Never perform this migration merely to enable evaluation.

If the selected evaluation endpoint is a copy of the production agent, label the resulting evidence as evaluation evidence. Do not claim it is live production telemetry.

Observe the existing Gemini runtime

The versioned evaluation sidecar records completed provider interactions without replacing or proxying the existing Gemini call. Return the provider response through the customer application first, then schedule the sidecar request from a trusted server process. Use a durable outbox or the hosting platform's after-response task mechanism when the process may terminate after sending the response. A Dharma timeout, outage, full queue, or judge outage must not alter the customer response.

Endpoint: POST /api/orgs/{orgId}/evaluation-sidecar/events
Contract: dharma.evaluation-sidecar.gemini/v1
Token scope: evals:run
OpenAPI: /api/v1/evaluation-runs/openapi.json
Generated TypeScript example: /examples/evaluation-sidecar-gemini-v1.ts

Use a unique Idempotency-Key tied to the provider request. Reusing the key or provider request identifier returns the existing receipt and does not create another scoring job or provider charge.

At most eight JPEG, PNG, or WebP images are accepted in total. For image-aware semantic evaluation, use native Gemini inlineData: { mimeType, data } parts containing base64 image bytes, with a combined decoded limit of 3 MiB. The server validates signatures and size, and the judge receives the validated images rather than only their references. Image receipts contain SHA-256 hashes, not image bytes.

For deterministic observation without image interpretation, HTTPS fileData references remain supported with MIME type, SHA-256, and byte size (at most 20 MiB per referenced image). Signed URL query credentials are removed before storage. Semantic requests containing reference-only images are rejected; Dharma does not fetch arbitrary image URLs. Use the inline-image starter only after the corresponding server release is available to your organization.

metadata_only is the default content policy. It stores text hashes and lengths rather than text and discards inline image bytes. Semantic scoring requires customer_authorized_content, semantic: true, and semanticMaxCredits. Only enable content disclosure for data your organization authorizes Dharma to evaluate. The synchronous endpoint checks the ceiling but does not debit it; the worker charges rated usage once only after a successful semantic result.

Observation charges

Capture and deterministic checks: 0 Dharma credits per observation under the current service configuration. These checks do not call an LLM and are not a semantic appraisal or a customer-specific evaluator.
Optional semantic evaluation: variable Dharma credits based on the evaluation's measured usage and your organization's active pricing. There is no universal flat per-observation price. The worker debits once after successful scoring, with a minimum of one credit, and only within your semanticMaxCredits ceiling.
Budget example: semanticMaxCredits: 500 authorizes at most 500 credits for that observation. It is a limit, not a quote or an upfront debit. At $0.0002 per credit, that limit is $0.10. A ceiling failure is reported without a customer charge; it must not alter the original application response.
Existing provider: your original Gemini call remains on your existing provider account. Recording its result does not run that appraisal again through Dharma. Evaluation campaigns and managed production executions are separate operations with separate usage.

Start with deterministic observation, then explicitly opt a small representative sample into semantic evaluation. Review settled usage before choosing a production sampling rate. Do not extrapolate semantic charges from the original appraisal's token count: the evaluator has its own input and output.

Use the versioned public starter above instead of old emailed copies. Starter v3 accepts up to eight images, supports native inline bytes, and requires explicit organization and model inputs. It never supplies a default customer organization. geminiRequest and geminiResponse must follow the documented sidecar subset, not an unrestricted raw provider response. Supplying semanticMaxCredits opts into semantic evaluation; omitting it keeps deterministic-only capture.

const providerResponse = await callExistingGemini(geminiRequest);
sendApplicationResponse(providerResponse);

// Import enqueueCompletedGeminiObservation from the versioned starter.
// Execute this in your durable after-response task, not the response path.
await enqueueCompletedGeminiObservation({
  apiOrigin: 'https://www.dharma-ai.io',
  organizationId: process.env.DHARMA_ORG_ID!,
  organizationToken: process.env.DHARMA_ORG_TOKEN!,
  model: actualProviderModel,
  providerRequestId,
  traceId,
  agentRevisionId,
  contentPolicy: 'customer_authorized_content',
  geminiRequest: geminiRequestWithInlineImages,
  geminiResponse: providerResponse,
  semanticMaxCredits: 500,
});

Retain the returned eventId and read GET /api/orgs/{orgId}/evaluation-sidecar/events/{eventId} until completion or failure. HTTP acceptance is not a completed evaluation. A generic semantic finding is not the same as your customer-specific numerical scorer; that requires an installed, versioned evaluator and parity verification.

End-to-end run

Record and smoke-test the existing production provider path.
Select observe_existing_runtime unless the organization explicitly requests a production migration.
Bind an isolated Dharma-managed or customer-hosted GCP BYOK evaluation agent.
Create a representative task package with visible evidence and scorer-only expected boundaries.
Validate the package and campaign contract locally.
Call preflight and review task count, trajectory count, arms, hard gates, and maximum credits.
Launch once with a unique idempotency key.
Poll every session and trajectory to a terminal state; a campaign acceptance response is not completion.
Audit the verdict contract, trace, tool receipts, scorer version, and evidence boundary for each result.
Review Failure Atlas clusters and remediation candidates.
Run a candidate-bound held-out backtest before approval or cross-product rollout.
Rerun the original production smoke command and verify its provider path is unchanged.

Evidence sequence

Capture: an endpoint records a trajectory and its provenance.
Deterministic checks: coverage, runtime failures, secret boundaries, redaction receipts, tool discipline, and incomplete evidence are evaluated without an LLM.
Evidence selection: exact trajectories or spans are selected for deeper review under the organization evidence policy.
Semantic judgment: approved evidence is scored against versioned rubrics with model, prompt, confidence, usage, and cost metadata.
Failure Atlas: recurring findings are grouped with evidence lineage, affected agents, business consequence, and remediation state.

Metadata can identify where review is needed. It cannot by itself prove factual grounding, reasoning validity, evidence interpretation, or sycophancy. If semantic evidence is unavailable, the result must remain insufficient_evidence rather than become an invented score.

Analysis windows

For recurring production learning, use the Automatic learning API. It has a separate enrollment, full-window review, calibration and gated-rollout contract. The manual analysis and Evaluation Runs interfaces below do not enroll production traffic into a learning policy.

Automatic analysis closes exact trajectory windows under the organization policy. The default production target is 100 completed trajectories; incomplete windows carry forward. Incident and manual requests may close an explicitly smaller window.

Use POST /api/v1/orgs/{orgId}/agent-fabric/evals with evals:run to request an organization, logical-agent, or endpoint scope. Use a unique idempotency key. Results are visible through evaluations, traces, and Failure Atlas resources.

Evaluation contracts

Generated rubrics and decision contracts are versioned proposals. An organization admin can activate, reject, or retire a contract through POST /api/v1/orgs/{orgId}/agent-fabric/evaluation-contracts. Activation binds future decisions to a specific contract version; it does not rewrite historical results.

Contracts are created by a completed Agent Fabric analysis window from captured trajectories. The public endpoint intentionally does not accept an arbitrary contract-create body: GET lists generated proposals and POST transitions one proposal. To obtain a contract, first connect a repository endpoint, capture eligible trajectories, and request analysis through POST /api/v1/orgs/{orgId}/agent-fabric/evals; then review and activate the resulting proposal.

Managed evaluation campaigns

Download the task-package template and validate it against the JSON Schema. A package contains:

visible task evidence supplied to the evaluated agent;
scorer-only hidden truth, which is never included in the runtime prompt;
the mandatory cognitive-integrity-v1 standard and its exact hard gates;
customer-configured task criteria using supported state, action, tool and threshold checks;
operational limits and required trace fields;
a direct-versus-stateful or held-out comparison contract;
the fixed release-decision expression.

The public starter uses executable standard gates and does not require a custom-domain evaluator or a preinstalled customer tool. Replace its example scenario and hidden truth with authorized cases, configure expected_state_gate, and set each task's scoring.pass_threshold (a percentage, for example 85). Add tool_expectations.should_call_or_reference only for actual registered adapters your task requires. The scorer checks a real evidence-bearing invocation, not merely a tool name mentioned in the answer.

New custom dimensions in evaluation_contract.customerDomainRubric.dimensions are not executable in the current managed campaign runtime and are rejected during preflight. Omit customerDomainRubric or leave its dimensions empty. This is separate from configuring supported task criteria; free-text success_criteria do not install a new semantic judge. Uploaded scripts and arbitrary verifier commands are rejected. You can keep a private domain scorer in your own stack and join its versioned outputs by task ID without making that scorer a prerequisite for standard evaluations.

In the portal, import the package through Evaluation Runs > New Evaluation. The Rubrics tab displays the selected campaign's scoring contract and results; it is not a free-form evaluator editor. For API use, wrap the same tasks and contract in a campaign request:

const campaignRequest = {
  name: 'Customer workflow smoke',
  agentId: process.env.DHARMA_AGENT_ID,
  packageId: taskPackage.package_id,
  integrationMode: 'observe_existing_runtime',
  arms: ['direct_baseline', 'stateful_dharma_runtime'],
  tasks: taskPackage.tasks,
  evaluationContract: taskPackage.evaluation_contract,
};

Run preflight before the separately authorized launch. A task-package schema validates the file shape; runtime preflight also checks campaign semantics and authority. Changing the package cannot silently switch the production provider or authorize a paid run.

Preflight

Call preflight with the same body that will be launched:

curl --fail-with-body \
  -X POST "https://www.dharma-ai.io/api/orgs/$DHARMA_ORG_ID/managed-evals/preflight" \
  -H "Authorization: Bearer $DHARMA_ORG_TOKEN" \
  -H "Content-Type: application/json" \
  --data @campaign-request.json

The response reports the validated task count, generated trajectory count, exact arms, hard gates, maximum credit preauthorization, and invoice-equivalent value. Preflight does not execute a model or debit credits.

Launch

After reviewing preflight, launch the identical request with a unique idempotency key:

curl --fail-with-body \
  -X POST "https://www.dharma-ai.io/api/orgs/$DHARMA_ORG_ID/managed-evals/campaigns" \
  -H "Authorization: Bearer $DHARMA_ORG_TOKEN" \
  -H "Idempotency-Key: eval-$(date +%s)-$RANDOM" \
  -H "Content-Type: application/json" \
  --data @campaign-request.json

direct_baseline and stateful_dharma_runtime run the same visible tasks. held_out_backtest runs alone against a candidate-bound immutable test revision. Store the returned campaign identifier and inspect individual verdicts rather than relying only on aggregate percentages.

Authoritative verdict

Each completed trajectory exposes metadata.verdict with:

contractVersion and scorerVersion;
the authoritative taskScore and threshold;
every deterministic hard gate and its observed value;
failed gate IDs and readable reasons;
customer-rubric observations and governed-evaluation status.

Free-text success_criteria remain non-gating observations. They cannot fail a run by lexical mismatch. A requirement becomes release-authoritative only through a standard gate, a canonical deterministic remediation requirement, or a governed evaluator bound by the evaluation contract.

Interpreting results

For every score, verify:

the evaluated organization, agent, endpoint, revision, and trajectory IDs;
the evidence revision and content authority;
deterministic and semantic findings shown separately;
the rubric and judge version;
confidence, cost, and failure state;
whether the result is source data, held-out data, or a regression control.