The judge

#Graders decide, the judge finds

A conversation's verdict, pass or fail by your expert's categories, comes from the published graders and nothing else: Overview's pass rate, Traffic's Health, the migration report and the rollout's gates all count it. Before a grader is published a conversation has no verdict, and no page puts the judge's opinion in its place.

The judge reads each ended conversation once, in the same pass the graders grade it, and writes three things: a one-line summary, the signals it hit, and issues. An issue is a rule of the instructions the agent broke, proven with verbatim quotes. Issues land in the agent's Review, under Issues, for you to accept or dismiss; an accepted issue that recurs when replayed becomes a test, and a conversation where the graders and the judge disagree heads the studio's Disagreements queue.

#What the judge sees

The judge sees only what the agent saw: its instructions, tools, runtime notes, the lead data sent and the transcript, plus a neutral locator of the scenario's decisive point. It never sees the attack's hypothesis, objective, persona brief, world or the findings already on record. A production conversation is judged against the config snapshot it was sent with, so findings cite only rules the agent had; without a snapshot it is judged against the agent's live version, and its findings say so.

#Checks in code

The judge's rules are enforced in code, evidence by evidence, and a finding is rejected when what remains proves nothing:

• A misquote is removed; the rule must be quoted verbatim from the instructions.
• Customer lines may only precede the agent's last cited act: they show what the agent was told, never the harm.
• A finding that rests only on where Duet Machines cut a conversation, or counts a reply the cut never let come, is rejected.
• A finding cites one directive: a sentence holding several ("do not offer a plan: call end_call") is split into its directives once per sentence, and each is its own rule. A quote of several directives cites them together. A finding that cites the same acts and directive as one kept before it is rejected as a repeat; one act breaking two directives stays two findings.
• A self-report from the agent is context, never evidence.

Rejected findings stay out of the inbox and fixes; the stress test page lists them with the reason, so the checks can be audited. Every finding stores the judge (prompts, schemas and model) and the transcript render that filed it.

#Severity and owner

Severity is critical for irreversible harm (a wrong amount, a compliance breach, confirming an action that never happened), high for a lost or damaged outcome, wrong data or tool misuse with consequences, and medium for a deviation with a concrete cost in that conversation. How a turn is put together is never a finding when it cost nothing. Repeats are merged into one issue, which takes its worst occurrence's severity.

Agent

The model broke a rule it had. Fixed in the prompt, the model or the tools.

Instruction conflict

Two of your instructions: the one broken and the one that required breaking it. The defect is in the prompt, and the prompt owner fixes it by editing the instructions.

Platform defect

A decision of the system around the model that contradicts your instructions: a guardrail that rejects a correct amount or blocks an authorized handoff, or a call the system let through although it must refuse it. The platform team fixes it.

#Test verdicts

Each cell of a comparison holds one verdict per conversation:

✓ Handled

The agent did what the test expects.

✕ Failed again

The failure the test was built from happened again.

– Never reached

The conversation never reached the point the test checks. Reported, not counted.

! Broke

The configuration stopped answering or errored, so it could not hold the conversation.

How accurate the judge is, and how that is measured, is in the FAQ.