FAQ

#How accurate is the judge?

Measured against a frozen gold set of 39 labeled conversations from 3 agents, 8 of them from a holdout agent never used to tune the judge. The labels hold 34 expected failures (6 critical, 14 high, 14 medium), 16 forbidden findings the judge must not report and 21 borderline ones that are never scored, each with its source: a reviewer, a contested critic's reading, or the gold entry's author. A finding is matched to a label by the agent turns it cites, never by a model. The deployed judge ran over the set 2 times for the committed baseline:

Precision

86.2% of the findings that landed on a labeled failure or a forbidden one were real; 84.9% when findings on unlabeled turns also count as wrong.

Recall

95.0% of critical and high failures found; 64.3% of medium ones. The judge is tuned to prove rather than to flag everything, so it misses 35.7% of the medium deviations.

Severity

75.0% exact, 98.2% within one level.

Owner

100%: agent, instruction conflict or platform defect.

Trap outcome

100%: failed, held, never reached or cut.

Agreement

79.5% overlap of the labels flagged between two runs on the same conversations.

Holdout agent

Precision 100%, critical and high recall 100%, medium recall 75.0%, trap outcome 100%, on 8 conversations: a small sample.

Every judge change is measured against this baseline, and the eval flags a drop of more than 5 points in precision, critical and high recall or trap accuracy. Rejected findings stay listed with their reason on each stress test, so you can audit the checks yourself.

#Does Duet Machines call my real tools?

No. Every tool call in a stress test, test or fix proof is answered by the scenario's scripted results or a simulation of the tool from its schema and the scenario's world. A Chat Completions endpoint that runs tools itself does touch your systems, which is why its tools must come back as tool calls. See Tool calls.

#Why was my conversation not judged?

A conversation is judged once it ends: send ended: true (the SDKs' .end(), or the span attribute duet.conversation.ended over OpenTelemetry), or it ends after 30 minutes without new turns. It waits when the month's judged allowance is used up, and it is never judged when it was sent as a check or an exclusion rule matches it. A conversation already judged is judged again only when its turns change.

#Why is "never reached" not counted as a pass?

A conversation that never reached the point a test checks says nothing about how the agent handles it, so counting it either way would move pass rates without evidence. It is shown, and a test that keeps not reaching that point is flagged so its scenario can be rebuilt.

#My fix clearly helped. Why is it not proven?

Proof needs patched to pass significantly more often than live (one-sided Fisher exact p < 0.05) with no other test regressing. Few replays cannot reach that: 0 of 3 to 3 of 3 is p = 0.05 exactly, so Duet Machines adds replays to the tests that need them, up to 10 per side. A cause that improved without significance is suggestive, and the report says what kept it from proof. See Fixes.

#What happens to my production data?

It is redacted before it is stored, stays in your workspace, is never used to plan stress tests, and is deleted after the workspace's retention (90 days after its last turn by default); an admin can delete one user's data at any time. It trains a model only through your own datasets with the workspace's training consent on.

#Can I pip install the SDK?

Not yet: the Python and TypeScript SDKs are not published, and the registry names duetmachines and @duetmachines/sdk are not ours. Send conversations with one HTTP request or OpenTelemetry, which need nothing installed; the setup prompt writes that request for your stack.