Model checks

#The verdict

Each test replays 3 times per side; a test that dropped, improved without significance or could not be compared gets 5 per side. A test is compared only when at least 2 replays per side reached the point it checks and none broke, and it regressed when the candidate passes it significantly less often (one-sided Fisher exact p < 0.05), the statistics fix proofs use. The check is regressed when any test regressed, pass when every test was compared and none passed less often, and inconclusive otherwise.

Before it starts, a check reserves the most it can run: 2 sides × 5 replays × the agent's tests, and what it does not use returns when it ends. When the plan has fewer simulated conversations left this month, it is refused before any replay, saying what it needs and what is left.

#New models

Every new model is cheaper or faster and may break your agent. A model check answers whether a hosted prompt agent can switch: it replays the agent's tests on the live configuration and on the live prompt run by a catalog model, judges every test and gives the verdict: pass (safe to switch: every test compared and none passed less often), regressed or inconclusive. Its card quotes the verifier's reason for each test the candidate failed. Start one with Check beside a model in the agent's Models; before anything runs it states what it reserves, 2 sides × 5 replays × the tests of findings not dismissed, stress tests' and production's, and starts on a second click. A test whose last replays on live never got to the point it checks is left out: rebuild it. When no test reaches its point the check does not start, and Models says why with Rebuild tests.

The catalog is the models Duet Machines hosts prompts on, each with its provider's list price per token; a model without a published list price shows none, and its cost reads unknown. Duet Machines adds a model once it has its list price and has checked that it runs with tools; that addition is the release. Duet Machines also reads OpenAI's, Google's and Together's model lists daily and shows the models they list that the catalog does not hold yet, which cannot be checked until they are added.

Watch new releases (owners and admins, off by default) checks the agent once on every model added after the watch started, within a monthly budget counted in checks of the current tests; a release the budget or the plan cannot pay for is listed as not checked, with why, and never retried. The verdict goes to the workspace's alert channels (Slack, Teams, webhook, and email to members who opted in to model verdicts) and shows on Home and on the agent.

Cost and latency are measured on the check's replays: the agent's own model calls at list price and the time to its first reply to each customer message, excluding simulated tool time. The savings line multiplies the agent's priced production spend of the last 30 days by the measured cost ratio, and is labelled a projection. Apply as configuration adds the model as a configuration to stress test and compare with. Endpoint agents are not checked: Duet Machines' contract carries no model, and a Chat Completions endpoint decides itself whether to honor the model it is sent.

#Savings opportunity

Once production conversations arrive (SDK, API, OpenTelemetry, LangSmith or Braintrust), the agent's Models shows what it spent by model over the last 30 days: the tokens its agent turns and model spans report in usage, at each model's list price (cache reads at its cached rate where usage reports them; calls that do not report caching are counted beside the figure, since their input is priced at the full rate), a model without one marked unpriced, and how many conversations have a known cost. Each catalog model that is cheaper on those same tokens at its list price shows that cost as a bar against what production spent, cheapest first, as a projection until a check finishes: list prices say nothing about whether the model holds your agent's behavior. The page leads with the one next step: the cheapest model that passed a check, or else the cheapest projection to check.

Check on a candidate runs a model check on it. An agent without tests gets them first, and the confirmation says so with the most it reserves: Duet Machines waits until its production conversations are judged (not past the plan's judged conversations this month: if they run out, it goes on with what is judged and, finding no issue, says how many conversations stayed unjudged), reproduces its most frequent open production issues (up to 3, as the plan's simulated conversations allow), and starts the check as soon as a reproduction becomes a test; if none recurs, it says so instead of checking. A finished check's verdict replaces that model's projection, and a regressed or inconclusive one offers Make it pass.

#Gate a deploy

The CLI replays the agent's tests on one configuration and exits 1 when a test failed or broke, or 3 on an error or timeout. It needs only Python 3:

DUET_API_KEY=... python3 cli/duetmachines.py run --agent <uid> [--variant <uid>]

{"status": "failed", "passed": 4, "failed": 1, "not_reached": 1, "broke": 0, "results_url": "https://…/comparisons/<job>"}