Savings

#Switches

A switch is recorded two ways. On a hosted prompt agent, Switch live on a passing model check runs the agent on the checked configuration from then on (POST /api/agents/{uid}/switches), refused unless the check passed on the configuration that is live now. Any other agent switches in its own code: send the model in the config snapshot (config.model, or gen_ai.request.model on the OpenTelemetry span that carries the snapshot; Braintrust, LangSmith and Vapi imports fill it from the model they log, and leave it out when they log none), and the first conversation naming a model the agent never ran records a switch from the model before it.

A switch is verified when a passing Duet Machines check justified it, or for a production switch, when a check of that model on the agent passed in the 90 days before it. Others are shown as not checked.

#How savings are measured

The baseline is the agent's cost per production conversation in the 30 days before the switch, priced at the earlier model's list price on the switch day: if that price later falls, the baseline falls with it on its logged tokens; a rise never raises it. Each UTC month after the switch, gross savings are (baseline minus the month's cost per conversation) times the month's conversations. Costs come from logged tokens at list price or the cost each conversation reports, with a 95% interval and the share of conversations priced.

Each declared outcome's rate after the switch is compared with the baseline by Fisher's exact test. A loss significant at p < 0.05 is subtracted at its measured value (the change of the amount per conversation times the month's conversations); an inconclusive loss is shown and not subtracted.

#Outcome guard

For 14 days after a switch (1 to 90, set per agent in its Models), Duet Machines compares the agent's primary declared outcome and its rate of conversations with an issue with the conversations before the switch, using a two-sided test of the two rates. A significant loss sends one alert through your channels (with none set, the page says it was not delivered, and it is sent once you set one, while the switch is still live), titled with what was lost and its p, with Revert configuration for a hosted agent, or copy for your coding agent when the switch was made in your code. No significant difference is shown on the page and sends nothing. Duet Machines never reverts on its own.