Gateway and rollout

#Connect the gateway

The gateway is OpenAI Chat Completions. Point your client at it with the agent's gateway key (created by owners and admins in the agent's Rollout, Gateway, shown once) and send Duet-Conversation-Id, your id of the conversation, on every request: a rollout keeps each conversation on one model by it, and a request without it is refused with a 400.

import os

from openai import OpenAI

client = OpenAI(base_url="https://platform.duetmachines.com/api/gateway/v1", api_key=os.environ["DUET_GATEWAY_KEY"])
reply = client.chat.completions.create(
    model="ignored",  # the gateway sends its own upstream's model
    messages=messages,
    extra_headers={"Duet-Conversation-Id": conversation_id},
)

The body goes upstream as you sent it except model, which is always the upstream's own; tools and tool calls pass through unchanged, and streamed answers are forwarded event by event. Every answer carries Duet-Arm: incumbent (the current model) or candidate. The gateway's own refusals are OpenAI-shaped: 401 for a bad key, 400 for a missing header or a body that is not Chat Completions, 409 before the current model is set, 429 over the rate limit and 502 when the upstream is unreachable.

Owners and admins set the workspace's requests per minute, across all its agents, in Gateway before any request is served: until then every request answers 503 rate_limit_not_set. Over the limit a request answers 429 rate_limit_exceeded with Retry-After, nothing goes upstream, and the throttled request shows in Requests without counting in any gate or cost.

Every served answer becomes a production conversation of the agent, redacted like any other, so grading, outcomes, Review, tests and datasets read gateway traffic as they read the rest. See Send conversations.

#Current model and candidates

The current model (incumbent in the API) is the model the agent runs today; candidates are the cheaper ones, a model you trained among them (see Datasets and training). Each is an OpenAI-compatible base URL, an API key kept sealed and write-only, the model and your own prices per million input, cached input and output tokens: nothing is looked up, so the cost gate prices both models the way you are billed. Base URLs are https and must resolve only to public addresses.

#Stages

Passthrough

The current model answers every request, and each is logged with its tokens, latency and cost.

Shadow

The current model answers; the candidate gets the same request in the background and its answer is stored, never returned or executed. It compares the same requests, so it says nothing about outcomes.

Split

The candidate serves a share of conversations, assigned at random and sticky: raising the share keeps every conversation already on the candidate there. Only this stage is randomized, so only its differences are causal.

Full

The candidate serves every conversation. A candidate that fails before any byte was streamed is retried once on the current model, and the failover counts against its error gate.

#Gates

Each gate is computed live per model over the stage's window, with each model's value, sample and 95% interval. A gate fails when the candidate is significantly worse and passes once each model has the minimum sample without that; a pass says no harm showed at that sample, not that the models are equal.

Each gate draws the candidate's 95% interval as a bar over a red zone: past the current model's p95 plus your margin for latency, the worse side of no change for cost and rates. The gate fails once the whole bar lies in the zone.

Cost

Cost per conversation, priced whole.

Compliance

Conversations your published graders fail, in any category, of those every one of them ruled: your standard, as your expert set it. Until a grader is published the gate says so and never passes; the production judge never stands in for it.

Business

Conversations reaching the declared outcome the rollout names, worse by its polarity. Without one it never passes. See Business outcomes.

Latency and time to first token

p95, failing when the candidate's interval lies above the current model's plus your margin.

Critical

Any candidate conversation failing a category your expert marked critical fails it. While a category marked critical has no published grader it never passes.

Errors

Errors and failovers per request.

Shadow requires cost, latency and errors; split requires every gate; passthrough and full have none.

Shadow cannot judge quality. A shadow answer is the candidate's one reply inside a conversation the current model carried on, so no conversation of the candidate's own exists for your published graders to rule, no customer heard it, and its call is not streamed. The defect rate, critical failures, business outcome and time to first token are first read in split; the page says so under shadow's gates.

A grader call that fails is retried a bounded number of times when the failure is transient (the provider unreachable, rate limited or erroring). A conversation whose call still fails has no verdict: compliance leaves it out and says how many it left out, and critical cannot pass while a critical category's grader call failed on one of the candidate's, since it could hide the failure that gate exists to catch.

#Advance and roll back

Owners and admins advance one stage at a time, or raise the split's share, only when every gate of the current stage passes. Serving customers (split, a higher share, full) also needs a candidate that qualifies under the recommendation rule: Rollout shows its standing, Qualifies, Not proven or Rejected, with the reason and the fix (rebuild the tests, or check the model in Models). An owner can still advance an unproven or rejected candidate by confirming an override with a reason; the history marks that change Advanced without proof. Past shadow every conversation the candidate serves must be gradable: while no grader is published, or a category marked critical has none, the rollout settings warn before it starts and advancing from shadow is refused with that reason. Rolling back to any earlier stage or share is always allowed. Each change needs a reason, starts the stage's window afresh and is kept in the rollout's history and the audit log.

Optional hard limits on the candidate's error rate and p95 latency are checked every 30 seconds in split and full: a breach with 95% confidence returns the rollout to passthrough on its own, records why, and sends a rollback alert.

#Limits

• One gateway request is one Chat Completions call: no Responses API, embeddings or other endpoints.
• Message parts other than text pass through, but that conversation is not logged; its request row says why.
• Nothing meters gateway usage per workspace or bills it yet.