Signals

#Polarity, presets and groups

Each signal is positive (a win: resolved without a handoff, a payment completed, the customer confirmed), negative or other, and can be filed into your own groups. Every agent has presets and one judged signal per instruction clause and per risk surface; the Grader lists the presets and your own first, and the clause and surface ones apart.

The presets: stalled or looped, unsupported action claims, frustration, handoff not done, capability gaps, refusals, handoff requests, sycophancy, internal leakage, malformed structured output, prompt-injection attempts, churn intent, resolved without handoff, customer confirmed success, repeated and un-retried tool-call failures, and one per self-report kind.

Add up to 10 custom signals per agent from one plain-language box; Duet Machines reads it into a situation, a behavior, a subject and a polarity, or says why it cannot be read from a transcript. It is previewed before it is saved: Duet Machines rules it on up to 20 recent judged conversations, one judged conversation of the month's allowance each, and you mark each hit Match or Not match (arrows, M, N); those labels and later ones on its hits are its validation set. Editing it makes a new version with its own rates and accuracy, and an earlier version can be rolled back to.

#Sources

Judged

Ruled by the judge model on each judged production conversation, with the situation and the behavior quoted and checked verbatim.

Self-report

The agent's own report_trouble calls of one kind, counted in code.

Tool status

Failed or blocked tool calls, counted in code: repeated failures of one tool, or a failure never retried.

Instrumented

Events your code sends: conversation.signal("payment_completed", 49.5) in either SDK, or "signals": [{"name": "payment_completed", "value": 49.5, "turn": 3}] on ingest (not over OpenTelemetry).

#Rates, precision and issues

Every number on the Grader follows its time range (24 hours to 30 days, stepping back a range at a time): events, users affected out of the active users and rate, each rate beside its precision against your team's labels with n and a 95% interval ("too few labels" under 5). Up to 4 signals can be pinned as trend cards.

A clause or surface signal names the issues proven on its clause. Without one, its hits are a rate no issue proves, labeled "rate only", and its Hits tab is where a reviewer accepts or dismisses them. A negative hit whose turn a proven finding quotes links to that issue.

#Backfill

A new signal is Building until it rules a conversation. Backfill it over the last 14 days of judged conversations after seeing how many it would rule and the judged conversations it reserves from the month's allowance: one per conversation for a judged signal, none for signals counted in code. It rules exactly those, and is refused when the month has fewer left.

A signal can alert through the workspace's channels on a spike (a day's hits at least 5 and 3 times the daily mean of the 7 days before it) or a rate threshold over a minimum of reached conversations. Both are ruled per UTC day of conversations by when they started, wait their cooldown (1 to 30 days) between alerts, and carry a prompt for a coding agent. The preview shows the days it would have fired over the last 14 exactly as the live check rules them; the live check rules yesterday and today after each judging. See Alerts.

#Production health

An agent's Traffic, under Health, shows how it does in production on the same definitions as its Overview: conversations per UTC day for the last 14 days, each day's bar split into failed a grader, passed and not graded, the grader pass rate (conversations that passed every published grader over those every one ruled, with a 95% interval) per release with the graded conversations it rests on, with each release marked on the day it was first seen and a release named when it fails significantly more or less than the one before it (two-sided p < ${LIMITS.significance}), the trouble the agent reported itself, each declared outcome's rate per release (the release's conversations with an event of it, over all of them), and the cost of failure (the spend on conversations a grader failed). Where issues stand (open, reproduced, fix proven, verified, regressed) and the latest fix proof of the live configuration head its Review, under Issues. Every count names its window, and every number links to the conversations or issues behind it.