Graders

#Failure categories

Group failures asks a model to propose categories from the fail critiques and their open codes, and to map each critique to them. You edit names, keys and definitions, mark the categories whose failure is critical (one candidate conversation failing one fails a rollout's critical gate; the model never sets it), remove or add categories, and approve them as the next criteria version. A version is never edited: new categories make a new version, and reviews mapped under an earlier version stay mapped to it until they are mapped again.

#Alignment

Each category's draft grader quotes up to 8 critiques from the reviews that build it that show the failure and 8 that do not. Run alignment grades every draft on the held-out conversations whose reviewers agree; measuring a grader on your own labels draws on no allowance. With a fail as the positive, each grader shows how often it catches your fails (TPR), passes your passes (TNR) and is right when it fails (precision), each with n and a 95% Wilson interval. Overall fails a conversation when any category fails it, against the review verdict.

Its panel lists where it disagrees with you: the missed fails and the false alarms, each with your critique beside the grader's reason, so you see what its prompt or code rules miss before you trust it.

A category with fewer than 60 labeled held-out conversations shows how many more labels it needs and cannot be published.

Code rules are checks in code over the agent's own turns: it says or never says a literal text, it calls or never calls a tool. A failed rule fails the category, and changing rules makes a new draft to align.

#Publishing

Owners and admins publish a draft once it is aligned: its prompt, examples, checks, model and effort are frozen under one sha256 as the category's next version, and that version grades every production conversation judged from then on. Its row then shows how many production conversations it failed of those it ruled, and how many have no verdict from it because its call still failed after its retries, with the last error in its panel.

#Outcome calibration

For the overall grader and each category, every declared outcome's rate after conversations the grader passed and failed, drawn as a before/after pair on one scale: the move is red when the outcome is significantly worse after a fail, amber when significantly better and gray while inconclusive, by the two-proportion p, with the difference's 95% interval beside it. Pick a metadata key to stratify by and the Cochran-Mantel-Haenszel test holds its values fixed and decides instead. These are associations, never causes: only a randomized rollout shows what a verdict causes.