Experiments

#Cohorts and metrics

Each cohort is an agent_version or one metadata key=value (a feature flag), over a window of start times; the two must be disjoint. The cohorts, the metrics (chosen issues or every issue not dismissed, chosen signals or every active one, chosen business outcomes) and the minimum sample of judged conversations per cohort (at least 10) are registered at creation and never change. Until both cohorts reach the sample, the result shows each cohort's counts and is collecting: no difference, p or verdict.

Metrics are the issue rate (judged conversations with an issue), each issue's occurrence rate, each signal's hit rate over the conversations that reached its situation, turns per conversation, each chosen outcome's rate and, for a money outcome, its amount per judged conversation. One of them is the primary metric, registered at creation: the judged-failure rate unless you pick another (turns, one of the chosen issues or signals, an outcome's rate such as conversion or refunds, or a money outcome's revenue per conversation). Every other metric is secondary.

#Statistics

Rates show a 95% Wilson interval; B − A has a Newcombe interval and a two-sided p (Fisher exact when an expected count is under 5, else the pooled two-proportion z-test). Turns show medians, the Hodges-Lehmann shift with its Moses interval and a Mann-Whitney p. An amount per conversation shows means, B − A with Welch's interval and p (normal approximation). For a good outcome higher is better; for failures, issues, signals and bad outcomes lower is. Verdicts read B against A at 0.05: better or worse, higher or lower, no difference, or not reached.

The primary metric's verdict reads its own p, unadjusted: it is the one question the experiment was registered to answer. The secondary metrics' p values are Holm-adjusted among the secondaries only, so adding issues and signals never dilutes the primary, and each secondary verdict reads its adjusted p. Intervals are 95% and unadjusted for every metric. Production health compares a release with the one before it with the same test as a primary failure rate (two-proportion p, better or worse at p < 0.05), so the same two releases over the same conversations get the same verdict on both pages; they differ only when the conversations differ (health counts the last 14 days, an experiment its cohorts' windows).

An experiment is observational unless both cohorts are values of one metadata flag over the same window that you declare randomized. When cohort B is a version first seen after an issue's fix was proven, that issue's row shows the simulation proof beside the production result. Every count opens the conversations behind it.