← Glossary
// THE CONTRACT

Success threshold

The metric value above which the experiment is shipped. Set up front, alongside the kill threshold, so the verdict is decided by the rules and not by the room.

// what it is

A success threshold is the inverse of a kill threshold. It is the value the metric must hit (or beat, depending on direction) by the end date for the verdict to be ship.

Setting both thresholds in the contract is what makes the verdict automatic. Hit the success threshold, ship. Hit the kill threshold, kill. Land between them, review with whatever you learned.

// when this matters

When to use it

Set one in every experiment contract, paired with the kill threshold. Together they create a decision band: above ship, below kill, between review.

// deeper

What this looks like in practice

Setting a success threshold forces a real conversation up front: what would actually be a win, and how confident are we that the change can deliver it? If the team cannot agree on a number, the experiment probably is not worth running yet — the disagreement is about strategy, not data.

The success threshold should be set where the result is meaningful enough to ship and live with. A 0.3% lift might be statistically real; if it is not enough to justify the maintenance cost of the new variant, ship is the wrong verdict regardless. Anchor the threshold to operational impact, not statistical existence.

Pairing the success threshold with the kill threshold creates a "decision band": land above the line, ship; land below the kill line, kill; land between, review. The review band is healthy — it is where the team learns. Do not collapse it by setting the two thresholds too close together.

// pitfalls

Common mistakes

  • Setting it at "any positive movement."A 0.5% lift can be statistically real and operationally worthless. Set the threshold at a value that justifies the cost of shipping and maintaining the change.
  • Collapsing the review band.If success and kill thresholds are right next to each other, every experiment forces ship-or-kill with no room for "we learned something." Leave a deliberate review band.
  • Treating the threshold as a forecast.Success thresholds are commitments, not predictions. Pick the line you would defend, not the number you expect to land on.
// related

Related terms

Kill thresholdThe metric value that ends an experiment early. You set it up front so you do not negotiate with yourself when the numbers come in.Experiment contractThe agreement, written before an experiment runs, that says what is being tested, how it will be measured, what kills it, and when it ends.VerdictThe outcome of an experiment after the end date or a threshold is hit. Three options: ship, kill, or review.

Pick a hypothesis. Vocabulary done.

The fastest way to learn this vocabulary is to commit one experiment. The contract takes about five minutes to write.

Run your first experimentBrowse the field guides