← Glossary
// THE CONTRACT

Experiment contract

The agreement, written before an experiment runs, that says what is being tested, how it will be measured, what kills it, and when it ends.

// what it is

An experiment contract is a four-part artifact: a hypothesis, a metric, a kill threshold paired with a success threshold, and an end date. Once the experiment starts, the contract does not change.

The contract is what stops drift. Without it, the team rewrites success criteria as the data arrives. With it, the rules decide the verdict, not the room.

// when this matters

When to use it

Write a contract before any experiment that costs money, time, or org attention to run. The act of writing it filters the experiments that should not exist in the first place.

// deeper

What this looks like in practice

A useful contract fits in four bullet points: the hypothesis (what change will move what number, by how much, in what window), the metric (one number, with a definition for how it is calculated), the thresholds (kill below X; ship above Y), and the end date (the calendar moment the verdict is forced). Five minutes is usually enough to write it.

The contract is "frozen" once the experiment starts. That sounds rigid, but the rigidity is the point — it is what prevents the human in the room from rewriting "success" the moment the metric goes the wrong way. If new information makes the contract obsolete, the experiment is killed and a new one is started, not patched.

The contract is also the unit the archive stores. A year later, you do not need to remember what was tested or why — you read the contract, the metric series, and the verdict. The next person making a similar decision starts from the same artifact.

// pitfalls

Common mistakes

  • Skipping the kill threshold.A contract without a kill threshold is a hope. Set the line where you will pull the plug before you start.
  • Rewriting the contract mid-flight.If you change the rules after data comes in, you are not running an experiment anymore — you are rationalizing a decision.
  • Hiding the contract.A contract no one else can find does not protect against drift. Save it where the next person making the same decision will look.
// related

Related terms

Marketing hypothesisA clear statement of what change you are making, what you expect to move, and the size of the move you would call a win, written before the experiment starts.Kill thresholdThe metric value that ends an experiment early. You set it up front so you do not negotiate with yourself when the numbers come in.Success thresholdThe metric value above which the experiment is shipped. Set up front, alongside the kill threshold, so the verdict is decided by the rules and not by the room.Hypothesis floorThe minimum measurable change you commit to before the experiment starts. The number below which a result is not interesting.VerdictThe outcome of an experiment after the end date or a threshold is hit. Three options: ship, kill, or review.

Pick a hypothesis. Vocabulary done.

The fastest way to learn this vocabulary is to commit one experiment. The contract takes about five minutes to write.

Run your first experimentBrowse the field guides