Kill threshold
The metric value that ends an experiment early. You set it up front so you do not negotiate with yourself when the numbers come in.
A kill threshold is committed in the contract before the experiment runs. If the metric drops below it (for higher-better metrics) or above it (for lower-better metrics), the experiment ends and the verdict is kill.
The kill threshold is not a hope. It is a pre-commitment. Its job is to make the decision automatic when the data is bad, so the team does not spend a quarter arguing about whether to keep going.
When to use it
Set one in every experiment contract. Especially when the experiment costs spend, eats team focus, or creates customer-facing risk that compounds the longer the test runs.
What this looks like in practice
The kill threshold is a precommitment device. The job is not to predict where the metric will land; the job is to make the decision automatic when the metric is going the wrong way. Setting one before the experiment starts protects the team from the bias they will have once the data is in and emotions are attached.
A good kill threshold is anchored to a real cost. Paid-channel experiments anchor to CAC. Outbound experiments anchor to reply rate. Pricing experiments anchor to conversion. The threshold is the number below which continuing the experiment costs more than killing it.
A kill threshold should also have a "trip condition" — when does the threshold need to be sustained, vs. crossed once? A single bad day on a paid campaign is noise; three bad days is signal. Specifying this in the contract prevents the trip from being argued about during the experiment.
A worked example
For a paid-channel test where blended CAC is the metric, the kill threshold might be $210 against a control of $185. If CAC clears $210 mid-flight, the experiment is killed and the spend reverts.
Common mistakes
- Setting it where the metric usually sits anyway.A kill threshold that the metric naturally floats above does nothing. Anchor to a value that, if hit, would be a real loss.
- Negotiating with it mid-flight.The whole point of the threshold is that it is non-negotiable once data starts arriving. If you move it, you do not have a kill threshold.
- Forgetting noise.A single bad reading should not trip the threshold. Define what sustained means up front so the kill is principled, not panicked.
Related terms
Pick a hypothesis. Vocabulary done.
The fastest way to learn this vocabulary is to commit one experiment. The contract takes about five minutes to write.