Hypothesis floor

The hypothesis floor is the smallest move on the metric that justifies the cost of running the experiment in the first place. It is a stricter version of the success threshold: instead of "the line above which we ship," it is "the line below which we did not learn anything new."

Setting a hypothesis floor up front prevents the most common marketing failure mode: shipping a 1.2% improvement that took six weeks of work and calling it a win.

When to use it

When the experiment costs real time or money to run. The floor is what stops you running it for a result you would not bother to ship.

What this looks like in practice

The hypothesis floor answers the question "is this experiment worth running at all?" If you cannot articulate the smallest win that would justify the work, the experiment probably is not the right next move. The exercise of setting the floor often filters out experiments before they consume real resources.

A useful hypothesis floor is calibrated to the cost of running the test plus the cost of maintaining the variant if it ships. A 0.5% lift on a CTA is not interesting if the variant requires ongoing engineering to keep alive. Set the floor at the value where the effort + maintenance + opportunity cost stays underwater.

The floor can sit below the success threshold (a value worth shipping) or above the kill threshold (a value catastrophic enough to abandon the variant), depending on the test. The point is not where it sits relative to the others; the point is that committing to it up front prevents post-hoc rationalization of meaningless results as "a small win."

Common mistakes

• Confusing it with the success threshold.The success threshold says "this is what we ship." The floor says "this is the smallest result worth even running for." They serve different decisions.
• Setting it implicitly.If the floor is in someone's head and not in the contract, the team will rationalize a 1.2% lift as a win. Write the number down.
• Picking it after the data arrives.A floor decided after results are visible is not a floor — it is the result you got, with a label.

Related terms