← Glossary
// FOUNDATIONS

Experiment OS

A system that captures every marketing decision as an experiment with a contract and a verdict, and stores the result so the next decision builds on the last.

// what it is

An experiment OS is the operating system around marketing experiments: the place where contracts are committed, metrics are tracked against thresholds, verdicts are decided, and the archive accumulates.

The category exists because most marketing teams already do experiments, but they do them as one-offs. An experiment OS makes the work compound: every test ends in a verdict, every verdict is saved, and the archive becomes the asset the team learns from.

// when this matters

When to use it

When a team has been running marketing experiments for more than a quarter and noticing the same questions get re-asked. The archive is what stops re-running last year is test.

// deeper

What this looks like in practice

An A/B testing tool optimizes a single surface — a landing page, a button, an email. An experiment OS works one layer up: it is the system that records what was tested across every surface, why, and what was decided. The unit it tracks is the contract, not the variant.

The "OS" framing matters because experimentation is process work, not feature work. The value is in the discipline the system enforces: every test starts with a written contract, every test ends with a verdict, every verdict is searchable. Without that, you have a folder of decks and a Slack thread.

You can build a usable experiment OS in a spreadsheet — many teams have. The point is not the tool, it is that the team treats marketing as something with a memory. Tools like Xi exist to remove the friction of keeping that memory honest under deadline pressure.

// pitfalls

Common mistakes

  • Confusing it with an A/B testing tool.A/B testing tools optimize a single surface. An experiment OS captures the contract behind any test, on any surface, and saves the verdict.
  • Skipping the archive.A system without persistent verdicts is a workflow tool, not an OS. The archive is what makes today is decision smarter than last quarter is.
  • Treating it as a tooling decision instead of a practice decision.No tool will save a team that does not commit to writing the contract. Pick the practice first; the tool is downstream.
Tool comparison. See how Xi compares with Statsig, VWO, Notion, and Spreadsheets.
// related

Related terms

Marketing experimentA marketing decision run as a test, with a hypothesis, a metric to move, a kill threshold that ends it early, and an end date that ends it on time.Experiment contractThe agreement, written before an experiment runs, that says what is being tested, how it will be measured, what kills it, and when it ends.Experiment archiveThe record of every experiment a team has run, with the contract, the metric movement, and the verdict. The compounding asset of an experiment-driven team.

Pick a hypothesis. Vocabulary done.

The fastest way to learn this vocabulary is to commit one experiment. The contract takes about five minutes to write.

Run your first experimentBrowse the field guides