Why we use experiment quality as the main KPI for our experimentation platform
The most important aspect of product success is that it solves its core customer problem. In our case, this core problem is making reliable evidence-based product decisions using experimentation, both on the tactical and strategic level. What really matters to us is not how many product decisions are made, nor how fast decisions are made, but how good those decisions are.
While using experimentation as part of product development and decision making is common practice nowadays, it does not, by itself, guarantee that good decisions are made. Executing experiments correctly can be difficult, and the data obtained from an experiment is only as reliable as the execution of the experiment itself. Running bad experiments is just a very expensive and convoluted way to make unreliable decisions.
The three categories we are currently most interested in are Design, Execution or Shipping:
- The Design category checks for things which happen before the start of an experiment. For example, we check whether a power calculation was done, and whether the expected outcomes on decision-making metrics were pre-registered.
- The Execution category is mainly about the planned experiment duration and the adherence to that plan.
- Finally, Shipping validates that the decision is in line with the shipping criteria.