Meta-Analysis
The Concept
As teams run a number of experiments, it is possible to glean learning across these experiments. This is meta-analysis. We're rolling out a suite of meta-analysis features this summer - feel free to reach out to help influence our roadmap in Slack.
Experiment Timeline View
This view lets you to filter down to experiments a team has run. At a glance you can answer questions like
- What experiments are running now?
- When are they expected to end?
- What % of experiments ship Control vs Test?
- What is the typical duration?
- Do experiments run for their planned duration - or much longer or shorter?
- Do experiments impact key business metrics - or only shallow or team level metrics?
- How much do they impact key business metrics?
Metric Impact (Batting Average)
The "batting average" view lets you look at how easy or hard a metric is to move. You can filter to a set of shipped experiments and see how many experiments moved a metric by 1% vs 10%. This quickly gives you a sense for whether betting your next experiment will move this by 15% is a good idea. Like with other meta-analysis views, you can filter down to a team, a tag or even if results were statistically significant.
Metric Correlation View
This view lets you visualize two metrics (each data point is an experiment) and visually inspect them for correlation.
Often the metric you want to move isn't very sensitive and takes a while to measure. It is helpful to find metrics that are more sensitive and faster to measure - and run experiments on this.
This view lets you plot two metrics on the same chart - each data point is an experiment's impact on them. You can quickly get a sense for whether the metrics tend to move together - or not. You can also remove outliers, filter down to a team's experiments or download the underlying dataset.
In this hypothetical example - "Checkouts" is the metric you want to move, but it's not very sensitive. "AddToCart" correlates well with "Checkouts", while "ViewItemDetail" doesn't.
Metric Insights
This view lets you pick a metric and see all experiments and feature rollouts that impact this metric. Learn more.
Knowledge Bank
(Coming soon) is a searchable view across all your experiments. Make it easy for new team mates to explore and find experiments a team ran, or where a topic was mentioned. In general the meta-analysis tools offer more structured and powerful means to discover and look across your experiment corpus, but when you do want free text search, this exists.