The reporting interface for MVTs offers a lot more detail than you'd find from simple AB or ABn tests. Instead of looking just at overall winners and probabilities per metric, we can study interaction effects and factor-level performance.
Performance
The performance tab offers a very similar output to what you'd see in an AB test, except that it is comparing combinations as opposed to individual variations.
Predicted optimal
Optimal performance and combination selector
The purpose of this section is to, by default, provide you with details of the best performing combination, along with clear details of each factor-level's contribution to that overall number.
The first section is interactive. It will start with what we suggest to be your best-performing combination, but you are allowed to then explore alternatives. Any changes made to the dropdowns (selected variant per factor) will update the numbers above which show the conversion rate from control to optimal with a projected uplift.
Factor Significance: This is how conclusive the observed changes have been for this particular factor.
Content level: Which variation we have selected to study for that given factor
Level effect: Conversion rate effect for that variation.
Conversion rate effect: Percentage point impact on conversion rate. E.g. 1% to 3% = 2pp impact.
Lift effect: Contribution to uplift. E.g. 5% + 0% + 2% would give you a total 7% uplift for that combination.
Factor Results
AB Tests would show results on a per variation basis, but the output is fairly slim. MVTs, on the other hand, show a similar output but as there are factors and many levels, there is a more detailed output.
Factor name: Category of change
Level name: Your variation
Optimal: Thumbs up if that variation is the best performing of it's factor.
Level effect with mean: Conversion rate with a bar graph to help comparisons across a given factor.
Conversion rate effect: Percentage point improvement of conversion rate. E.g. 18% to 20% = 2pp improvement.
Chance table
The chance table describes probabilities for the combinations "experiments" that we've run.
Chance to beat control: How much better than the control for all Factors is this combination?
Chance to beat all: What is the likelihood that this combination would outperform everything else we've seen?
Traffic
This contains the same information you'd expect to find in an normal AB test, and so won't be elaborated on in this article.
Stabilisation
This contains the same information you'd expect to find in an normal AB test, and so won't be elaborated on in this article.
The only difference is that it reports by combination instead of a single variation, given MVTs run combinations.
Factor/Level influence
Info coming soon
Experiment content overview
Info coming soon
Advanced
Info coming soon