Introduction
In Webtrends Optimize, our ethos is easily replicable statistics. Our core calculations for most of the metrics you see are those which are used in popular calculators like https://cxl.com/ab-test-calculator/.
Why do we do this?
Many of our customers supplement analytics in our platform with their Web Analytics and a local or online calculator. Using proprietary calculations, no matter how useful, would be impossible to replicate yourselves, and so decision-making becomes far more challenging - "WTO is saying it's not significant but my calculator is saying it is - what do we do?".
We avoid all of this by using well-recognised, vetted calculations that are considered "industry standards".
Summary of statistics used in Webtrends Optimize
The below may help summarise the landscape of statistics for your RFIs.
Calculation / Statistical test | Used for |
2 tailed z-test | Binomial metrics, AB(n) tests and targets |
Mann-Whitney U-Test / Wilcoxon Rank Sum test | Non-binomial / continuous metrics, AB(n) tests and targets. |
Taguchi/Fisher Design of Experiments | MVT, experiment design |
ANOVA | MVT, Predicted optimal, Factor/level influence |
Types of metrics
Binomial Metrics
The important part of the word binomial is "bi", as in 2. Binomial data in Experimentation is therefore data that has 2 possible outcomes. For us, this is that an event did, or did not happen.
When considering our traditional goals - purchases, page loads, clicks, etc. - binomial calculations consider whether or not these happened. Statistics on this data consider the likelihood of these actions.
Non-binomial metrics
Unlike event-level data, where we're capturing a tally of how often things happen, non-binomial data is a numeric data collection.
Examples include Revenue, Units Per Transaction, etc. Such data is as useful as binomial data - knowing you've made more sales is great, but money earned is as important if not more.
Statistics for AB/ABn Tests and Targets
AB/ABn Tests and Targets have a single factor or controlled variable of change. You may have several variations as part of these tests, i.e. an AB/n as opposed to an AB test.
An AB test for button colour could test Blue vs. Red
An ABn test for button colour could test Blue vs. Red vs. Black
Either way, there's one controlled variable (colour).
Binomial calculations for AB/ABn tests and Targets
We employ 2-tailed Z-tests for these metrics and these experience types.
Read more here: Z-Tests in Webtrends Optimize.
Non binomial calcluations for AB/ABn tests and Targets
We employ Wincoxon Rank-Sum / Mann-Whitney U-Tests.
Read more here: Mann-Whitney U-Tests / Wilcoxon Rank Sum Test.
Statistics for Multi-Variate Tests
Multi-Variate Tests (MVTs) have multiple variables that we're observing. In Webtrends Optimize, we refer to the variables as Factors, and the variations plus control are collectively referred to as Levels.
Taking the previous example of a button colour AB or ABn test, an MVT would consider other Factors or types of change.
For example:
Factor 1: Button Colour
Factor 1 Level 1: Control (blue)
Factor 1 Level 2: Red
Factor 1 Level 3: Green
Factor 2: Button Size
Factor 2 Level 1: Control (medium)
Factor 2 Level 2: Bigger
Factor 2 Level 3: Biggest
Factor 3: Button Icon
Factor 3 Level 1: Control (no icon)
Factor 3 Level 2: Bag Icon
Factor 3 Level 3: Padlock Icon
The Test Array is what we refer to as the count of variations per factor. In the above example, the test array would be 3x3x3.
The calculations for MVTs differ strongly from what you'll see from a single-factor experiment.
Instead of making a 1 vs 1 comparison and looking for a single winner, MVTs look to find a combination of Factor-Levels that work the best together. As such, different calculations are required.
Design of experiments for Fractional Factorial MVTs
We employ the Taguchi/Fisher Design of Experiments for Multi-Variate Tests.
Read more here: Taguchi Fisher Method for Design of Experiments
ANOVA
ANOVA (ANalysis Of VAriants) is used to calculate the best performing experiments.