What is FWER (Family-Wise Error Rate)?
When you run an experiment, you’re usually asking a simple question:
“Did this change really cause an improvement, or did it just look that way by chance?”
Statistical significance helps answer that, but comes with an important caveat: Multiple comparisons. For experimentation in particular, where we see:
Multiple variants tested at once
Multiple metrics tracked for assessing performance
Post-test segmentation
Repeated analysis
etc.
If we assume each individual test might have a 5% false-positive risk, these risks are believed to compound as the number of comparisons made grows.
What are the corrections?
Briefly - corrections are adjustments made to the alpha value that is used to calculate Statistical Significance.
Across various strategies, and so in different ways, we adjust the alpha value to compensate for making multiple comparisons.
How to enable FWER corrections
In the Settings area of your reporting screen, you'll find these settings allowing you to pick a control method:
You also have the ability to pick an account-wide default strategy:
Available Strategies
Bonferroni correction
The Bonferroni correction applies an equal correction across all metrics that you're analysing.
α = α / number of metrics
So, if your Confidence Level is 95% and there are 5 metrics, we get:
α = 0.05 / 5
α = 0.01
This is the equivalent of running your experiment at 99% Confidence Level.
Holm-Bonferroni correction
This is a staggered strategy where lower-ranked metrics get a progressively more stringent alpha value. This is calculated as:
α = α / rᵢ
So, if your Confidence Level is 95% and there are 5 metrics, we get:
For metric 1: α = 0.05 / 1 = 0.05
For metric 2: α = 0.05 / 2 = 0.025
For metric N: α = 0.05 / N
For metric 5: α = 0.05 / 5 = 0.01
Hochberg correction
This is an inversed approach to Holm-Bonferroni, in which your key metrics must meet more stringent requirements to be Statistically Significant than lower-ranked metrics. This is calculated as:
α = ( N - rᵢ + 1)
So, if your Confidence Level is 95% and there are 10 metrics, we get:
For metric 1: α = 0.05 / 10-1+1 = 0.05/10 = 0.005
For metric 2: α = 0.05 / 10-2+1 = 0.05/9 = 0.0056
For metric N: α = 0.05 / 10-n+1
For metric 10: α = 0.05 / 10-10+1 = 0.05/1 = 0.05