In layman's terms, significance signifies the moment where it's reasonably unlikely that the change you're seeing in conversion rate is due to chance, and that the result can be relied upon.

As described in our document Z Tests in Webtrends Optimize, it's taking two distributions and measuring their overlap.

Note that non-binomial and MVT calculations vary somewhat.

Against any/every metric in the interface, you'll find a measure of significance:

If the results are negative, it might be red but you'll still see a similar output:

There are three core aspects that influence signifiance.

A 1000% improvement in conversion will reach significance almost immediately. Where you struggle with traffic, taking bigger swings will still allow you to see significant experiment results.

Smaller nudges in conversion rate require traffic to prove the effectiveness. As we get more traffic, the normal distribution curves "tighten" and so experiments are less likely to overlap.

Along with volume, confidence levels are another way to tighten normal distribution curves.

As significance is the result of these distrubutions not overlapping, trimming the edges of the curve gets us there faster/more often.