Once we have set up and configured your account we like to run an AA test to validate everything is working as expected. This allows us to check all the conversions and goals are coming through and it validates any other custom work which has been done as part of the onboarding.
Please note - An AA test is the final part of onboarding. We only set an AA test live once you are comfortable everything is tested and checked. The AA test will not make any changes to your site/pages and will run for a limited time, we do not expect to run an AA test longer than a week unless necessary to do so.
What is an AA test and why run one?
An AA test is where we create a test which makes no changes to the page but still tracks user behavior. It works exactly the same as an AB test and will assign users to a control and a variation but in this example there are no differences. It allows you to see how the testing will work, validate everything is working and understand the output in terms of the reporting. Its the final step we take in the onboarding.
Setting up an AA test
Typically this is set up by whoever is configuring your account and is set live once we have checked with you that you are happy to do so. This may appear in your account and look something like this on your dashboard:
Initially this will be in 'Staging mode', you can find out more about this here. Essentially its in preview mode and is in this mode until it is pushed live once you are happy to do so.
If this has not been set up, its very easy to create one in your account.
Navigate to 'New test' from the dashboard:
Select 'Web page', then 'ABn', then 'Advanced editor':
This will take you into the page where you specify the specifics for the test. You do not need to understand this area of the product to set this up however if you want more information please see this article.
At this stage we need to fill in some of the fields:
Project name - this is the name of the project and can be something like 'AA test'.
Project location - this is where we want to run the AA test, you can select either a pre-existing location or create one from this screen. We suggest running AA tests sitewide however you can run them on a single page. Find out more about locations here.
Test name - this is the name we want to give the test, we can use the same name as the project here.
After this the screen may look something like this:
The last thing we need to do is click 'Variations' and on the next screen add a name for the variation. In this scenario we can just call it 'Variation 1':
We can now navigate through the final 3 screens (scripts/conversions/segments). On the final screen (segments) you have the option to 'Save'. When you select this the test and project will be created and you will see a confirmation screen:
Navigating back to the dashboard will bring you back to step 1 with your AA test showing on the dashboard in staging.
What happens next?
We will ask to get the AA test set live and we will let it run for typically one week. Throughout this we will check in to see if things look ok and you can also check everything from your side.
It is very common for AA tests to never reach significance and results can vary wildly, for more information on this please see this article.
Other than this we will stop the AA test, check with you that you are happy and then look to kick off some more real life testing.