A/A Tests Explained: Check Your Testing Setup First

An A/B test compares two different versions. An A/A test compares two identical ones. That sounds pointless until you realise what it protects you from: running months of experiments on a tool that quietly gives you false winners.

What an A/A test does

You split traffic evenly between two copies of the same page or email, then measure the same metric you would in a real test. Because nothing differs, the correct result is "no meaningful difference." If your tool declares a clear winner, something is wrong with the setup, or you have seen normal random variation.

Why bother

• Check the plumbing: confirms traffic is split evenly and tracking fires the same way for both groups.
• Learn natural noise: shows how much metrics bounce around when nothing changes, which helps you judge real results later.
• Build trust: a team that has seen the process behave sensibly is less likely to argue over borderline results.
• Catch bugs: flicker, uneven loading, bot traffic or a tag that fires only on one variant.

When to run one

An A/A test is most useful when:

1. You adopt a new testing tool or change how it is installed.
2. You move to a new site platform or tracking setup.
3. Your past tests produced results that later failed to hold up.
4. You are introducing experimentation to a team that is new to it.

If you only run an occasional email subject test, you do not need an A/A each time.

How to run it

1. Pick a page or email with enough traffic. Low traffic makes noisy results.
2. Create two identical variants and split traffic 50/50.
3. Choose the primary metric, such as sign-up rate, and decide the run time in advance.
4. Let it run for full weeks, so weekday and weekend behaviour are covered.
5. Do not stop early because one side looks ahead.
6. Review the split, the metric and the tool's significance reading.

How to read the result

• No clear difference: good. Your setup looks sound.
• Sample ratio mismatch: if one group received noticeably more visitors than the other, investigate redirects, caching or bot filtering.
• A "significant" winner: can happen by chance occasionally, which is itself a lesson about false positives. If it repeats across several A/A runs, suspect the tool or tracking.

A hypothetical example

An online course seller installs a new testing tool on a sign-up page. The A/A test splits visitors unevenly and shows more conversions for one copy. The team discovers the tool loads late on one variant, leading to tracking gaps. After the fix, a second A/A shows no difference, and they start real experiments with more confidence.

Related reading

Once the setup is trustworthy, follow the process in our A/B testing guide for marketers and the landing page A/B testing guide. For broader causal questions, see incrementality and testing.