Statistical Significance for Marketers, in Plain English

You change a subject line, and open rate goes up. Did the new line work, or did you just get a lucky batch of readers? Statistical significance is the framework for answering that. You do not need to be a mathematician to use it well.

The core idea

Results from any sample vary by chance. If you flip a coin ten times, you will not always see five heads. Significance testing asks: if there were truly no difference between version A and version B, how likely is it that we would see a gap this large by luck alone?

That probability is the p-value. A small p-value means the result would be unusual under "no difference," so you have evidence of a real effect. Many teams use a threshold such as 0.05, but it is a convention, not a law.

Key terms

• Sample size: how many visitors or recipients are in the test. More data reduces noise.
• Confidence level: how sure you want to be before declaring a winner. A higher level demands stronger evidence.
• Effect size: the actual difference between versions. Tiny differences need large samples to detect.
• Minimum detectable effect: the smallest improvement worth caring about, decided before the test.
• False positive: declaring a winner when nothing changed.

Rules that protect you

1. Decide the metric and the stopping point in advance. Do not hunt for a metric that happened to win.
2. Do not peek and stop early. Checking daily and stopping at the first good-looking result inflates false positives.
3. Run full weekly cycles. Behaviour differs on weekdays and weekends.
4. Calculate sample size first. Free calculators let you enter baseline conversion rate and desired lift.
5. Avoid running too many variants. Each extra comparison adds chances of a fluke.
6. Consider practical value. A statistically significant 0.1% lift may not be worth the effort.

A hypothetical example

A coaching institute tests two landing page headlines. After two days, version B has a higher sign-up rate, and the owner wants to switch. But only a small number of visitors have seen each version. The team keeps the test running for the planned period, and the gap shrinks to something that could easily be chance. They keep version A and try a bigger idea instead.

When you have little traffic

Small sites often cannot reach significance on subtle changes. Options:

• Test bolder changes that could create larger effects.
• Use higher-volume metrics, such as clicks, as a secondary signal.
• Combine qualitative evidence from heatmaps and session recordings or customer feedback.
• Accept directional evidence for low-risk decisions, and be honest about uncertainty in reports.

Putting it into your workflow

Follow a structured process like the one in our A/B testing guide, verify the tool first using an A/A test, and document results whether they win or lose.