A/B Testing for Marketers: Run Tests You Can Trust

Most marketing opinions are guesses dressed up as confidence. A/B testing replaces the guess with evidence by showing two versions of something to similar people and comparing what happens. It sounds simple, yet many tests teach nothing because they were set up carelessly. This guide covers A/B testing for marketers who want reliable learning, not just a pile of experiments.

What an A/B test really is

In an A/B test you create two versions of one thing: version A is the current one, version B has a single deliberate change. Visitors are split between them at random, and you measure which version does better on one agreed outcome.

The key words are single change, random split and agreed outcome. Remove any of these and the result becomes unreliable. If you change the headline, the image and the button together, you will never know which change mattered. If you show version B only to weekend visitors, the day of the week might explain the difference. If you decide the success measure after seeing the results, you can always find something that looks like a win.

Testing is a way to reduce risk, not to prove you were right. A test that shows your idea did not help is still valuable, because it stopped you from rolling out something pointless.

Start with a hypothesis

Before touching any tool, write one sentence in this shape:

Because [observation], changing [element] to [new version] should improve [outcome].

Notice the observation. Good tests come from evidence, such as people dropping off at a certain step, a recurring customer question, or a page that gets traffic but few actions. Random ideas from a brainstorm tend to produce random results.

A hypothetical example: a small online tailoring service notices that many visitors view the pricing section but never open the enquiry form. The hypothesis might be that a short line explaining the process directly above the form will reassure people and raise form submissions. That is specific, testable and linked to an observation.

To find those observations, use your analytics, read customer messages, and watch a few real sessions or ask friends to try your page. The Google Analytics reports for small business guide helps with the first part.

Choose what to test

Not every element deserves a test. Prioritise ideas that sit close to the action and could plausibly change behaviour.

Good candidates include:

• Headlines and opening lines on landing pages.
• The wording, colour contrast and placement of a main button.
• The length and number of fields in a form.
• The offer itself, such as a free guide versus a discount.
• Email subject lines and preview text.
• Ad copy and images.

Weak candidates are tiny cosmetic tweaks on pages with very little traffic, because you will rarely collect enough data to see an effect. If your page has only a trickle of visitors, test bolder changes, or use other forms of research instead.

For page-level ideas, the landing page A/B testing guide goes deeper, and the ad creative testing guide covers paid campaigns. Email senders can start with email subject lines that get opens.

Pick one primary measure

Decide in advance which single outcome will decide the winner. It should be the closest meaningful action to the change: for a button test, clicks to the next step; for a form test, completed submissions; for an email subject line, opens, with clicks as a sanity check.

It is fine to watch secondary measures, but only to catch side effects. A subject line that raises opens while lowering clicks might be misleading readers. A shorter form might bring more submissions of lower quality. Look at both, but decide the winner using the primary measure you named beforehand.

Run the test fairly

Fairness is where most tests fall apart. Follow these rules:

1. Split traffic randomly. Use a testing tool or the built-in experiment features of your platform so people are assigned automatically.
2. Run both versions at the same time. Do not show A this week and B next week.
3. Keep everything else steady. Avoid launching a sale or changing ads in the middle of the test.
4. Decide the duration beforehand. Cover at least one or two full weekly cycles so weekday and weekend behaviour are both included.
5. Do not peek and stop early. Early leaders often fade. Stopping the moment one version looks ahead is a classic way to fool yourself.
6. Do not edit mid-test. If you fix a typo or tweak the design, you have started a new test.

Be honest about how much traffic you have. A small business with modest visits may need several weeks for a clear signal. If it would take months, the test is probably not worth running; try a larger change or gather feedback from customers directly.

Reading the results

When the test ends, resist the urge to declare victory at the first sign of a difference. Ask three questions.

Is the difference large enough to matter?

A tiny improvement that might be random noise should not trigger a redesign. Most testing tools offer a confidence indicator. Use it as a guide, understand roughly what it means, and remember that no test gives certainty.

Is it consistent?

Check whether the winning version also wins across devices, or at least does not lose badly on one of them. A version that wins on computers and loses on phones may need a different treatment.

Does it make business sense?

More clicks are not helpful if the people who click never buy. Where possible, follow the winning group through to the final outcome.

If there is no clear winner, that is a result too. It tells you the element you changed was not the barrier, and you can move on to a different hypothesis.

Keep a testing log

A spreadsheet with a row per test is one of the most valuable marketing assets you can build. Record:

• The date and the page, ad or email.
• The observation and hypothesis.
• What versions A and B were.
• The primary measure and how long it ran.
• The result, in plain words.
• What you learned and what you will do next.

Over time, the log reveals patterns, for example that your audience responds to clarity more than cleverness. It also prevents repeating failed ideas when team members change.

Common mistakes to avoid

• Testing too many things at once. One change per test, or use a more advanced multivariate method only when traffic is plentiful.
• Copying other brands blindly. What works for another audience may not work for yours.
• Ignoring seasonality. A test run during a festival may not reflect normal behaviour.
• Calling every test a failure or success. The goal is learning.
• Skipping the follow-up. Roll out the winner, then test the next idea.

If your analytics and goals are not yet reliable, fix that first. The conversion tracking setup guide explains how, and the broader topic of improving results is covered in what is conversion rate optimization.

A hypothetical test from start to finish

Imagine a small online course seller whose sign-up page attracts steady visitors but few registrations. Looking at the page, the owner notices that the main button says Submit, which tells visitors nothing about what happens next.

The hypothesis: because the button label is vague, changing it to describe the benefit should increase registrations. Version A keeps Submit. Version B says Reserve my free seat. Nothing else changes. The primary measure is completed registrations, and the test runs for three full weeks so weekdays and weekends are both covered.

At the end, the owner checks the result by device and notes whether registrants from each version attend the first session. Suppose version B is ahead but only slightly, and attendance is similar. The sensible conclusion is that the label helped a little, so it is adopted, and the next test targets the length of the form. Each test is small, but together they steadily remove friction.

Ethics and respect for users

Testing should never trick people. Avoid fake countdown timers, hidden costs or misleading claims in a variation, even temporarily. A variation that wins by deceiving visitors damages trust and may break advertising rules. Test clarity, relevance and helpfulness, and you will usually find that honest improvements win anyway.

Key takeaways

Good A/B testing for marketers comes down to discipline: start with an observation, change one thing, split traffic randomly, name your measure beforehand, run the test long enough, and record what you learned. A tie is not a waste; it narrows the search.

Pick one page or email this week, write a hypothesis, and set up your first test. If you want help planning a testing programme for your business, get in touch.