5.3A/B testing & optimization
The essentials in 30 seconds
- Test rather than guess: compare two versions across equivalent groups to keep the best performer — a scientific approach.
- One variable at a time: a single element changes between A and B, so you know what made the difference.
- Enough volume: wait for enough data (significance) before concluding — otherwise it is chance.
Threaded mini-case — Sofia tests instead of guessing
Sofia hesitates over her newsletter subject line: her partner prefers one version, she another. No more opinion debates: this section tells how she states a hypothesis ("the first name in the subject increases opens"), sends two subjects differing ONLY in that variable to two equivalent halves, waits for enough volume and adopts the winner — then chains the next test.
Continuing the 5.1-5.2 case — Sofia optimises what she measures, and continues in 5.4 and 5.5.
1. Test rather than guess
Rather than guessing what works best, test it. A/B testing compares two versions of an element (email subject, page headline, ad visual, button) by showing them to equivalent groups, to keep the best performer.
It is the scientific method applied to marketing: deciding by data, not opinion. This is how Sofia settles the debate over her email subject.
2. One variable at a time
The key principle: test one variable at a time to know what made the difference, and wait for enough volume for a reliable result (significance: enough data to conclude without error).
Two subjects differing ONLY in the first name, sent to 2 halves: if Sofia also changed the time and the visual, the effect could not be isolated.
3. Continuous optimization
Continuous optimization — small test after small test — compounds gains that, added up, transform performance over time. Adopt the winner, note the learning, launch the next test.
Reflex: test high-traffic elements first — an email subject sent to thousands of people, a highly visited page headline: where a gain shows fast.
🔎 Concrete example — Isolating the variable
A single thing changes.
Protocol: hypothesis “the first name in the subject increases opens”; two subjects differing ONLY in that variable, sent to 2 halves — wait for enough volume before concluding.
Lesson: one isolated variable + reliable volume = data-based decisions, not intuition.
Section vocabulary
- A/B testing
- Comparing two versions with equivalent audiences, keeping the best.
Simple example: Sofia compares two email subjects on two halves of her list.
What it is not: copying a competitor or guessing "by feel".
Reflex: was my last design decision tested?
- Tested variable
- The single element changed between A and B to isolate its effect.
Simple example: Only the first name in the subject changes between Sofia's two versions.
What it is not: changing everything at once then randomly attributing the gain.
Reflex: does a single thing change in my test?
- Significance
- Enough data to conclude without error.
Simple example: Sofia waits for hundreds of opens before deciding.
What it is not: declaring a winner after 20 sends.
Reflex: does my volume allow a conclusion?
- Continuous optimization
- Chained small tests whose gains compound.
Simple example: Subject, then button, then visual: Sofia adds up the gains.
What it is not: one big test a year then nothing.
Reflex: what is my next small test?
- Hypothesis
- A clear assumption the test validates or invalidates.
Simple example: "A more visible CTA will increase clicks" before testing.
What it is not: testing with no hypothesis and reading the result afterwards.
Reflex: did I write my hypothesis before launching?
In practice — Running a simple A/B test
- State a clear hypothesis (e.g. “an email subject with the first name will increase the open rate”).
- Create two versions differing ONLY in the tested variable, sent to two equivalent halves.
- Wait for enough volume, then compare: clear and reliable difference, or marginal?
- Adopt the winning version, note the learning, and launch the next test on another element.
Ready-to-say script — proposing a test instead of a debate (1 min)
"We have two views on this subject — let us settle it with a test: written hypothesis, two versions differing only in that variable, sent to the two halves, and we wait for enough volume before concluding. The winner goes live, and we note what we learned. Agreed?"
Say in a team; write the hypothesis before launching, never after.
Copy-paste template — A/B test sheet
HYPOTHESIS: _______________
TESTED VARIABLE (just one): _______________
VERSION A / VERSION B: _______________ / _______________
AUDIENCES (2 equivalent halves): _______________
EXPECTED VOLUME BEFORE CONCLUSION: _______________
WINNER + LEARNING: _______________
NEXT TEST: _______________Low traffic
Few visits: favour bold changes (visible effect) and clear gaps — micro-variations need volume.
Regular emailing
Sofia's newsletter is ideal ground: each send tests a subject on thousands of contacts.
Flagship landing page
Test the most visited page headline: with high traffic, a small rate gain is worth many conversions.
Ads
Two visuals, one changing at a time: keep the winning creative then test the copy — never the reverse together.
✓ Best practices
- State a clear hypothesis before testing.
- Change only one variable at a time.
- Wait for enough volume before concluding.
✗ Common pitfalls
- Changing several variables at once (effect can't be isolated).
- Concluding on too little data (chance).
- Guessing instead of testing.
Key takeaways
- Test rather than guess: A/B testing decides by data.
- Change ONLY one variable at a time to isolate what causes the effect.
- Wait for enough volume for a reliable result.
- Continuous optimization compounds small gains into a big difference.
48h challenge — apply tomorrow
Launch 1 real A/B test with the sheet above: written hypothesis, a single variable, two equivalent halves. Note the volume from which you will accept a conclusion — and hold firm until then.
- I explain the A/B principle in one sentence.
- I know why one variable and enough volume matter.
- My first test is launched with a written hypothesis.
Frequently asked questions
What can you test with A/B testing?
Almost anything: email subjects and content, landing page headlines and visuals, ad copy and creatives, button colour and label, forms… Start with high-impact, high-traffic elements (where a gain shows fast), like an email subject sent to thousands of people or a highly visited page headline.
Do you need lots of traffic to test?
A minimum is needed for a reliable result (otherwise you conclude on chance). On low volumes, tests take longer and need clear gaps to decide. Focus on your highest-traffic channels, and favour bold changes (not micro-variations) whose effect will be more visible.
Sofia changes her page headline, visual AND button, and sales rise. What is this test worth?