A/B Testing & Experimentation

The Problem

Most "tests" are not tests

Teams run A/B tests every day and reach wrong conclusions. Not because the data lies, but because the experiment was designed in a way that makes valid conclusions impossible. Bad testing is worse than no testing: it creates confident wrong decisions.

Invalid experiment

Changed two things at once (button colour + copy)
Stopped the test when results "looked good"
Ran it for 3 days over a holiday weekend
Didn't pre-calculate required sample size
Declared winner at p = 0.09 ("close enough")

The result might be directionally right. But you can't know, and acting on it as fact is dangerous.

Valid experiment

One variable changed, everything else identical
Duration decided in advance based on sample size
Ran across representative traffic (full week minimum)
Sample size calculated before launch (MDE defined)
Decision made at pre-agreed significance threshold

The result can be acted on with known confidence. You understand the probability of being wrong.

Foundations

Start with a hypothesis, not a button colour

Every experiment starts with an observation about user behaviour, a proposed change, and a predicted outcome. Without this structure, you're not experimenting. You're decorating.

The hypothesis format: "We believe that [change] will cause [outcome] for [user segment], because [reasoning]. We'll measure this with [metric]."

01

Observation: What behaviour prompted this?

Start with data, not ideas. Heatmaps show users ignoring the CTA. Session recordings show drop-off at a specific form field. Funnel analysis reveals 60% of users abandon checkout at the address step. The observation is your evidence that something is broken; it justifies the test.

02

Hypothesis: What change will you make and why?

Be specific about the mechanism. "We believe that adding a trust badge near the checkout button will increase conversion because users are hesitant to share payment details with an unfamiliar brand." The "because" clause forces you to think causally, not just correlationally.

03

Primary metric: What's your single success measure?

Pick one primary metric before the test runs. If you measure 20 metrics and declare victory on whichever one moved, you'll find a "winner" by pure chance every time. Secondary metrics can inform interpretation; they can't retroactively become the primary measure.

04

Minimum Detectable Effect: What size of change matters?

The MDE is the smallest improvement that would justify rolling out the change. If your conversion rate is 4% and you need a 20% relative lift (to 4.8%), you need ~9,500 users per variant to detect it at 95% confidence. Calculate this before the test; it determines how long you need to run.

Statistics

Statistical significance: what it actually means

Statistical significance is the most misunderstood concept in experimentation. "95% confidence" does not mean you're 95% sure the variant is better. It means: if there were actually no difference, you'd see results this extreme by random chance only 5% of the time.

p-value

The probability of seeing this result by chance. A p-value of 0.05 means there's a 5% chance of seeing this difference even if the variant has no real effect. The standard threshold is p < 0.05 for most product decisions. For high-stakes decisions (pricing, core checkout flows), use p < 0.01. p-value is not the probability that your hypothesis is true.

Statistical power

The probability of detecting a real effect. Power is the flip side of significance. Standard power is 80%, meaning if there truly is an effect of your MDE size, you'll detect it 80% of the time. Low-power tests produce too many false negatives: you declare "no winner" when there actually was one. Under-powered tests are as misleading as peeking early.

Sample size

Non-negotiable: calculate before you start. Sample size depends on three inputs: your baseline conversion rate, your MDE, and your target significance/power. Smaller effects require larger samples. A 2% relative improvement on a 50% conversion rate needs far more users than a 20% relative improvement on a 2% rate. Use an online calculator. Do not guess.

Confidence interval

The range where the true effect probably lives. A 95% confidence interval of [+1.2%, +4.8%] means the true effect is probably somewhere in that range. Wide CIs signal underpowered tests. Narrow CIs (more data) give precise estimates. A CI that includes zero means no significant effect, even if the point estimate looks positive.

Pitfalls

The mistakes that invalidate experiments

These are the errors that produce confident wrong conclusions, the most dangerous outcome in data-driven decision making.

Watch out for these

Peeking, stopping early when results look good: The most common mistake. Every time you look at live results and consider stopping, you're running multiple implicit tests, inflating your false positive rate. If you peek at p = 0.05 at day 3 and it's significant, there's a much higher real chance you're wrong than 5%. Pre-commit to a duration and don't look early.

Testing multiple variants without correction: Testing A vs B vs C vs D simultaneously multiplies your false positive rate. With 4 variants and p < 0.05, random chance alone produces a "winner" ~19% of the time across comparisons. Use Bonferroni correction or a Bayesian framework when testing multiple variants.

Novelty effect: New users engage more with anything new, especially returning users encountering a changed interface. A test that runs for 3 days may capture novelty-driven lifts that disappear after a week. Always include at least one full week of traffic and check for time-based trends within the test period.

Sample Ratio Mismatch (SRM): If you assign 50/50 but see 48/52 in actual traffic, your randomisation is broken. SRM invalidates any result regardless of significance. Check this before declaring a winner: chi-squared test between expected and actual traffic split. Always fix SRM before interpreting results.

Survivorship bias in segmentation: Finding "winners" in post-hoc subgroups (the test won for mobile users in Germany who signed up in the last 30 days!) is almost certainly a false positive. Pre-specify any subgroup analyses in your hypothesis. Post-hoc segment wins are hypotheses for the next test, not actionable conclusions.

Advanced Methods

Beyond standard A/B testing

Classic A/B testing has clear limits. When traffic is low, changes are complex, or you need faster iteration, these alternatives are worth understanding.

Multivariate Testing

Test multiple elements simultaneously (headline + image + CTA) to find the best combination. Requires much larger samples; traffic needs scale with the number of combinations. Best for high-traffic pages with multiple independent elements to optimise.

Bayesian A/B Testing

Instead of binary "significant/not significant," Bayesian methods give you a probability that variant B is better than A, updated continuously as data arrives. Better for teams that need to make decisions with imperfect data and can't wait for classical sample sizes.

Bandit Algorithms

Multi-armed bandit algorithms dynamically shift traffic toward the winning variant during the test, minimising the revenue lost by exposing users to an inferior experience. Useful for short-lived campaigns where waiting for classical significance is too slow.

Scope

When A/B testing is the wrong tool

A/B testing is powerful but not universal. Knowing when not to test is as important as knowing how to test.

Good use cases for A/B testing

CTA copy, button placement, form length
Onboarding flow variations
Email subject lines and send times
Pricing display and packaging
Homepage headline and hero image

High volume, measurable outcome, isolated change: classic A/B territory.

Poor use cases for A/B testing

Complete product redesigns (too many variables)
Changes with network effects (contamination between groups)
Low-traffic pages (years to reach significance)
Ethical or brand decisions (not everything is a metric)
Long-term behaviour changes (test duration too long)

Use qualitative research, staged rollouts, or expert judgement instead.

Takeaway

Experimentation as a discipline

The goal isn't to run more tests. It's to learn faster. Teams that build experimentation into their culture ship better products, make fewer costly mistakes, and compound learning over time.

Design before you build

Write the hypothesis, define the primary metric, and calculate the required sample size before writing a single line of code. Most bad experiments are designed badly, not executed badly.

Learn from losses

A well-designed test that fails to show an effect is still valuable; it rules out a hypothesis and refocuses effort. Teams with a high test velocity but low win rate are still learning faster than teams that barely experiment at all.

Statistical rigour is respect for decisions

Every shortcut in test design (peeking, underpowering, ignoring SRM) is a shortcut toward wrong conclusions. The downstream cost of a bad decision based on invalid data far exceeds the cost of running the test properly.

A/B Testing&Experimentation