What is an A/B test?
An A/B test shows two versions of the same thing to comparable groups of people at the same time and measures which one does better. Teams use it to settle questions with evidence, such as which wording, layout or offer works best.
A/B testing in plain terms
In an A/B test, people are split at random into two groups. One group sees version A, usually what you have today, and the other sees version B, with one change. Because the groups are alike in every other way and see the versions at the same time, a clear difference in the result can be put down to the change.
A/B tests are common for web pages, emails and product features, and they work just as well on surveys. You can test the wording of a question, the length of a survey, the order of questions or the welcome text, and see which version gets more answers or clearer ones.
The method needs enough answers on each side and a significance test at the end. Without them, an A/B test is two numbers that differ by chance, and the winner is a coin toss.
- Compares
- Two versions shown at the same time to similar groups
- Also called
- Split test, or an A/B/n test with more than two versions
- Decided by
- One measure chosen before the test starts
- Needs
- Enough answers per version and a significance test
- Used for
- Question wording, survey length, layouts, offers and pages
How to run an A/B test
Keep each test to one change, so you know what caused the result. Write down the measure that will decide it, and the size of change you care about, before anyone sees a version.
- Pick one change, such as the wording of the first question
- Choose the measure that decides, such as the share who answer or the average score
- Work out the sample size per version before you start
- Split people at random, and keep each person on the same version
- Run both versions at the same time until each reaches its sample
- Check significance, then roll out the winner or keep the original
An A/B test example
Say a software company ends its onboarding survey with a call to action that invites people to join a beta. Version A says “Join the beta” and Version B says “Try new features first”. 300 people answer the call to action in each version.
In Version A, 45 people click, a click rate of 15%. In Version B, 66 click, which is 22%. The gap of 7 points gives a p-value of about 0.03, so it is significant at 95% confidence, and Version B can replace Version A.
- Version A
- 45 of 300 clicked, 15%
- Version B
- 66 of 300 clicked, 22%
- Gap
- 7 points
- p-value
- About 0.03, below 0.05
- Verdict at 95% confidence
- Version B wins
What to A/B test in a survey
Survey A/B tests answer questions about how you ask. Small changes in wording, length or order can move both the number of answers and the answers themselves.
When a wording test raises a score, check that the winning wording is also clear and neutral. A question that hints at the answer it wants raises scores and teaches you less.
- Question wording, such as a plain phrasing against a friendlier one
- Length, such as three questions against five
- The order of questions or of answer options
- Layout, such as one question at a time against all on one screen
- The welcome text, or the call to action at the end
How long should an A/B test run?
Run it until both versions reach the sample size you planned, and for at least a full week, so weekdays and weekends are both in it. Stopping the moment one side pulls ahead often crowns a false winner.
If you get few answers, test bigger changes. A large difference shows up with fewer answers than a small one, so a bold change gives a verdict sooner.
How Usertick runs A/B tests
On Business, a survey can have up to 8 variants beside the Base survey. Each variant shows to visitors who match its conditions, such as a country, a device, a page, a user trait or a share of visitors that stays the same for each visitor, and everyone else sees Base. A variant can change, hide, add or reorder questions and change the texts, the theme and the layout, while triggers, targeting and frequency stay shared.
After an AI rewrite of a question in the editor, Test as variant sends the new wording to a new variant shown to 50% of visitors, so you can test it before it replaces the original.
The Variants tab on Results compares each variant with a baseline at 95% confidence once each side has 30 answers, with a summary sentence, each side's main score, responses and median time, and a line per question. Better or Worse appear only when the difference is clear and the question has a better end, such as NPS or a rating.
Mistakes to avoid with A/B tests
Small choices that keep the results useful.
- 01
Changing several things at once
If Version B has new wording, a new order and a new color, a win can't be traced to any one of them. Test one change at a time.
- 02
Stopping at the first lead
Early leads often fade as more answers arrive. Wait until both sides reach the sample you planned.
- 03
Rewarding a leading wording
A question that hints at the answer it wants can raise a score without making it more accurate. Check each wording for bias before you test it.
- 04
Running versions at different times
Showing A one week and B the next mixes the change with everything else that moved that week. Run both versions side by side.
Frequently asked questions
Enough on each side to detect the size of change you care about: a large gap can show with a few hundred answers per version, and a small one can take thousands. Usertick gives a verdict only once each side has 30 answers.
An A/B test changes one thing between two versions. A multivariate test changes several things at once in every combination, so it needs far more people to say which change mattered.
No, but an even split reaches a verdict fastest for a given number of answers. An uneven split, such as 10% on the new version, limits the risk of a change while you test it.
Variants run on website and link surveys. In a survey email, the question in the email itself always comes from Base, and Usertick doesn't test email subject lines.
Listen better, act faster, and build what truly matters.
Start free, see what your customers mean, and get each answer to the person who can act on it.
- 150 responses a month on Free
- Every channel and AI analysis
- No card needed