A/B test calculator
Compare how often people gave an answer in two variants of a survey, and see whether the difference is real or chance.
Enter how many people answered each variant and how many of them gave the answer you compare, such as Yes or a score of 9 or 10.
The two rates differ by 9.2 points. A two-sided test calls a difference clear at 95% confidence (p below 0.05), and Usertick waits for at least 30 answers on each side before it gives a verdict.
Is the difference between two variants real?
The A/B test calculator compares how often people gave an answer in two variants, such as the share who said Yes or rated 4 or 5, and says whether the difference is clear or could be chance. Enter the answers on each side and how many of them gave the answer you compare.
It runs a two-sided test for two shares at 95% confidence and, like Usertick, gives a verdict only once each side has 30 answers. Everything runs in your browser, and nothing you type is sent anywhere.
- Test
- Two-sided test for two shares
- Clear at
- 95% confidence, p below 0.05
- Minimum
- 30 answers on each side
- Your data
- Stays in your browser
How the A/B test calculator decides
It works out how far apart the two shares are compared with how much they would vary by chance with that many answers. The p-value is the chance of seeing a gap at least this large if the two variants were really the same.
A p-value below 0.05 means the difference is clear at 95% confidence. Above it, the honest verdict is no clear difference yet.
z = (p₂ − p₁) ÷ √(p × (1 − p) × (1 ÷ n₁ + 1 ÷ n₂))
p₁ and p₂ are the two shares, p the share of both groups together, and n₁ and n₂ the answers on each side.
An A/B test example
Say a shorter version of a survey goes to half the visitors. 240 answer variant A and 72 of them say Yes; 250 answer variant B and 98 say Yes.
Variant B's 39.2% against variant A's 30% gives a p-value of about 0.032, below 0.05, so B is clearly higher. Enter the same numbers above to see the verdict.
- Variant A
- 72 of 240, 30.0%
- Variant B
- 98 of 250, 39.2%
- Difference
- 9.2 points
- p-value
- 0.032, clear
Run a fair survey A/B test
The test is only as good as the split behind it.
- Change one thing between the variants, such as the wording of one question
- Show both variants at the same time, to the same kind of visitor
- Decide the answer you compare before the test starts
- Wait for enough answers on each side, and stop when you planned to
Why you shouldn't stop at the first clear result
Checking the result every day and stopping the moment it turns clear makes a false winner more likely, because chance swings early on. Decide how many answers you need, using the sample size calculator, and read the verdict then.
A clear difference can still be small. Look at the size of the gap as well as the verdict before you change the survey for everyone.
Survey A/B tests in Usertick
On the Business plan, a survey can have variants: a version with different wording, questions or design shown to a share of visitors. Results compares each variant with everyone else, question by question.
Usertick tests at 95% confidence and waits for at least 30 answers on each side: Welch's t test for averages, a two-sided test for NPS and chi-square for distributions. It marks a variant Better or Worse only where a question has a better end, such as a rating.
Questions about this calculator
No. It compares two shares, such as the share who said Yes. Usertick's own variant comparison also tests averages, NPS and the spread of answers.
No. Decide the number of answers first and read the verdict then, since a result checked every day turns clear by chance more often.
With fewer answers a test is unreliable and a single answer moves a share a lot. Usertick's variant comparison waits for the same 30.
No. The calculation runs in your browser and nothing you type is sent.
Listen better, act faster, and build what truly matters.
Start free, see what your customers mean, and get each answer to the person who can act on it.
- 150 responses a month on Free
- Every channel and AI analysis
- No card needed