What is statistical significance?
Statistical significance means a difference between two results is too large to be explained by chance alone. At the usual 95% confidence level, a result is significant when its p-value is below 0.05.
Statistical significance in plain terms
Whenever you compare two groups of answers, such as two versions of a question or this month against last, some difference shows up by chance. Statistical significance tests whether the difference you see is larger than chance would usually produce. If it is, you can treat the gap as real and act on it.
The test gives a p-value: the probability of seeing a difference at least this large if the two groups were truly the same. Teams usually set the line at 0.05, which matches 95% confidence. Below the line, the result is significant. Above it, the test can't tell a real gap from noise, which is different from proving there is no gap.
Significance comes up in A/B tests, in comparisons between countries, devices or plans, and whenever a score moves and someone asks whether the move is real. It depends heavily on sample size: the same gap can be significant with 500 answers per side and invisible with 50.
- Tells you
- Whether a difference is likely real or chance
- Usual threshold
- A p-value below 0.05, which is 95% confidence
- Depends on
- The size of the gap, the number of answers and how spread out they are
- Common tests
- t test for averages, z test for shares, chi-square for distributions
- Doesn't tell you
- Whether the difference is big enough to matter
How statistical significance works
Start from the assumption that the two groups are the same, called the null hypothesis. The test asks how surprising your data would be under that assumption. A small p-value means the data would be rare if the groups were the same, so you reject the assumption.
For two shares, such as the share who said yes in each version, the test turns the gap into a z value. At 95% confidence, a z above 1.96 or below −1.96 is significant. The 95% level accepts a 5% chance of calling a chance difference real, so about one test in twenty can mislead you even when nothing changed.
z = (p₂ − p₁) ÷ √(p × (1 − p) × (1 ÷ n₁ + 1 ÷ n₂))
Significant at 95% when z > 1.96 or z < −1.96
p₁ and p₂ are the shares on each side, n₁ and n₂ the answers on each side, and p the share across both sides together.
A statistical significance example
Say you test two wordings of a yes or no question. 72 of 240 people say yes to Version A, which is 30%, and 98 of 250 say yes to Version B, which is 39.2%. Across both sides, 170 of 490 said yes, about 34.7%.
The test gives a z of about 2.14 and a p-value of about 0.03. That is below 0.05, so the gap of 9.2 points is significant at 95% confidence. With 100 answers on each side and shares of 30% and 39%, the p-value would be about 0.18, and a gap of almost the same size would not be significant.
- Version A
- 72 of 240 said yes, 30%
- Version B
- 98 of 250 said yes, 39.2%
- Gap
- 9.2 points
- z and p-value
- z ≈ 2.14, p ≈ 0.03
- At 95% confidence
- Significant, since 0.03 < 0.05
Statistical significance vs practical significance
A significant result says the gap is likely real. It says nothing about whether the gap is worth acting on. With tens of thousands of answers, a change of 0.1 in an average rating can be significant and still too small to change any decision.
Look at both: the p-value for whether the gap is real, and the size of the gap for whether it matters. Decide before the test how large a change would make you act.
| Statistical significance | Practical significance | |
|---|---|---|
| Asks | Is the gap likely real? | Is the gap big enough to matter? |
| Measured by | p-value and confidence level | The size of the gap in points or percent |
| Grows with | More answers | Only a bigger real effect |
| Decided by | The test | Your team, before the test |
Which significance test to use
The right test depends on the kind of answer, since each test compares the two sides in a way that fits the data.
- Averages, such as a 1 to 5 rating: a t test, in Welch's version when the two sides spread differently
- Shares, such as the share who said yes: a two-proportion z test
- Answers spread across several choices: a chi-square test
- Several options tested at once: a correction such as Bonferroni, so one lucky option doesn't look like a winner
How many answers do you need for significance?
Small differences need large samples. To have a good chance of detecting a 10-point gap between two shares near 50% at 95% confidence, plan for about 400 answers per side. To detect a 3-point gap, plan for several thousand.
When answers are scarce, test bigger changes. A large difference shows up clearly with fewer answers than a small one.
How Usertick tests for statistical significance
Survey variants, on Business, show different versions of a survey to different visitors. The Variants tab on Results compares each variant with a baseline, Everyone else by default, and reading the comparison works on every plan.
Usertick runs the statistics at 95% confidence once each side has at least 30 answers: Welch's t test for averages, a two-sided test on NPS, chi-square for distributions, and per-option tests with a Bonferroni correction for choices. Better or Worse appear only when the difference is clear and the question has a better end, such as NPS or a rating. Other questions read Answers differ.
Each question gets its own line, and you can filter the lines by Differ, Collecting, No clear difference and Can't compare to see where the versions part. The AI reading also reports a clear difference between variants when it writes up a period.
Mistakes to avoid with statistical significance
Small choices that keep the results useful.
- 01
Peeking and stopping early
Checking daily and stopping at the first significant result raises the chance of a false win. Set the sample size first and read the result once you reach it.
- 02
Reading no difference as proof
A result that isn't significant means the test couldn't tell. With more answers, a real gap may still appear.
- 03
Testing many things at once
Compare twenty options and one will likely look significant by chance. Correct for the number of comparisons, or test fewer things.
- 04
Ignoring the size of the gap
A significant gap of half a point may not change any decision. Decide in advance how big a change is worth acting on.
Frequently asked questions
It means that if the two groups were truly the same, a difference at least this large would show up less than 5% of the time. It is the usual line for calling a result significant at 95% confidence.
Not exactly. It means the method calls a chance difference real only 5% of the time. How likely a given result is to be right also depends on how likely the change was to begin with.
Yes, when the gap is large. A small sample can show a big difference clearly, and small differences need many answers on each side.
With fewer answers, a handful of people can swing an average or a share a long way. Usertick gives a verdict on variants only once both sides have 30 answers, so a few early answers can't decide a test.
Listen better, act faster, and build what truly matters.
Start free, see what your customers mean, and get each answer to the person who can act on it.
- 150 responses a month on Free
- Every channel and AI analysis
- No card needed