What is sample size?
Sample size is the number of people whose answers you collect and analyze in a survey. The bigger the sample, the smaller the margin of error, so the results say more precisely what your whole audience thinks.
Sample size in plain terms
When you survey some of your customers instead of all of them, the people who answer are your sample, and the count of their answers is the sample size. Their answers describe the whole group, called the population, only up to a point. A larger sample brings the result closer to the truth, because chance has less room to tilt the numbers.
Sample size decides how far you can trust a result. With 40 answers, a satisfaction score can move several points from one week to the next for no reason at all. With 400, the same move is far more likely to mean that something changed. The number you need depends on how precise you want to be, how sure you want to be and how large the group you study is.
Sample size comes up whenever you plan a survey, compare two groups or run an A/B test. It works together with the margin of error, the confidence level and the response rate, which tells you how many people to invite to reach the answers you need.
- Measures
- How many answers a result rests on
- Depends on
- Margin of error, confidence level, population size and the expected share
- Common target
- 385 answers for ±5% at 95% confidence from a large audience
- Used for
- Planning surveys, comparing groups and A/B tests
How to calculate sample size
Pick the margin of error you can accept, such as ±5%, and a confidence level, usually 95%. The confidence level sets the z value: 1.645 for 90%, 1.96 for 95% and 2.576 for 99%.
The share p is the result you expect, such as the share of customers who will answer yes. When you don't know it, use 50%, which gives the largest and so the safest sample. At 95% confidence and ±5%, the first line gives 384.16, which you round up to 385. When your whole audience is small, the second line lowers the number.
n = z² × p × (1 − p) ÷ e²
Small population: n ÷ (1 + (n − 1) ÷ N)
z is 1.96 at 95% confidence, p is the expected share (0.5 when unknown), e is the margin of error as a decimal and N is the population size. Always round up.
A sample size example
Say your software product has 2,000 active customers, and you want to know what share of them would use a new feature, within ±5% at 95% confidence. The basic formula gives 384.16. Because the population is small, the second line brings it down to about 322.4, so you need 323 answers.
Then plan the invitations. If about 20% of the customers you email usually answer, invite 323 ÷ 0.2, which is 1,615 customers. With a population of 2,000, that means asking most of your customer base.
- Population
- 2,000 customers
- Margin of error and confidence
- ±5% at 95%
- Basic sample size
- 384.16
- Adjusted for 2,000 customers
- 322.4, rounded up to 323
- Expected response rate
- 20%
- Customers to invite
- 323 ÷ 0.2 = 1,615
How many survey responses do you need?
No single number fits every survey, because it depends on the decision the answers support. A first look at which topics come up most can work with a few dozen written answers. A score you report to leadership, or a difference between two groups, needs hundreds.
Precision grows with the square root of the sample. Going from 100 to 400 answers halves the margin of error, and halving it again takes 1,600. The table shows the margin at 95% confidence for a share near 50%, the widest case.
| Answers | Margin of error at 95% |
|---|---|
| 100 | ±9.8% |
| 400 | ±4.9% |
| 1,000 | ±3.1% |
| 2,500 | ±2.0% |
Sample size for comparing groups
Every group you compare needs enough answers on its own. To compare mobile and desktop customers, or two versions of a question, plan the sample for each side. A survey with 400 answers spread over eight countries has about 50 per country, which leaves a wide margin on each.
- Decide which groups you will compare before the survey starts
- Plan the sample per group, then add the groups together
- Expect small differences to need large samples on each side
- Work out the number before an A/B test, and run it until both sides reach it
Does a bigger sample fix bias?
No. A bigger sample shrinks random error only. If the people who answer differ from the people who don't, more answers repeat the same error with more confidence. A survey that only your happiest customers see gives a precise picture of your happiest customers.
Ask the whole group you want to describe, at a neutral moment, with neutral wording. Then a sample of a few hundred can describe thousands of customers well.
How Usertick handles sample size
A survey's Summary on Results shows its responses next to the completion rate and the median completion time, and every question card shows its answer count. A score that rests on fewer than 30 answers says how many it has, so you can see when a number rests on few answers.
Usertick holds back where the sample is thin. The variant comparison names a difference only once each side has 30 answers, at 95% confidence. The Audience tab shows a group's score from 5 answers and flags a group as clearly lower only from 20 answers. The AI reading writes up a period once it has 30 responses.
To stop at a planned sample, set a response limit that closes the survey after a set number of responses. A website survey can show to a share of your traffic, on 1 survey on Free and on every survey from Starter, and the sample size calculator gives the number to aim for before you start.
Mistakes to avoid with sample size
Small choices that keep the results useful.
- 01
Splitting a small sample into groups
400 answers feel like plenty until you split them by country or plan. Check the number of answers in each group before you compare them.
- 02
Forgetting the response rate
The formula gives the number of answers you need. Divide it by your expected response rate to find how many people to invite.
- 03
Stopping as soon as results look good
Checking every day and stopping at the first clear gap makes chance look like a finding. Decide the sample size first and wait until you reach it.
- 04
Trusting size over who answered
A large sample of only your most engaged customers still misleads. Ask the whole group you want to describe.
Frequently asked questions
For a rough read, often yes: 100 answers give a margin of error of about ±10% at 95% confidence on a share near 50%. For a precise score, or to compare groups, aim for several hundred.
Only when the population is small. For ±5% at 95% confidence, 10,000 people need about 370 answers and 10 million need 385, so beyond a few thousand people the size of the audience barely changes the sample.
It depends on how small a difference you want to detect, since small gaps need many answers on each side to show up as clear. Usertick gives a variant verdict only once each side has 30 answers.
95% is the usual choice for business decisions. Use 90% when a quick read is enough and 99% when a wrong call would be costly, knowing that a higher level needs a bigger sample.
Listen better, act faster, and build what truly matters.
Start free, see what your customers mean, and get each answer to the person who can act on it.
- 150 responses a month on Free
- Every channel and AI analysis
- No card needed