Published · · Last reviewed · · 6 min read
English · 한국어
How 1,000 people can speak for millions
"We surveyed 1,000 adults nationwide. At the 95% confidence level the margin of error is ±3.1 percentage points." You hear sentences like this in the news. With a population of tens of millions, how can 1,000 people speak for everyone? And what exactly does "A 42%, B 38%, within the margin of error" mean? This page calculates where the margin of error comes from, and then points to the problems that a bigger sample does not solve. Every case below is an invented survey for explanation and not a real poll.
Where the margin of error comes from
Suppose the true share of people in the population who give some answer is p, and n people are chosen independently at random and asked. The share found in the survey wobbles around p, and the size of the wobble (the standard error) is √(p(1−p)/n). A 95% confidence level means the true value lies within about 1.96 standard errors of the result 95% of the time, so the margin of error is 1.96 × √(p(1−p)/n). If p is unknown the largest value, p = 50%, is used, and with n = 1,000 this gives 1.96 × √(0.25/1000) = 3.1 points. That is the "±3.1" in the news.
| Sample size n | Margin of error |
|---|---|
| 100 | ±9.8 points |
| 400 | ±4.9 points |
| 1,000 | ±3.1 points |
| 2,500 | ±2.0 points |
| 10,000 | ±1.0 points |
To halve the margin you need four times the sample (the square-root law). Going from ±3.1 with 1,000 people to ±1.5 takes about 4,270, and ±1 takes 9,604. The sample needed for a target margin (share 50%, 95% confidence):
| Target margin | Sample needed |
|---|---|
| ±5 points | 385 |
| ±3 points | 1,068 |
| ±2 points | 2,401 |
| ±1 point | 9,604 |
Why the size of the population does not matter
The population size does not appear in the formula. One spoonful tastes a pot of soup, and doubling the pot does not call for a bigger spoon; what counts is whether the soup was stirred (whether the sample is random). A sample of 1,000 has the same margin whether the population is 50 million or 500 million. The margin shrinks only when the population is small enough for the sample to be a sizeable fraction of it. In a school of 300 students, asking 100 at random gives a correction factor √((300−100)/(300−1)) = 0.82, which brings ±9.8 down to ±8.0 points.
What "within the margin of error" means
Say A gets 42%, B 38%, n = 1,000. Each has a margin of about ±3, so 42±3 and 38±3 overlap and people call it "a close race". Strictly, you need the uncertainty of the difference (4 points). For two shares from the same respondents the standard error of the difference is √((pₐ + p_b − (pₐ−p_b)²)/n), which here gives 1.96 × √((0.42+0.38−0.0016)/1000) = 5.5 points. A 4-point difference lies inside an error of 5.5, so one cannot say that A leads, and one cannot say B leads either. The accurate reading is that this survey cannot tell whether A is ahead or behind.
Subgroups are rougher. If 200 of the 1,000 are in their twenties, the margin for that group alone is ±6.9 points. When reading figures split by age, region or sex, first check how many people are in that cell.
Run your own surveys
Enter a sample size and a true share, and the tool repeats an imaginary survey 2,000 times; about 95% of the results fall within the margin.
Run the tool above and the share of results inside the margin comes out near 95%. That is what "95%" means: out of 100 surveys, about 95 land within the margin of the true value.
How to read the interval
Suppose an invented survey finds that 38% support some policy (n = 1,000). The margin is ±3.0 points (3.01 at p = 38%), so the interval is 35.0% to 41.0%. It is tempting to say "there is a 95% chance the true support lies in this interval", but the strict meaning is a little different: if surveys of this kind were repeated again and again, about 95% of the intervals built this way would contain the true value. Whether this one interval contains it cannot be known; only the method has a 95% success rate. In daily life reading it as "the true value is probably somewhere between 35% and 41%" is not far off, but the edge of the interval is not as plausible as the middle. Values nearer the middle are more likely.
Reading a change
Three weeks later the same survey is repeated and support comes out at 41% (n = 1,000, different people). It is tempting to say "up 3 points", but the difference between the two surveys has an error too. For two independent surveys the standard error of the difference is √(0.38×0.62/1000 + 0.41×0.59/1000), so the margin is 1.96 × 0.0219 = 4.3 points. A 3-point change lies inside 4.3, so one cannot tell whether support rose or stayed the same. Each survey being ±3 does not make the error of the difference ±3; it is larger (about √2 times). When a result is reported as "up" or "down", check whether the change exceeds a value like this 4.3.
What "random" means in sampling
The margin formula assumes a simple random sample in which everyone has the same chance, but other designs exist. Stratified sampling divides the population into strata such as age and region and draws randomly within each; the more alike the people inside a stratum, the smaller the error for the same sample size. Quota sampling sets each group's share to match the population and then fills it with people who meet the condition; it is convenient, but if the interviewer picks who fills the quota the sample is not random and the margin formula cannot be used as it stands. How the sample was drawn decides what the margin of error means. Whether a report says "random sample", "quota" or "online panel" is a key to reading the news.
Problems that a bigger sample does not solve
The margin of error covers only the chance error from drawing at random. The problems below do not go away with a larger sample, and none of them is inside the margin.
A sample that is not random. Ask 1,000 customers outside a lottery shop "Do you play often?" and you can compute a ±3.1 margin, but the respondents do not represent adults at large, so the figure means nothing. A margin of error is meaningful only if everyone had a known chance of being selected.
People who did not answer. The more people refuse or do not pick up, the more the result is skewed if responders differ from non-responders. This page claims no figures for how much bias a given response rate produces.
Question wording. The same matter can get different answers depending on phrasing and order (compare Why big numbers blur our judgement, where the way a fact is expressed changes judgement).
Weights and adjustment. Real surveys adjust the sample's makeup to the population with weights. Weighting reduces the effective sample size, so the margin is larger than the simple-random-sample value. Whether the ±3.1 in a news item includes this effect can only be found in the survey report.
What to check when reading survey figures
When you meet a survey result, ask: how was the sample drawn (at random, or a convenient group)? How many were asked and how many answered? If two values are compared, what is the margin of their difference? How many people are in the subgroup? These questions decide not whether a figure is accurate but what it is entitled to say.
When these formulas stop working
The formulas assume a simple random sample and the normal approximation; with a share near 0 or 100 or a very small sample, other interval methods are needed. Official rules for election polls in Korea and the methods of survey organisations are not covered, and no real survey is cited. All calculations were checked by script and are pinned by a test.
Sources and verification
- LottoFlick: Why big numbers blur our judgement (framing and the size of numbers) — computed by LottoFlick
- LottoFlick: What a positive test really means (conditional probability with invented numbers) — computed by LottoFlick