Survey weighting is not about "fixing" a sample to make it representative; it is about altering the contribution of each record to estimates, based on the sampling design, response process, and credible population information. It corrects for known and measurable selection differences, such as when researchers deliberately oversample small areas or when certain client groups have lower response rates; it cannot compensate for people entirely missing from the sampling frame, nor can it fix flawed questions, false responses, or self-selection bias not captured by the weighting variables. Without a clear target population and reliable benchmarks, it is often more honest to report unweighted results than to assign arbitrary weights.
Formal weights often are not computed in a single step but form a chain: start with a base weight based on selection probabilities, apply nonresponse adjustments, and finally calibrate the sample to known population totals. The U.S. Census Bureau's SIPP weighting description includes base weights, subsampling within clusters, nonresponse, panel integration, and post-stratification calibration. Business surveys may not require such complexity, but they must know which step was omitted and what can still be claimed after that omission.
First, Diagnose the Type of Imbalance
Whether weighting works depends on how the imbalance arises. Design imbalance occurs when researchers intentionally make one group more likely to be selected, e.g., collecting 400 responses from new and 400 from existing customers when these groups are not equally represented in the population; this difference has a known design basis and should usually be undone when estimating population parameters. Nonresponse imbalance occurs when sampled individuals have different probabilities of participating; if auxiliary information is available from the sampling frame for both respondents and nonrespondents, adjustments can be made within similar groups.
Coverage imbalance is more problematic: if only online channels are used, those entirely offline have no chance of entering the sample; if the data also lack adequate variables associated with this omission, post-hoc calibration on age and region cannot reconstruct their answers. Volunteer opt-in samples may also differ from the population in interest, trust, time, and intensity of opinion. Even if demographics are balanced, within-group bias may persist. Pew Research Center experiments with online opt-in samples have found that larger sample sizes mainly reduce random noise but do not automatically eliminate systematic selection bias; and more complex weighting methods do not guarantee higher accuracy.
Therefore, before proceeding, classify the problem into one of the following categories:
- Known unequal selection probabilities: Keep design weights; treat records as not having equal selection probabilities by default.
- Differential response after selection: Use auxiliary variables from the sampling frame to create nonresponse adjustments, but explain the model assumptions.
- Sample structure deviates from credible population totals: Post-stratification or iterative proportional fitting (raking) can be applied, provided variable definitions are consistent.
- Population or benchmarks unknown: Report only within-sample results, subgroup ranges, and limitations; do not create population estimates with self-imposed quotas.
A Case with 800 Responses: Why 58% Becomes 64%
The following is an illustrative calculation. A subscription product wants to estimate renewal intention among all active paying customers over the next three months. The customer database shows 30% new and 70% existing customers. To analyze the experience of new customers separately, the team deliberately collects 480 responses from new and 320 from existing customers. In the sample, 240 of the new customers (50%) and 224 of the existing customers (70%) say they intend to renew.
A simple pooled calculation gives 464/800 = 58%. But the sample has increased the proportion of new customers from 30% in the population to 60%. If the database boundary, status timing, and unique customer definition all match the survey, the population renewal intention should be calculated using target shares: 0.30 * 0.50 + 0.70 * 0.70 = 0.64, or 64%.
Alternatively, one can assign a relative weight of 0.5 to each new customer record and 1.75 to each existing customer record. The new customer weight sum is then 480 * 0.5 = 240; for existing customers it is 320 * 1.75 = 560; combined, the total still normalizes to 800, but the contribution is restored to 30% and 70%, respectively. The raw 58% answers "What do these 800 participants say?", while the weighted 64% is an estimate for all active paying customers, under the design assumptions.
What matters most is not the six-percentage-point difference, but whether the target benchmark is correct. If the database defines "existing customer" as registered for one year, but the survey asks based on first payment a year ago, direct weighting is not valid even if the labels match. If the study targets only customers with usage activity this month, the historical customer database structure cannot serve as the benchmark. Weights do not check definitions; they precisely implement the target provided by the researcher.
Weights Correct Bias but Also Cost Precision
In the example above, the sum of weights is 800, while the sum of squared weights is 1100. Using the common Kish approximation, effective sample size equals (sum of weights)^2 / (sum of squared weights) = 640,000 / 1100 ≈ 582. This means that, solely considering the unequal weights, these 800 responses provide information equivalent to about 582 equally weighted independent observations for population estimation.
This calculation only reflects weight variation, not the full design effect caused by cluster sampling, repeated measures, or complex nonresponse models, but it quickly reveals the cost. The more extreme the weights, the greater the influence of a few records on the results, and typically the lower the effective sample size. One cannot publicly claim a sample of 800 while presenting weighted proportions and still assuming 800 equally weighted independent observations in confidence intervals and significance tests.
Extreme cells are particularly dangerous. Suppose existing customers from a certain region constitute 9% of the target population but only 2% of the 800-response sample. A simple cell adjustment would give each of these 16 records a relative weight of 4.5. When total weights are normalized to 800, one person's answer could shift the overall proportion by about 0.56 percentage points—an influence 4.5 times the average record. The most effective fix is usually to collect more responses in that cell, collapse categories with substantive rationale, or reduce the number of weighting dimensions, rather than letting the algorithm amplify noise further.
How to Choose Among Post-Stratification, Raking, and Propensity Weighting
Post-stratification requires the target counts or proportions for complete cross-classified cells, e.g., a two-by-four table of new/existing customers across four regions. It is easy to interpret and accurately reproduces each cell, but empty or very small sample cells result in undefined or extreme weights. If only marginal distributions are known, one cannot pretend multiple marginals are a known joint distribution.
Iterative proportional fitting (raking) adjusts a series of variables sequentially, making marginal distributions for age, region, customer stage, etc., gradually approach their respective targets until changes are sufficiently small. It does not require known population shares for all cross-classified cells, but it still requires sample coverage for target categories, and convergence criteria, iteration limits, and weight bounds must be specified. Matching marginals does not guarantee that unspecified cross-class relationships are correct.
Propensity weighting (response propensity or selection propensity) uses a model to estimate the probability that a person responds or enters a nonprobability sample, and weights by the inverse of that probability. Models can accommodate more variables and nonlinear relationships, but the assumptions are stronger: the sample and the reference data must have identically defined auxiliary variables, and the model must capture the selection mechanism related to the outcome. If the variables truly driving participation and answers are not measured, no model can see them.
When choosing a method, first select useful variables, then choose the algorithm. A variable is generally suitable for weighting if it meets three criteria: it has a credible population benchmark with the same definition; it is reliably measured in the sample; and it has a clear relationship with the response or selection mechanism and key outcomes. Statistics Canada's published research on nonresponse weighting emphasizes that the predictive power of adjustment variables for survey outcomes is particularly important; predicting response propensity also helps. Including a dozen variables simply because they exist creates sparse cells, extreme weights, and models that are hard to interpret.
Trimming Is Not a Lossless Fix—It's a Bias-Variance Trade-off
Trimming overly large weights to a cap, or collapsing small adjacent cells, can reduce the influence of a few records and improve effective sample size, but the weighted distribution may no longer exactly achieve the target. For example, if the 4.5 weight is capped at 3, that cell becomes underrepresented; re-calibrating is likely to push the lost weight onto other records. Trimming rules should be pre-specified, ideally before examining key results, and both untrimmed and trimmed versions should be retained for sensitivity analysis.
Do not replace diagnostics with "the industry usually trims at some value." A reasonable threshold depends on sample size, benchmark sparsity, key indicators, and acceptable variance. Teams should compare weight minimums, maximums, percentiles, coefficients of variation, effective sample sizes, as well as point estimates under unweighted, untrimmed, and several candidate trimming rules. If the direction of conclusions only holds at a specific threshold, the report should state that results are sensitive to the weighting treatment, rather than selecting the most favorable version.
Weighting Quality Checks Should Cover Both Numbers and Meaning
- Population boundary: Do the benchmark and the study use the same objects, geography, status, and time point?
- Variable mapping: Are category definitions, missing treatment, and "don't know" handling consistent between sample and benchmark?
- Coverage check: Do categories with target proportions greater than zero have any records in the sample? Are there extremely small cells?
- Calibration results: Does the weighted distribution of specified variables reach the target? Did the algorithm converge?
- Weight distribution: Examine the minimum, maximum, median, quantiles, coefficient of variation, and the origins of high-weight records.
- Precision cost: Calculate the weight design effect and effective sample size; use appropriate variance estimation methods for complex samples.
- Sensitivity of conclusions: Compare unweighted, full-weight, and trimmed versions; explain which groups drove changes.
- Subgroup reporting: Always show true unweighted base sizes; do not present weight sums as if they were numbers of respondents.
When weighting variables themselves have missing values, records should not be silently dropped. Methods such as a "missing" category with justification, imputation, or modeling can be used, but the original variables, derived weighting variables, and treatment flags should be stored separately. Formal complex surveys may also require stratification, clustering, and replicate weights; analysis software must incorporate these design elements into standard errors, so ordinary equal-weight tests are no longer appropriate.
Studies Where Weighted Proportions Should Not Be Used as Conclusions
Exploratory in-depth interviews and purposive recruitment focus on the range of experience; they do not need to weight a dozen people up to population totals. For product concept tests without a defined target population, report reactions by key user groups separately rather than imposing a market structure yourself. Convenience samples from open links, social media sharing, or a single partner organization may match the census on age and region, but that does not mean each person had a known probability of inclusion.
Nonprobability samples are not completely unable to use adjustments, but conclusions depend on models and observable variables. Such results should be written as model-assisted estimates, and unweighted results, benchmark sources, and sensitivity analyses must be provided. In particular, do not directly apply conventional margins of error that assume only random sampling error to such results. Weighting addresses the question "How should observed differences be adjusted?", not the proof "Unknown biases have disappeared."
A Methodological Description You Can Include in a Report Directly
The above case can be described as follows: "The target population was unique paying customers still active on the reference date; customer stage distribution was obtained from the de-duplicated account database as of the same date. The study oversampled new customers for separate analysis; the sample contains 60% new and 40% existing, while the population contains 30% and 70%, respectively. Population estimates use post-stratification weights by customer stage, with relative weights of 0.5 for new and 1.75 for existing customers, normalized to the sample total. The Kish effective sample size for the 800 nominal responses is approximately 582. National results are based on weighted estimates; subgroup tables also report unweighted base sizes. No corrections were made for unobserved selection bias."
For true reproducibility, weight files should also include benchmark tables and their versions, input weights at each step, variable recodes, missing data treatment, algorithm and convergence thresholds, trimming rules, final weights, design effects, and generating code. When benchmarks are updated in the next cycle, keep old versions to avoid historic dashboards changing due to background overrides. To improve precision at the source, consider quota and oversampling strategies to reduce extreme imbalance; for combined national and regional targets, consult the regional sample allocation guide; before release, follow the survey methodology reporting template to disclose sample, weights, and limitations.
Methodological Basis and Further Reading
- U.S. Census Bureau: SIPP Weighting: Base weights, nonresponse adjustments, panel integration, and post-stratification calibration.
- Statistics Canada, Survey Methodology: The role of survey outcome variables in nonresponse adjustment: Relationship between adjustment variables and survey outcomes and response mechanisms.
- Pew Research Center: How different weighting methods work: Raking, matching, propensity weighting and experiments with online opt-in samples.
- AAPOR Transparency Initiative: Disclosure of weighting variables, computation methods, benchmark sources, and precision for nonprobability samples.