“Should I use a T-test or a chi-square test for this data?” Usually, the question is asked too early. A statistical test only assesses how incompatible the data are with a specific null hypothesis and its underlying model; it does not decide for the researcher whether the focus is on mean differences, proportion differences, entire distributions, within-individual changes, or the probability of meeting a business threshold. Choosing a test first and then interpreting software output can easily yield results that are mathematically computable but fail to answer the substantive business question.
A more reliable sequence is: first, write a single sentence specifying to whom, comparing what, and which effect size will guide the decision; then, determine whether the data come from independent samples, the same individuals over time, clustered stores, or complex sampling designs; only then select estimation methods and tests based on the outcome scale, sample sparsity, and model assumptions. Tests should be paired with the effect sizes and intervals reported—not just a P value.
Write an “Analysis Prescription Card” First—Don’t Proceed Until All Six Gaps Are Filled
- Target population: Do the conclusions aim to describe the actual respondents, current customers, or all potential consumers in a region? How did the sample enter the survey?
- Research question: Is it descriptive, comparative between groups, change within the same subjects, association among variables, prediction, or causal effect?
- Primary outcome and estimator: Which directly corresponds to action—proportion difference, risk ratio, mean difference, median difference, odds ratio, correlation coefficient, or model prediction error?
- Observational relationship: Are rows independent, do the same individuals respond repeatedly, are people nested within stores, classes, or households?
- Sampling and assignment: Is it probability sampling, quota/convenience sampling, or a randomized experiment? Are there stratification, clustering, weights, or randomization groups?
- Decision rule: How large a difference is practically meaningful? What interval or costs of errors guide the decision? Which outcomes are primary, secondary, or exploratory?
For example: “Compare the proportion of questionnaire completion between versions A and B of a landing page among randomly assigned users in this experiment; the primary effect is the percentage-point difference (B minus A). If the 95% interval contains zero but excludes the business-required 8-percentage-point improvement, do not scale.” This card already defines the population, outcome, estimator, design, and action. In contrast, “Test whether A/B is significant” does not specify how large a difference warrants implementation or whom the results can be generalized to.
In a 500-Person Experiment, Calculate the 9.2-Percentage-Point Difference Before Getting to P
Assume 500 eligible users are randomly assigned with equal probability to two invitation versions: 250 per group. In version A, 112 complete the questionnaire (44.8%); in version B, 135 complete it (54.0%). The primary result is not “significant/not significant” but:
Absolute proportion difference = 54.0% − 44.8% = 9.2 percentage points
Relative ratio = 54.0% ÷ 44.8% ≈ 1.21Under simple independent random assignment and using a normal large-sample approximation, the 95% interval for the absolute difference is approximately 0.5 to 17.9 percentage points. This interval shows that the data still support a wide range of improvement: the low end is only half a percentage point, and the high end approaches 18 percentage points. If the cost of implementation requires at least an 8-percentage-point improvement, the result just barely excludes zero but does not exclude effects below the business threshold. Continuing data collection, incorporating cost decisions, or acknowledging uncertainty is more honest than declaring “statistically significant.”
For two independent proportions, one can report the difference and its interval, along with a two-proportion test or a chi-square test on the contingency table; if expected frequencies are very small, consider Fisher’s exact test. The test does not provide an effect size, so report the denominator, proportion, and difference for each group. If adjustment for pre-existing covariates is desired, use an appropriate regression model but output marginal proportion differences or other business-friendly effects rather than merely copying a column of coefficients.
Random assignment supports the causal comparison of the two invitation versions on this experimental sample; it does not automatically make these 500 individuals representative of all customers. Conversely, probability sampling facilitates describing a population, but without random assignment, observed differences between user groups cannot be directly interpreted as being caused by a feature. Sampling generalization and causal estimation are two distinct gateways.
The Same “Satisfaction” Variable: Four Designs Change the Unit of Analysis
| Practical Question | Data Relationship | Preferred Report | Common Candidate Methods |
|---|---|---|---|
| Do average scale scores differ between two groups of users (A vs. B)? | Two independent groups | Distributions per group, mean difference with interval | Welch T-test or linear model; robust/resampling methods if needed |
| How do scores change for the same users before vs. after an upgrade? | Paired individuals | Distribution of per-person change, mean or median change | Paired T-test, sign/signed-rank or longitudinal models appropriate to the target |
| Users belong to 20 stores; compare store-level interventions | Clustering within stores | Intervention difference with interval accounting for stores | Cluster-robust variance, multilevel models, or analysis at assignment-unit level |
| Stratified by region, oversampled rare populations | Complex survey sample | Weighted estimates with design-based intervals | Survey analysis methods incorporating weights, stratification, and clustering |
Treating two rows from the same individual as independent samples discards within-person correlation and miscomputes uncertainty entirely; treating 500 customers in a store-randomized experiment as 500 independent random units overstates effective information, because the actual randomized units may be only 20 stores. Determining “who was randomized, who was repeated, who shares environments” is more important than first checking normality.
Errors in complex sampling are especially subtle: the weighted proportion may be computed correctly, but the standard error might still be calculated as if from a simple random sample. The CDC’s NHANES Variance Estimation Tutorial clearly distinguishes that weights affect parameter estimates, while stratification, clustering, and unequal weighting affect standard errors, test statistics, and confidence intervals. Adding a weight column before an ordinary chi-square or T-test does not necessarily amount to survey design analysis.
Variable Scale Narrows the Candidate Set but Does Not Solely Specify the Answer
Binary outcomes such as completion or purchase: prioritize interpretable metrics like absolute proportions, proportion differences, or risk ratios; when adjusting for covariates, use binomial outcome models and convert to predicted probabilities or marginal effects. The odds ratio is not the same as the risk ratio, especially when outcomes are common—use caution in wording.
Nominal categories such as preferred brand can be described with frequencies, proportions, and contingency tables; for testing independence, chi-square tests, and for sparse small tables, exact methods. With multiple categories and covariates, multinomial models may be considered. Confirm whether “other,” “don’t know,” and missing values belong in the analysis categories.
Ordinal scales such as a single five-point satisfaction item: at least show the proportion in each category. If the question concerns whether entire response tendencies differ, consider ordinal models or rank-based methods; if the intent is to treat scale scores as an approximately continuous measure, specify that the average score difference is being estimated and check the distribution and robustness. Method names are not a substitute for stating the estimation target.
“The data aren’t normal, so we use Mann-Whitney to test medians” is a common misconception. The BMJ methods note points out that Mann-Whitney is not a general median test; shifts in distribution shape or variability may also affect results. If the actual interest is in median differences, estimate medians and their intervals directly, or use resampling/quantile methods that match that objective; do not rename the P value of a rank test to “medians are significantly different.”
Continuous, duration, or monetary outcomes can be reported with means, medians, quantiles, and distribution plots; then choose mean models, robust standard errors, transformations, quantile regression, or resampling according to the goal. Heavy tails do not automatically ban means: budget problems might legitimately care about per-capita costs; they just require assessing the influence of outliers, sample size, and modeling choices.
Comparison, Association, and Prediction Answer Different Questions
“High-frequency users are more satisfied” can be a descriptive association: correlation coefficients, group differences, or regression coefficients quantify co-variation, but usage frequency may be simultaneously influenced by product value, user tenure, and self-selection. One cannot conclude from a significant association alone that “increasing usage will raise satisfaction.” If the goal is causal effects, randomization, natural experiments, or an explicit and defensible causal identification design is needed.
“Which variables predict renewal” concerns out-of-sample predictive performance; do not select variables simply by the P value of each coefficient. Distinguish training from validation, choose loss functions or discrimination metrics, and check calibration, overfitting, and data leakage. Models for explaining effects and for achieving predictive accuracy may be similar but have different evaluation criteria.
Similarly, post-test satisfaction among completers cannot be casually used as the effect of the A/B invitation experiment. If the invitation version also affects who completes, then comparing satisfaction only among completers conditions on an outcome influenced by the treatment, making the two groups of completers no longer comparable. The primary outcome and measurement time should be fixed in the design phase, not chosen after data collection based on who has data.
With 20 More Metrics to Look At, at Least One “Significant” Result Is Not Surprising
If a team compares 20 metrics across 10 subpopulations, they are conducting many opportunistic tests. Even if all null hypotheses are true, small P values may still appear by chance. The solution isn’t to cram all results into the same correction table but to distinguish—in a pre-collection analysis plan—between one or few primary outcomes, limited secondary outcomes, and exploratory findings.
For a set of tests that jointly inform a decision, control the family-wise error rate or the false discovery rate, depending on the purpose; label exploratory results as requiring replication. Don’t run everything first and then name the smallest P value the “primary finding.” Subgroup analyses should also check actual base sizes and interactions: a significant result in one group and not in another does not imply that the two groups’ effects are significantly different.
A Publishable Result Should Address at Least Seven Items
- Analysis population, inclusion/exclusion criteria, unweighted counts per group, and weighted bases if relevant;
- Outcome variable, units, coding direction, handling of missing values and “not applicable”;
- Effect size and its direction—for example, a 9.2-percentage-point increase—not merely “there is a difference”;
- Confidence intervals or other expressions of uncertainty, plus business action thresholds;
- How independence, pairing, clustering, stratification, weights, and random assignment enter the methods;
- Model assumptions, outliers, sparse cells, and whether conclusions are stable under alternative methods;
- The boundary between primary and exploratory analyses, and how multiplicity was handled.
The ASA’s Statement on Statistical Significance and P-values emphasizes that a P value does not measure the size of an effect or the importance of a result, and decisions should not be based solely on whether a threshold is crossed. The NIST/SEMATECH Engineering Statistics Handbook serves as a systematic entry point for checking distributions, comparisons, modeling, and assumption diagnostics.
In a survey kickoff, first verify item coding, denominators, skip patterns, weight fields, and cross-tabulations, then export to a statistical environment that can properly handle paired, clustered, or complex survey data. AI can help list candidate methods and interpret results, but the input must contain the six elements from the analysis prescription card; if you only provide a column of numbers and a sentence like “see if it’s significant,” any automated recommendation will likely guess the research question for you.
The ultimate criterion is simple: Does the single number in your report precisely quantify the difference the decision-maker asked about? Does its interval respect the true independent units and sampling design of the data? If you substitute a reasonable alternative method, does the action remain the same? Only when these three points are met does the test name carry meaning.