Before finalizing a survey, write the last page of the report first—not as a conclusion, but as a blank results page: which metric drives the decision, who is in the denominator, which groups are compared, how large a difference warrants a change in plans, and what happens if results are uncertain. If any cell cannot be filled in, the survey is often not yet ready for formal fielding.
That is the most practical value of an analysis plan. It is not a software command list for statisticians, nor does it turn a project into an academic paper. It is an analytical contract made before data collection: the team agrees in advance how answers will become evidence and under what conditions the agreement may be amended. It reduces post hoc re-specifications of metrics, slicing of subgroups, and cherry-picking of indicators, while also surfacing disconnects among questions, samples, and decisions early on.
Example: Start with a “conclusion page without numbers” for a product concept test
Suppose a team has two new concepts, A and B. In three weeks, they must decide whether to move into development with A, with B, or with neither. Business stakeholders initially ask for ten evaluation ratings and plan to see “which one scores higher.” If all questions are equally weighted, it is almost always possible to find a winning metric after the data arrive: A is more novel, B is more credible; younger groups like A, existing customers prefer B. Results abound, but there is no rule for the decision.
Before collecting data, write this blank page:
Decision: A / B / Defer development
Primary outcome: Correct rate for core task comprehension
Valid denominator: Qualified target role, fully exposed to randomly assigned concept, and completed the comprehension task
Primary comparison: Difference in correct rates between A and B with confidence intervals
Business threshold: At least an 8 percentage point lead to justify switching from the current approach
Key risk: If either concept falls below a preset credibility floor, do not proceed directly to development based solely on the comprehension lead
Planned subgroups: First-time buyers, existing buyers; used only for consistency checks, not to select a separate winner
Uncertain outcome: If confidence intervals include both meaningful positive and negative differences, add sample or conclude insufficient evidenceThis page immediately imposes specific requirements on survey design: a comprehension task that does not rely solely on self-report, randomization and full exposure of concepts, reliable identification of first-time versus existing buyers, a sample size sufficient for precision around the 8-point difference, and the credibility risk question not buried among ten secondary ratings. If the team cannot explain why “an 8 percentage point difference is worth acting on,” the problem lies not in statistical methods but in an as-yet-undefined business decision.
The blank results page filters out useless questions in reverse
Assign a destination to every question in the survey: it generates a primary metric, a secondary metric, sample eligibility, a grouping variable, a quality check, or an explanatory outcome. A question without a destination is not necessarily worthless, but it should be justified in terms of the added respondent burden. If the only rationale is “might be useful later,” prioritize deletion or assign it to a clearly labeled exploratory module.
A reverse check often reveals four types of disconnects:
- Research question but no variable: The report plans to compare comprehension rates, but the survey only asks “Do you feel you understood?”
- Variable but no valid denominator: A satisfaction item is shown only to users, but the plan still uses all completers as the denominator.
- Grouping but no sample: The plan compares three industries and four customer sizes, but there is no quota or sufficient base to support twelve cells.
- Indicator but no action: Ten scores—brand appeal, innovativeness, credibility, uniqueness, etc.—are collected, yet none changes the decision to move into development.
Drawing the results page first also exposes structural gaps caused by response coding and skip logic. Before full launch, use cognitive pretesting and small-scale pilot runs to confirm how respondents interpret questions and whether data are indeed recorded as planned.
Classify metrics as primary, secondary, or exploratory—rather than fighting over “most important” later
Primary metrics directly carry the core decision and should be limited in number. A concept test might set the task comprehension correct rate as the primary outcome; a satisfaction tracking study might set a pre-defined overall satisfaction proportion as the primary outcome. The definition must include the source question, coding, direction, denominator, time window, and calculation formula.
Secondary metrics explain the primary outcome, monitor side effects, or support supplemental decisions with a business rationale. For example, credibility is a risk floor for moving into development, and completion time checks whether comprehension gains come at the cost of higher cognitive burden. Secondary does not mean ignorable; it merely is not the sole winner criterion.
Exploratory metrics are used to identify new leads, such as open-ended impressions, unpre-specified segment differences, or new composite indices. They may enter the report, but they must be labeled as hypotheses generated after seeing the data and should not use the same definitive language as pre-specified primary outcomes. A subsequent research round may upgrade genuinely important findings to pre-specified metrics.
This tiering is not meant to discourage discovery, but to avoid collecting twenty metrics and then showcasing only the two “best-looking” ones. If every metric claims to be primary, there is effectively no primary metric.
Valid denominators and exclusion rules must be written as executable statements
“Use valid responses” is not a rule, because different analysts will interpret “valid” differently. The plan should separately define:
- Who enters the study population, e.g., those passing eligibility screening and belonging to the target role;
- Who had the opportunity to answer a specific question, e.g., those who saw concept B in full;
- What constitutes a valid value for the metric, e.g., correct answers coded as 1, errors as 0, and “not answered” not automatically coded as 0;
- Which quality problems exclude the entire response, and which cause only single-item missingness;
- How multi-select, skip patterns, break-offs, refusals, and “don’t know” enter the denominator.
Exclusion rules should rely as much as possible on evidence that is relevant to the study purpose and verifiable. Deleting an entire response solely because completion time was shorter than an arbitrary threshold may mistakenly disqualify those familiar with the task; adding exclusions only because a subgroup’s results seem “unreasonable” makes rules follow conclusions. Combine multiple quality signals in advance and retain sensitivity analyses that include or exclude borderline cases.
Sample plan: not just total number, but aligned with the real comparisons
If the main conclusion is the difference between A and B, the sample and randomization must support that comparison. If a separate decision is needed for first-time buyers, the recruitment and quota stage must support that base as well. A total sample of one thousand does not mean every subgroup has sufficient information, especially when eligibility rates are low, bases shrink after skip logic, or weights are applied.
The plan should state the estimand, expected precision, minimum meaningful difference, estimated eligibility and completion rates, and assess sample needs under different assumptions. For detailed calculations, refer to survey sample size and error design methods. If the budget supports only directional judgments, narrow the comparisons and wording rather than retaining numerous subgroups that are doomed to be uninterpretable.
Statistical thresholds cannot replace business thresholds. The American Statistical Association’s statement on p-values notes that scientific conclusions and business or policy decisions should not depend solely on whether a p-value passes a threshold; nor do p-values measure effect sizes or the importance of results. The analysis plan should also agree to report point estimates, intervals, actual differences, and the costs of actions, rather than writing “go live if significant.”
A good plan anticipates “what to analyze if the unexpected happens”
Real data will always present unplanned situations. Pretending they will not happen is as dangerous as changing the rules arbitrarily after the fact. A better approach is to write conditional branches in advance:
- If a subgroup’s effective base falls below an interpretable minimum, report descriptively only and do not compare winners;
- If key baselines are severely imbalanced after randomization, retain the primary unadjusted result and provide adjusted results per a pre-specified model;
- If missingness on a metric exceeds a preset proportion, report missing patterns, handle it per the agreed method, and conduct sensitivity analyses;
- If quality rules inadvertently flag a particular device or population, pause the formal exclusion, investigate the mechanism, and document revisions;
- If the primary result conflicts with the risk floor, follow a pre-defined priority order for review rather than ad hoc selection.
The UK government’s TIGER transparency guidance (2026) notes that a statistical analysis plan should document data transformations, exclusions, variables, models, and uncertainties before analysis, and may use “if–then” rules to handle unpredictable data attributes such as missingness. The key is not the specific form of public registration, but that rules are time-stamped, saveable, and detailed enough that a third party could execute them independently.
Plans can be amended, but history cannot be rewritten
Issues found in pretests, anomalies during formal fielding, or actual missing structures that differ from expectations may require amendments to the plan. Each revision should record at least: the time of the change, whether results had been seen at that point, the reason for revision, the affected metrics and samples, the old versus new rules, and whether the primary analysis changes as a result. Keep the original plan and versions rather than overwriting with new files and claiming “this was always the plan.”
Revising before seeing primary results is usually easier to avoid outcome-driven bias. Reasonable additions after seeing results are also permissible, but they should be explicitly labeled as exploratory or sensitivity analyses. Pre-specified results answer the initial questions; post-hoc analyses explain anomalies, check robustness, or generate next-round hypotheses—both matter, but their roles differ.
In survey platforms, you can first map research questions, questions, and variables, then use pilot data to check skip logic, coding, completion times, and planning subgroup bases. During formal analysis, retain raw responses, quality flags, derived rules, and report versions so that the blank results page can later be filled with real data under the same definitions. AI can help identify rule gaps or generate analysis descriptions, but it cannot decide for the team which metric is primary, nor can it unilaterally change decision thresholds after results appear without a trace.
A serviceable analysis plan is not measured by page count. Small, low-risk surveys may need only a one-page blank results matrix plus one page of rules; high-cost, strongly causal, or regulatory research requires fuller protocols, code, and review. The common standard is singular: can another analyst, without consulting the original author, determine from the plan who is analyzed, what is computed, how comparisons are made, how exceptions are handled, and what evidence would actually change the decision?