What to Ask After a Usability Test? Subjective Completion ≠ Actual Task Success

A participant clicked “Start Export,” saw a green confirmation, said “Done,” and even gave the task the lowest difficulty rating. But no file was generated in the background: they had only saved the export configuration, not executed the final action. If the study only asked “Did it go smoothly?” at the end, this silent failure would have been recorded as an easy success.

Post-test questionnaires in usability testing can only measure how participants interpreted and evaluated their experience; they cannot substitute for objective task determination. A publishable conclusion should keep four layers of evidence separate: whether users achieved the intended outcome, what they did in the process, what they believed happened, and how they felt. When the layers agree, the conclusion is straightforward; when they conflict, that is often exactly where the most valuable problems lie.

“Is It Easy to Use?” Involves Outcomes, Effort, and Experience at Minimum

NIST describes usability testing as having representative users operate under representative tasks and realistic conditions, combined with observations of effectiveness, efficiency, and satisfaction. The three cannot substitute for one another:

Visual appeal is not equal to task usability, and completion does not guarantee ease. Conversely, someone may take a long detour and still succeed, or produce an incorrect result with no effort at all. Overall satisfaction is suitable for evaluating long-term relationships, whereas task-specific usability should focus on clear goals; the boundary between the two can be understood in connection with the product satisfaction survey guide.

Define the Task Judgment Table First, Then Write Questions for Participants

“Explore the data page and tell us what you think” cannot determine success. “Export the responses from completed projects in the previous quarter as a CSV, ensure the file does not include names or phone numbers, and save it to the designated folder” yields a verifiable outcome. Researchers should maintain a judgment table separate from the task card:

• Starting point: Logged into the project homepage with export permissions;
• Correct artifact: CSV for the specified date range generated and actually downloaded;
• Privacy condition: The name and phone number columns are not included in the file;
• Partial success: File generated, but date range or anonymization fields are incorrect;
• Failure: No file generated, wrong project exported, or the ten-minute stop condition reached;
• Allowed behaviors: In-page help is permitted, but the moderator must not point to buttons.

The task text participants see explains the goal and realistic context, without revealing navigation names or the correct path. The GOV.UK usability testing guide similarly requires that tasks and success criteria be clear and that prototype fidelity match research questions. When testing a low-fidelity prototype, you cannot treat unimplemented backend functionality as interface failure; to validate final export security, you need near-real data and system states.

First Round with 8 Users to Identify Mechanisms, Not to Claim “75% Success” as a Population Ratio

In the hypothetical case, the first round recruited 8 project administrators who had recently exported survey data, conducting moderated tests on their usual computers. The research team covered novices, experienced users, and participants using keyboard and screen readers. Three mechanisms were observed across the 8:

1. Some interpreted “save export configuration” as having started the export;
2. Some were unaware that anonymization field settings would carry over from the last configuration;
3. Some saw only the visual on-page cue and did not receive an accessible status notification that the file generation had failed.

This round can powerfully illustrate “how these failures happen,” but it cannot support declaring an overall success rate of 75% just because 6 completed the task. Participants were purposively recruited to cover key experience levels and devices; the sample is small, and moderation, probing, and task order can also alter behavior. GOV.UK explicitly distinguishes qualitative insight–driven usability research from quantitative testing that introduces task time, clicks, and other metrics. The exploratory round should deliver problem mechanisms, evidence snippets, and hypotheses for the next iteration, not precise percentages.

Second Round with 240 Users to Measure How Much the New Version Improves, Not “People Prefer It”

Based on the first round, the team designed a new version: separating “save configuration” and “start export” into distinct steps, showing a field list before export, and providing file status and error recovery after completion. They then recruited 240 people from populations matching the same roles and recent export needs, randomly assigning 120 each to the old version A and new version B; devices, tasks, test environment, and stop conditions were consistent. The following remains illustrative fictional data:

Complete success improved by 11.7 percentage points. Using a large-sample approximation for two independent proportions, the 95% interval for the difference is approximately 0.7 to 22.6 percentage points: the direction supports the new version, but the effect size remains highly uncertain. Do not treat the upper bound of 22.6 as a guarantee, nor slice the data into many small subgroups to seek significance. If the sample is clustered by account, weighted, or involves repeated measures, use models and intervals consistent with the design.

Time is calculated only among successes and must be presented alongside success rates. If the old version caused many slower users to fail outright, the remaining successful users' times will be distorted by the selection mechanism; excluding failure times and looking only at averages can lead to the wrong conclusion. When tasks have a cap, retain censored records that reached the stop condition instead of treating them as ordinary missing data.

Hidden Risk of the New Version: 88.3% Believed They Succeeded, but Only 80% Really Did

After task completion, participants were first asked, “Do you believe the file was successfully generated and saved as required?”, and then the research record and file contents determined actual success. The cross-tabulation for new version B is as follows:

If relying only on self-report, the success rate would be 106/120 = 88.3%, 8.3 percentage points higher than the objective 80.0%. More critically, 18 of the 24 who failed objectively believed they had succeeded—75%. This is not an ordinary “low experience score”; it is a silent outcome error: participants will not even proactively seek help. The team should prioritize fixing outcome confirmation, error alerts, and artifact verification, rather than releasing because the difficulty rating looks good.

Confidence is not always better. Correct but low confidence indicates insufficient feedback; incorrect but high confidence indicates the interface created false certainty. Cross-tabulating objective outcomes with subjective confidence is more actionable for design decisions than averaging the two into a single usability score.

Keep Post-Task Questionnaires Within the Scope of Explaining the Experience

Record immediately after each task is completed or stopped; the recommended order is:

1. The research system first saves objective state, time, errors, and help requests to avoid later discussion overwriting the original record;
2. Participants self-report whether they believe they completed the task and rate their confidence;
3. Use a stable scale to rate the difficulty of this specific task, keeping anchors unchanged across versions;
4. Open-ended probe: “Which step most influenced your judgment?”, asking participants to refer to specific events;
5. After all tasks are completed, ask about overall experience to prevent overall questions from shaping task-by-task ratings in advance.

Do not ask “Why did you click here?” at every step; this alters the natural flow. In moderated tests, observe first and review key moments after the task; unmoderated tests require event logs, page states, and clear anomaly classification. Digital.gov’s testing model separates task execution from subsequent questioning as distinct phases and also requires specific research objectives and tasks that map to those objectives—not “test the whole product” as a plan.

Problem Severity Is Determined by Consequences and Recovery, Not by the Number of Mentions

Frequency still matters, but a single privacy-leaking silent failure generally outranks a button color preference mentioned by several users. For each problem, record the affected task, user characteristics, device/assistive technology, evidence of occurrence, whether the user recovered on their own, consequences, and level of confidence. After iteration, rerun the same critical task to verify whether the mechanism is gone.

Testing with Participants with Disabilities and Standards Checks Must Run in Parallel

W3C notes that involving users with disabilities can uncover real barriers that consistency checks alone may not easily expose, but one or two individuals cannot represent all people with similar disabilities, and user testing cannot replace standards checks like WCAG. Coverage should span different assistive technologies, proficiency levels, and environments based on target users, and the scope should be clearly reported. A keyboard user encountering no issues does not mean screen reader, magnification, voice input, or cognitive accessibility all pass.

The research environment itself must also be accessible: task materials, consent processes, questionnaires, and meeting tools need adaptation. Code assistive technology failures, browser issues, prototype limitations, and product defects separately, while preserving their impact on real-world completion.

Screen Recordings Are Not Default Permissions, and Observations Are Not Promotional Material

Screens, audio, faces, file names, and test accounts may contain personal or confidential data. The GOV.UK research recording guide requires informed consent before recording, use only for agreed purposes, secure storage, and clear access boundaries for the team. Research consent does not automatically extend to using clips for marketing; observers, transcription vendors, retention periods, and withdrawal methods should all be explained in advance.

Survey platforms can handle participant screening, short post-task scales, version randomization, and open-ended answer organization; if tasks occur in an external prototype or live product, the research team still needs stable but minimal identifiers to link events and check non-respondents. A participant not returning to the questionnaire after a redirect is not the same as task failure, nor does completing the questionnaire prove success on an external task.

The final release decision should return to pre-set thresholds: whether the new version improves key task success, eliminates silent failures, does not introduce new accessibility or privacy risks, and whether the uncertainty in the difference is small enough to support the current decision. The value of usability testing is not to collect a “it works well” comment, but to prove that specific users, under specific conditions, can achieve the correct outcome and know that they have achieved it.

References for Task Usability, User Assessment, and Research Recording

• NIST — Usability Testing
• GOV.UK — Usability testing: qualitative studies
• Digital.gov — How to conduct a usability test
• W3C WAI — Involving Users in Evaluating Web Accessibility
• GOV.UK Service Manual — Taking notes and recording user research sessions