The storyboard ad achieved 75% liking, while the demo version scored only 57%, leading the creative team to assume a decisive victory. Yet when participants were asked to recall the ad's message and brand without any prompted options, only 39% of storyboard viewers answered both correctly, versus 52% for the demo. People may love the story, but that does not mean they know who it is for or what it says.
Ad creative testing should first assess whether a specific communication task has been accomplished—for example, “ensuring that first-time viewers know the brand can distill interview quotes into traceable themes.” Attention, comprehension, brand attribution, credibility, emotion, and action intent are distinct dimensions and should not be averaged into a single composite creativity score. The most-liked version may be selected, but only if emotional connection is the primary objective; if the goal is to establish a new use case for the product, high liking without a link between the benefit and the brand does not constitute a win.
Before Testing Creative, Freeze What Must Be Communicated
First, draft a one-page communication task card:
- Who: The target audience not yet familiar with the feature, with a recent relevant task;
- Where: In-feed short video, pre-roll ad, in-store screen, or landing page—do not mix different environments;
- What to create: Correct brand association, one core benefit, and a credible reason-to-believe;
- What not to create: Misinterpreting auxiliary analysis as automated decision-making, or remembering the story but attributing it to a competitor;
- Next-step behavior: Understanding the feature, viewing case studies, or trying the product—not vague “drive conversions.”
If version A emphasizes time savings and version B emphasizes accuracy, while also changing actors, music, brand exposure, and price, the two are not a comparison of expressions of the same communication task. The team can only choose “which entire ad performs better in this study” and cannot attribute differences to any single element. If the product’s value proposition itself is not yet validated, begin with a product concept test; creative research is not responsible for proving an unsubstantiated claim to be true.
Don't Compress Six Layers of Evidence into One "Effectiveness Score"
| Layer | Research Question | Typical Evidence |
|---|---|---|
| Exposure Opportunity | Whether the asset loaded, stayed visible, and was audible | Load, playback, visibility, and audio status |
| Attention/Continued Engagement | Whether viewers stayed or skipped under real-world distraction | Stay rate, skip rate, playback progress, with validation |
| Unaided Comprehension | What participants believe the ad was mainly saying | Verbatim recall and pre-defined codebook |
| Brand Attribution | Whether the message is linked to the correct brand | Unaided brand recall, confusion objects, and confidence |
| Judgment & Feeling | Whether the ad is relevant, credible, distinct, and what emotions it evokes | Separate rating scales and reasons |
| Next Step | Whether viewers are willing to search, learn more, or try | Stated intent and follow-up behavioral experiments |
The IAB and MRC 2025 attention measurement framework distinguishes between "opportunity to be seen" and actual attention, clarifying that attention is a supplementary signal that must be jointly validated with creative, placement, and outcomes—not treated as standalone ad effectiveness. Full completion may occur simply because skipping is unavailable; staying may be due to confusion; attention does not guarantee accurate comprehension or behavioral intent.
Forced Viewing Suits Diagnostics; Simulated Feed Is Closer to Natural Competition
Playing a 15-second video full-screen without the option to skip ensures exposure to the entire asset and is ideal for identifying which line is misunderstood—but it removes the challenge of earning attention that ads face in real media. A simulated feed preserves content competition and the ability to skip, more closely approximating natural viewing, yet controlling device, loading, scrolling, and surrounding content becomes more difficult.
Research need not permanently choose between the two. Early-stage work can use controlled exposure for diagnostic purposes, while mature versions are then placed back in the target environment for validation. Any results should specify display size, default sound state, skippable timing, playback count, and surrounding content; laboratory full-screen results must not be presented as in-feed performance.
1,000-Person 2×2 Experiment: Why the Liking Champion Did Not Win
The following is a hypothetical example and does not represent the products, ads, or industry benchmarks of Wenjuanpai. The study recruited 1,000 people from those who recently had relevant analysis needs but were unaware of the new feature. They were randomly assigned to either demo version A or story version B, and then further randomized into forced complete viewing or simulated feed, resulting in four groups of 250 each. The primary metric, defined pre-launch, was "core message correct unaided and brand attribution correct," with all assignees as the denominator.
| Metric | Demo Version A (N=500) | Story Version B (N=500) |
|---|---|---|
| Unaided core message correct | 315 (63%) | 235 (47%) |
| Unaided brand attribution correct | 310 (62%) | 345 (69%) |
| Both message and brand correct | 260 (52%) | 195 (39%) |
| Liking top-2 box | 285 (57%) | 375 (75%) |
| Intent to learn more top-2 box | 220 (44%) | 225 (45%) |
Story version B’s brand attribution was not poor, but many participants failed to recall the core benefit. Demo version A's marginal metrics were not remarkably high, yet more people linked the correct benefit to the correct brand simultaneously. One cannot claim a 62% successful communication from “63% message correct” and “62% brand correct,” as these could be two different sets of people. The joint metric must be computed per individual.
Version A’s joint valid reception exceeded version B’s by 13 percentage points. Based on a large-sample approximation for two independent random groups, the 95% confidence interval for the difference is roughly 6.9 to 19.1 points. Intent to learn more differed by only 1 point, with an approximate interval of −5.2 to 7.2, unable to support a clear difference. If the communication task is to establish a link between the new feature and the brand, the evidence supports A; if the task were to enhance emotional affinity for a mature brand, the study would need to be redesigned around relevance, emotion, and long-term brand metrics—not retroactively change the primary metric to allow B to win.
The Same Creative: How Much Is Lost Across Two Environments
| Joint Valid Reception | Forced Complete Viewing | Simulated In-Feed | Environment Gap |
|---|---|---|---|
| Demo Version A | 160/250 (64%) | 100/250 (40%) | −24 points |
| Story Version B | 125/250 (50%) | 70/250 (28%) | −22 points |
Both creatives declined markedly when placed in the simulated feed, confirming that forced viewing inflates message reception. Calculations here still use the randomly assigned 250 per group as the denominator, even if some scrolled away immediately; analyzing only completers would select for those initially more interested or capable of watching the full ad, undermining comparability. Loading failures can be reported separately with sensitivity analysis, but undesirable exposures should not be silently discarded.
The environment did not change the rank order between versions, but that provides no guarantee it will hold across all platforms. Short video, muted feeds, out-of-home screens, and landing pages differ in size, duration, audio, and user task—validation should occur in the target medium, not by treating "video assets" as a uniform environment.
Question Order Determines Whether You Measure Natural Memory or Prompted Answers
After exposure, first record technical and behavioral states, then ask in sequence:
- "In your own words, what did this content mainly try to tell you?" without showing benefit options;
- "Which brand do you think this content was for?" unaided first, then if necessary an aided check;
- Rate relevance, credibility, and distinctiveness separately, with a follow-up on key reasons;
- Record specific emotions and discomfort, avoiding a simple like/dislike scale;
- Only at the end ask about intent to learn more, search, or try.
Pew’s questionnaire design evidence shows that open versus closed wording, option content, and order all alter responses, while earlier questions provide context for later ones. If you first ask viewers to choose a highlight from “auto-summarization, time savings, accurate insights” and then ask the main message, the supposedly unaided comprehension has already been taught by the questionnaire. Randomizing brand lists, selling points, and emotion words only disperses positional bias; it does not restore contaminated natural recall.
Open-Ended Coding Must Allow for "Brand Correct, Message Wrong"
Before formally reviewing data, establish a core message codebook: fully correct, directionally correct but overclaiming, plot-only recall, miscomprehension, and unjudgeable; brand is coded separately as correct, competitor, category term, and unknown. Multiple themes per response are permitted, but "joint valid reception" equals 1 only when both necessary conditions are met. High-risk misunderstandings are reviewed by two coders, retaining original text and the rationale for disagreements.
For example, "AI will automatically make market strategy decisions for me" includes analysis-related words but exceeds the promise of "assisted summarization into traceable themes"; it must not be deemed a partial success to inflate the accuracy rate. Research is meant to check communication risks, not to adopt lenient grading standards for ad copy.
Pre-Launch Testing Cannot Answer True Incrementality and ROI
Laboratory or panel tests control for asset comparison but lack real bidding, frequency, placement, audience reach, and purchase consequences. A 45% intent-to-learn-more figure cannot be directly converted into click-through or sales. After launch, qualified exposures, brand searches, landing page tasks, and actual conversions should be tracked against hypotheses, preserving a randomly unexposed control group where possible.
Google Research’s Brand Lift methodology for online ads uses randomized experiments to estimate ad impact while addressing survey invitation and response, mismatches between expected and actual exposure, and biases from comparing only actors. Google Ads Brand Lift similarly treats ad recall, brand association, awareness, and consideration as distinct outcomes. Post-hoc comparisons of "those who saw the ad" versus "those who did not" are generally not causal evidence, as the two groups differ in platform activity, interests, and reachability.
Research Reports Must Allow Others to Reconstruct Test Conditions
AAPOR disclosure standards require making public the measurement materials, preceding context, study population, sample generation and recruitment, collection mode, dates, group sizes, weighting, data processing, and limitations. Creative tests should also attach asset versions or hashes, display environment, randomization unit, loading/viewing rules, primary metrics, and open-coding rules. Two creative thumbnails with a list of percentages cannot withstand independent verification.
Wenjuanpai can assist with target audience screening, random assignment of image/video versions, unaided open-ended responses, subgroup analysis, and the synthesis of extensive recall; the research team must still review asset rights, portrait permissions, sensitive or misleading content, verify AI coding, and connect pre-launch results to real-world incremental validation.
The conclusion this case supports is: "Under the current population and display conditions, the demo version better enables the same participant to correctly link the core message and brand; the story version is more liked but does not increase intent-to-learn-more. The demo version is recommended for small-scale incremental validation in the target medium, while emotional expression continues to be refined." It does not support "the demo version will certainly sell more," nor does it suggest "the story version is ineffective." Good creative research does not substitute for aesthetic debate—it locks together the communication task, the evidence, and the next validation step.