If you ask a group of job seekers whether they have been screened out because of where they earned their first degree, you learn what they perceived, remember, and are willing to report. You do not learn how often equally matched applications are actually treated differently. Estimating that second quantity requires sending identical materials through a real application process and observing the responses, which is what a correspondence audit does. Both measurements matter, and neither substitutes for the other.
Split the research question before choosing a design: to describe who feels discriminated against and what follows from that experience, use a survey or interview; to estimate how often matched applications are treated differently at a given stage, use a survey experiment or an audit. The three families of design differ in evidence strength, coverage, and ethical cost, so a report must state which layer its conclusion occupies — otherwise a perceived-incidence figure will be read as a discrimination rate.
What follows is a discussion of measurement choices and their limits. It is not a legal conclusion and does not concern any specific employer or individual. The experiments cited come from published research in the United States and comparable labour markets; transferring them elsewhere requires fresh verification, because screening methods, industry structure, and automated filters all change results.
A perception survey measures perception, not incidence
Perceived discrimination is a substantive construct rather than a complaint. A meta-analysis covering 134 samples found that perceived discrimination was significantly associated with worse mental and physical health, was accompanied by heightened stress responses, and correlated with more unhealthy behaviour and less healthy behaviour. Even though it is not an incidence rate, it remains worth measuring in its own right and carries real explanatory power.
The problem begins when a perception rate is converted into an incidence estimate. Three sources of variation sit inside those answers. First, attribution is ambiguous: the same outcome can be credited to a degree's origin, gender, age, experience, or interview performance, and respondents differ systematically in how they attribute. Second, self-presentation and social desirability shape answers to a normatively loaded question. Third, recall and reference periods bias the account toward the few episodes that were most memorable or consequential.
Employer self-reports are weaker still. In one study, employers who had unknowingly participated in an audit months earlier were contacted and given a vignette that closely matched the real scenario, then asked how likely they would be to hire. Their answers showed essentially no difference between majority and minority applicants, while the same employers' actual callback behaviour did differ. Asking how someone would screen a résumé measures the version of themselves they believe they should describe, not the screening they perform.
A perception survey can therefore support conclusions about experience, attribution, and consequences. Writing its result as an incidence rate oversteps the evidence.
A survey experiment manipulates one attribute cleanly, but yields a judgement
When the goal is to compare the direction and relative size of an attribute's influence, a survey experiment (vignette or factorial design) is often the efficient choice. Materials are built so that every version is identical except for the manipulated attribute, versions are randomly assigned to reviewers or members of the target population, and respondents judge whether they would invite the candidate, how they would score the profile, or which of two versions they prefer. Randomised version assignment inside a questionnaire is enough to carry this design.
Its advantages are concrete. Randomisation makes the between-group contrast interpretable. Several attributes — institution tier, internship experience, certification — can be manipulated at once so their interaction can be tested. It is cheap and fast, and it supports follow-up questions about reasons, which behavioural observation cannot provide.
Its limits are equally concrete. It measures judgements and intentions in a hypothetical scenario, not behaviour. Knowing they are being studied, reviewers are exposed to social desirability and demand effects. Because the gap between intention and action is well documented, a difference in average ratings cannot be converted into a difference in callback rates. Material realism determines external validity, and a visibly artificial template can itself shift judgements. The sensible role for a survey experiment is mechanism exploration and material refinement, not the final incidence claim. This matches the general distinction drawn in exploratory, descriptive, and causal research: capable of comparison and explanation, still short of behavioural evidence.
A correspondence audit measures behaviour, at the cost of covering one stage
An audit study sends fictitious applications to real job postings, matched on all observable characteristics except the attribute under study, and records whether a response arrives. The method is defined in the literature as a field experiment used to test discriminatory behaviour when survey and interview questions induce social desirability bias.
Its evidential strength rests on two features. What is observed is the employer's actual handling of an application rather than a description of that handling, and matched materials make "all else equal" an experimental control rather than an assumption.
The published magnitudes are worth knowing. A classic paired-résumé experiment sent several thousand fictitious applications to newspaper job advertisements in Boston and Chicago: applicants with white-sounding names received about 50 percent more callbacks, roughly ten résumés per callback against about fifteen, and a higher-quality résumé raised callbacks by about 30 percent for white names but far less for Black names. A later meta-analysis combined 28 field experiments covering 55,842 applications to 26,326 positions and found that, since 1989, white applicants received about 36 percent more callbacks than Black applicants and 24 percent more than Latino applicants, with no measurable change in discrimination against Black applicants across a quarter century — the authors are explicit that this holds "at least at the point of hire".
The costs and distortions are specific. Audits observe only stages that are externally visible; interviews, evaluations, and offers usually are not. They are expensive and attrition-heavy, and the required volume of applications is routinely underestimated. Pairing strategy and application density change the estimate: one analysis found that adjusting for applicant-pool composition raised measured discrimination by about 20 percent on average, because depositing several fictitious applications into the same pool alters how employers process that pool. Encoding identity in names is a noisy signal whose strength shifts measurements. Fictitious applicants do not face what real applicants face, and because the design involves concealment, ethics review, debriefing, and data-handling arrangements are part of the protocol rather than administrative overhead.
From callback to offer, the gap does not stay constant
Treating callback differences as the discrimination level of the whole process understates later stages. A meta-analysis of the 12 available field experiments that follow applications through to job offers, covering more than 13,000 applications, found that majority applicants received 53 percent more callbacks but 145 percent more offers than comparable minority applicants. The level of discrimination at the callback stage correlated only weakly with the level at the offer stage.
That fact changes reporting practice. Different stages are not repeated measurements of one number; they can be weakly related segments. A report must therefore place the stage, the occupations and regions covered, and the observation window next to every conclusion, instead of asserting that "discrimination exists in hiring".
A worked example: how many applications are needed
The following power illustration uses entirely hypothetical figures to show the most common failure in audit planning: an observed difference that looks directional but rests on too little data to support a claim. Assume two independent groups, one application as the unit of analysis, a two-proportion approximation, a two-sided 0.05 significance level, 80 percent power, and no adjustment yet for the correlation between multiple applications to the same employer.
| Applications per group | Observed callback rates | Difference | Approximate 95% interval | Claim supported |
|---|---|---|---|---|
| 500 | 10.0% vs 7.4% | 2.6 pp | about −0.9 to 6.1 pp | No, the interval crosses zero |
| 2,000 | 10.0% vs 7.4% | 2.6 pp | about 0.9 to 4.3 pp | Yes, the interval excludes zero |
| about 1,844 | 10.0% vs 7.4% | 2.6 pp | — (required for 80% power) | Floor for the design stage |
| about 13,495 | 10.0% vs 9.0% | 1.0 pp | — (required for 80% power) | Smaller gaps cost far more |
Two lessons matter more than the specific figures. The same 2.6 percentage-point gap crosses zero with 500 applications per group and excludes zero with 2,000; with too little data, a result can appear positive or null, and its direction is unstable. And as the gap to be detected shrinks, the required volume grows quickly — moving from 2.6 percentage points to 1.0 requires nearly seven times the sample.
A real audit needs more than this table suggests, because repeated applications to the same employer are not independent and materials inside the same pool influence each other, which widens the interval beyond the independent-case approximation. Conversely, when the stage being studied does show a large gap — as in the offer-stage result above — a few hundred applications per group can be enough to distinguish it. The planning sequence should run from the smallest meaningful gap, to the required sample, to the posting volume and geographic scope.
Nine items an audit design must record
A design that can be recomputed and challenged documents at least the following. The first six determine what is actually being measured; the last three determine how far the conclusion can travel.
| Item | What to record | Frequent source of distortion |
|---|---|---|
| Manipulated attribute | Which attribute is varied and how it is operationalised in the materials | Readers see an attribute label but not the manipulation strength |
| Material equivalence | Field-by-field checks that everything else matches, and who verified it | "Roughly identical" lets experience, skills, or formatting leak in |
| Pairing strategy | Paired, unpaired, or multiple applications per pool; pool-composition adjustment | Pooled submissions shift employer processing and bias the estimate downward |
| Outcome definition | Any response, interview invitation, next round, or offer; coding rules | Stages are merged into a single "responded" variable |
| Unit and denominator | Per application or per position; treatment of invalid submissions | The denominator changes between analyses, so rates cannot be recomputed |
| Sampling frame | Source of positions and regions, occupational mix, documented gaps | Only easy-to-reach roles are used, but conclusions are written for the sector |
| Sample and power | Smallest detectable gap, power assumptions, clustering treatment | Precision is inferred after seeing the observed difference |
| Time window | Application period, concurrent market and policy events, seasonality | A recruitment-policy change during the window is attributed to the attribute |
| Ethics and disclosure | Concealment and debriefing arrangements, data minimisation, platform agreements | Ethics review is treated as paperwork rather than part of the design |
When not to run an audit, and what surveys still do better
- Deciding whether one specific person was discriminated against. Audits produce group-level differences; inferring an individual case from an average is a classic ecological inference error.
- Producing a legal or administrative finding. Whether discrimination was unlawful is decided by labour dispute, inspection, and judicial processes; experimental evidence is one input, not the ruling.
- Where ethics or compliance approval is unavailable. Fictitious applications consume recruiters' time, involve concealment, and require storage and deletion arrangements for submitted data.
- Answering why. Audits measure outcome differences; explaining mechanisms requires survey experiments and interviews alongside them.
- Where the budget cannot support the power requirement. In that case, run a survey experiment and label the result as intention evidence.
Surveys and interviews remain irreplaceable in four areas: measuring perceived experience and its consequences; documenting screening processes, policies, and training inside organisations; capturing how people judge appeal channels and fairness; and tracking the same population over time. Reporting these alongside experimental evidence is more useful than replacing one with the other.
How this division of labour maps onto WeJot
The survey-experiment stage can run inside a questionnaire: use randomised version assignment to distribute material variants, collect reasons through conditional logic or option references, and compare versions after export. Before comparing means across versions, confirm that the items mean the same thing in both groups using the measurement invariance check.
For the perception stage, avoid collapsing experience, attribution, and consequences into one yes-or-no item; measure them as separate blocks. The risk of inflated relationships when causes and outcomes are collected in a single instrument is covered in common method bias. When the question is how someone reasoned at the time, combining the survey with AI interviews adds more than further closed questions. Finally, record all three evidence types in one methods statement, labelling the layer each one measures so that a perception share is not read as an incidence rate.
The real-application and response-recording stage sits outside the platform. It needs live recruitment channels, compliance arrangements, and an external observation mechanism, and WeJot neither performs nor replaces it. Doing the three stages the platform can support properly, and handing the rest to the right design, is the boundary this article draws.
Sources
- Bertrand & Mullainathan (2004), American Economic Review 94(4) — Are Emily and Greg More Employable Than Lakisha and Jamal? (matched fictitious résumés: about 50% more callbacks for white-sounding names; roughly 10 versus 15 résumés per callback; about 30% higher callbacks from a stronger résumé for white names)
- Quillian, Pager, Hexel & Midtbøen (2017), PNAS 114(41) — Meta-analysis of field experiments shows no change in racial discrimination in hiring over time (28 experiments, 55,842 applications, 26,326 positions; about 36% more callbacks for white versus Black applicants and 24% versus Latino applicants since 1989)
- Quillian, Lee & Oliver (2020), Social Forces 99(2) — Substantial Additional Racial Discrimination after the Callback (12 studies reaching offer outcomes, over 13,000 applications; 53% more callbacks but 145% more offers; weak correlation between stages)
- Gaddis (2018), Audit Studies: Behind the Scenes with Theory, Method, and Nuance — An Introduction to Audit Studies in the Social Sciences (when audits apply: testing behaviour when survey and interview questions induce social desirability bias; paired and unpaired designs; pool-composition adjustment raising measured discrimination by about 20%; employer vignettes diverging from observed callbacks)
- Bertrand & Duflo (2016), NBER Working Paper 22014 — Field Experiments on Discrimination (definitions of audit and correspondence studies: all observable characteristics matched, one attribute varied; the gap between fictitious applicants and real participants, and the associated ethical questions)
- Pascoe & Smart Richman (2009), Psychological Bulletin 135(4) — Perceived discrimination and health: a meta-analytic review (134 samples; perceived discrimination associated with worse mental and physical health and with stress responses, supporting it as a distinct measurable construct)