Place survey answers next to orders, attendance, disbursement, or administrative records and a share of them will not agree. Those differences are not automatically evidence that respondents lied, and the record is not automatically the truth. More often, the two datasets are not measuring the same thing, the linkage process itself manufactures part of the disagreement, and only the remainder is about individual reporting error or genuine change. Separating those three layers and producing evidence for each is more useful than arguing about which source is more trustworthy.
Assess linkage quality before interpreting any disagreement rate. Without a separately reported linkage rate, a profile of the unmatched cases, and some estimate of false matches and missed matches, the agreement and disagreement rates you compute are not yet interpretable. Administrative records are not a gold standard — they carry their own definition, coverage, and timing problems — and probabilistic linkage will fold some genuinely different records together, quietly erasing disagreement.
The discussion below is about data management and measurement. It does not address any specific organization or individual. Except where a source is cited, every number is illustrative data constructed to demonstrate arithmetic; it is not an empirical benchmark and cannot be used to infer any real rate of compliance or violation.
Three layers behind a single disagreement
When a case shows "the survey says the benefit was received but the record does not," or "the record shows usage but the survey says no," locate the problem in one of three layers: the record side, the linkage side, or the individual side. Each needs different evidence and a different action.
| Layer | Typical pattern | Evidence you owe the reader | Action |
|---|---|---|---|
| Record side: definition and coverage | The record captures whether eligibility, filing, or a payment was booked at a point in time, not whether the person knew about it or actually received it; it covers only registered or correctly captured entities. | The record's definition document, coverage, timestamp fields, unit and currency. | Align the concepts first; change the wording of the conclusion if needed, never the data. |
| Linkage side: misses and false matches | One person is split into two records, or two people are merged into one. | Linkage keys, matching rule and threshold, and either estimates or a manual audit of misses and false matches. | Quantify and disclose linkage error; test sensitivity to the threshold. |
| Individual side: real difference and reporting error | The record is right and the respondent misremembered, placed the event in the wrong period, or understood a different question; a lagging or missing record can also make the self-report correct. | Recall period, construct definition, sensitivity of the item, and the record's own posting date. | Fix the question before reaching for statistics, and state where the error direction is unknown. |
These layers are a sequence to rule out, not a menu to pick from. Skipping the first two and debating whether respondents were honest almost guarantees that definition problems and matching problems will be charged to individuals.
Layer one: does the record measure what you asked?
Alignment has four parts: the time unit (calendar month, billing period, or calendar year, and how cross-period events are assigned); the entity unit (person, household, establishment, or store); the event definition ("filed" versus "received," "ordered" versus "completed," "issued" versus "posted" are not the same event); and the reference point in time, since the event date, posting date, and cutoff date can fall in different periods. Miss any one and the disagreement rate mixes different things.
Coverage matters just as much. Records usually contain only entities that registered, took a specific action, and were captured correctly. Absence from a record does not mean the event did not happen; it may mean the person never entered that system. Reading "not in the record" as "did not happen" converts a coverage gap into an apparent individual difference.
Benefits, subsidies, and policy data are the classic case: a payment in the system does not mean the person knew, remembered, or understood it, and a respondent saying "never heard of it" does not overturn a payment that was made. The two statements measure different constructs — recorded receipt versus awareness and perception — and neither should be used to grade the other. If awareness is the research target, the self-report is the construct itself, and the record is only there to check whether receipt occurred.
Layer two: linkage manufactures invisible disagreement
Linkage is the act of deciding that records in two datasets belong to the same entity. Exact matching on a national ID or a single primary key looks simple, but de-identification, duplicate IDs, name changes, and inconsistent coding across systems often make it unavailable. In practice, teams combine name, date of birth, region, or phone number into probabilistic matching: each candidate pair gets a probability of being the same entity, and a threshold decides whether to link.
Probabilistic linkage carries two errors at once. A miss (false non-match) sends a genuinely matching pair into the unmatched pile. A false match is more insidious: it welds two unrelated records together, generating spurious disagreement in the comparison, or — just as often — erasing real disagreement. In a study linking the 2014 National Hospital Care Survey to the National Death Index, the U.S. National Center for Health Statistics noted that probabilistic linkage leaves uncertainty about whether any given pair is truly a link, and that this uncertainty propagates into the estimates built on the linked data; they used re-sampling to quantify that propagation. For survey work the implication is direct: a single point estimate after linkage is not enough; you must report how sensitive it is to the linkage decision.
| Error | How it arises | Effect on conclusions | How to check |
|---|---|---|---|
| Miss (false non-match) | Threshold too strict; key fields missing or mistyped; inconsistent coding across systems. | Effective sample understated; "unmatched" misread as "does not exist." | Sensitivity analysis across thresholds; manual audit of a sample of unmatched cases. |
| False match | Threshold too loose; low-discrimination key combinations (same name and birthday); region or institution fields too coarse. | Creates or erases differences; systematically inflates the agreement rate. | Estimate the false-match rate or audit a small sample; report how key variable distributions shift before and after linkage. |
A worked 1,000-record example
The linkage below is entirely illustrative; the numbers exist to show the arithmetic. Questionnaires from 1,000 respondents are matched to 1,000 backend records: 900 pairs link and 100 do not, a linkage rate of 90%. Among the 900 linked pairs, 720 agree on the field and event and 180 disagree. Computed on the linked sample alone, the disagreement rate is 180 ÷ 900 = 20%.
But whether the 100 unmatched pairs agree is unknown. Switch to the conservative convention that every unmatched pair counts as a disagreement, and the overall rate becomes (180 + 100) ÷ 1,000 = 28%, eight percentage points higher. The same data, a different denominator, an eight-point difference — which is why the denominator and the treatment of unmatched cases must be fixed in advance.
Now see how linkage error washes disagreements away. Suppose a manual audit estimates that 20 of the 900 pairs are false matches. Those two records should never have been compared; counting them as agreement or disagreement both misrepresent reality. If those 20 pairs happened to be counted as agreement, the true agreeing pairs are 720 − 20 = 700, and on 900 comparable records the agreement rate is 700 ÷ 900 ≈ 77.8%, not 80%. The gap is small, but its direction is fixed: false matches tend to manufacture agreement and make the data look cleaner than it is.
Finally, match bias. Suppose a key characteristic (for example, "moved home in the past year") is true for 34% of all 1,000 people: 30% among the 900 matched and 45% among the 100 unmatched. Analyzing the matched sample alone understates the characteristic by four points (30% versus 34%); ignoring the unmatched and calling the sample representative makes the bias larger. The lower the linkage rate and the more distinctive the unmatched cases, the less this bias can be ignored.
A high linkage rate is an entry point, not a clean bill of health
A high linkage rate does not mean there is no bias: even at 95%, if the remaining 5% differ systematically on key variables, comparisons built on the matched sample still shift. Treat unmatched cases as a separate sample rather than deleting them as missing.
- Profile the unmatched. Compare matched and unmatched groups on demographic and behavioral variables the survey already contains, not on variables that exist only in the record — the unmatched have none of those.
- Report at least three denominators. Linked sample only; the conservative convention that treats unmatched as disagreement; and a version that treats unmatched as its own stratum. Side by side, the reader can judge how stable the conclusion is.
- Run a threshold sensitivity analysis. Vary the matching threshold and see whether the direction survives. If the conclusion flips with the threshold, it is a product of the linkage method, not a fact about the population.
Four decision rules for checking self-reports against records
| Situation | What to do | What to write |
|---|---|---|
| The record is the authoritative file for whether an event occurred, but it under-registers. | Use the union of record and self-report as an upper bound and the record alone as a lower bound, or apply capture–recapture reasoning to estimate under-registration; do not assume the record is complete. | State the assumption behind each bound, and who the union over-counts and the record under-counts. |
| The research target is awareness, perception, or understanding. | The self-report is the construct; the record only checks receipt. Never use the record to overrule the self-report. | Report receipt and awareness as two separate variables, and treat their disagreement as a finding in itself. |
| The disagreement itself is the outcome, such as filing accuracy or truthful usage. | Estimate the false-match and miss rates first, then correct the disagreement rate for measurement error or fold linkage uncertainty into an interval. | Present corrected and uncorrected values together, noting the false-match assumption the correction relies on. |
| The data are sensitive: income, health, benefits, subsidies. | Weigh ethics, authorization, and re-identification risk first; if aggregate data can answer the question, do not perform individual-level linkage. | Document the authorization, why this granularity was chosen, the retention period, and where the linkage keys are stored and destroyed. |
Write the linkage so someone else can reproduce it
However simple the conclusion, a methods note is only reproducible if it answers these questions:
- Which fields were used as linkage keys, exact or probabilistic, at what threshold, and why;
- What the linkage rate was, and how unmatched cases differ from matched ones on observable variables;
- What checks were run on misses and false matches, including any manual audit and its results;
- How definitions, coverage, and timing were aligned between the two sources, and on what basis a difference was judged to be definitional;
- Which denominators were used and excluded, and what the sensitivity analyses showed;
- Who authorized the linkage, how long the data are kept, and where the keys live and how they are destroyed.
When not to link at all
- No legal basis, authorization, or ethics review, especially for individual-level matching across organizations.
- The sample is too small to estimate misses and false matches, so linkage error exceeds the effect you set out to measure.
- The record's covered population is badly misaligned with the survey population, leaving the linked sample representative of no defined population.
- The goal is to judge a specific individual from the record (whether they violated a rule or reported truthfully); that conclusion exceeds what survey evidence can support and belongs in the relevant verification process.
Where WeJot fits in the chain
In WeJot, records and self-reports can be separated from the design stage: put behavior items ("did it happen, was it received") and perception items ("do you know about it, how do you rate it") in different question groups so a single option does not carry two meanings; use a stable anonymous identifier rather than a name field as the future linkage key; and keep timestamp, source, and status fields on export so definitions can be aligned with an external record.
After collection, the codebook must record linkage keys, missing-value codes, and derivation rules so they can be recomputed; cleaning follows the raw-response-to-analysis-dataset sequence and treats "unmatched" as an explicit state rather than blank or zero; missing-data handling distinguishes "no record exists" from "the record's field is empty." If individual-level merging is planned, audit combination-field risk first with the anonymity and re-identification review and decide whether a coarser grain is required. Finally, write the linkage rate, the unmatched profile, and the sensitivity analyses into the methods note so the conclusion traces back to a specific definition.
Note that individual-level matching with external systems happens outside the platform: it requires data authorization, compliance arrangements, and cooperation from the other system. WeJot's role is to get the survey-side fields, definitions, and exports right so that linkage can be reproduced outside the platform — not to replace authorization or compliance judgment.
Sources and method basis
- Groves & Lyberg (2010), Public Opinion Quarterly 74(5):849–879 — Total Survey Error: Past, Present, and Future (the total-error framework splits error into measurement and representation sides; linkage error is an additional source beyond both)
- Sakshaug & Antoni (2017), Total Survey Error in Practice, 557–573 — Errors in Linking Survey and Administrative Data (linkage error is treated as its own component of total survey error: matching rules and linkage quality change estimates, so the linkage rate alone is not enough)
- National Center for Health Statistics, IJPDS 3(5) (2018) — Simulation Approach to Assess the Precision of Estimates Derived from Linking Survey and Administrative Data (linking the 2014 National Hospital Care Survey to the National Death Index; probabilistic linkage leaves uncertainty about each link, and a re-sampling approach quantifies how that uncertainty propagates to estimates)
- Kreuter, Sakshaug & Tourangeau (2016), International Journal of Public Opinion Research 28(1):142–152 — The Framing of the Record Linkage Consent Question (the wording of the consent question changes consent rates, so the linked sample is not a random subset)
- Sakshaug & Vicari (2018), Journal of Survey Statistics and Methodology 6(1):46–71 — Obtaining Record Linkage Consent from Establishments (in an establishment sample, question placement changed both consent rates and the resulting bias)
- Bound, Brown & Mathiowetz (2001), Handbook of Econometrics — Measurement Error in Survey Data (validating self-reports against external records shows substantial, non-random reporting error; external records serve as a check but carry their own error)
- National Academies of Sciences, Engineering, and Medicine (2023) — Toward a 21st Century National Data Infrastructure: Enhancing Survey Programs by Using Multiple Data Sources (recommends assessing each source's quality and bias separately and treating linkage and privacy protection as prerequisites)