Whether a report someone hands you can support your own conclusions does not depend on the reputation of the lab or the formality of the cover page. It depends on four questions you can actually answer: who chose the samples, whose interests the testing body is independent from, whether the method and raw records can be reviewed, and how far the conclusion generalizes. If all four can be answered, you can cite the report as evidence. If any one of them cannot, the report is a lead at best, and belongs in a “to be verified” section rather than in your findings. The order matters too, because the last three checks only matter after the first: if the samples were not drawn by a sampling plan, no amount of methodological review extends the conclusion beyond the items that arrived at the bench.
Three kinds of reports answer different questions
Reports are often lumped together as “test results,” but who decides the samples and who carries the responsibility differ sharply. In a sampling inspection, the executing party draws items from the population under a pre-specified sampling plan, so coverage can be evaluated. In commissioned testing, the client selects the items and the lab only reports whether those items meet the inspected criteria. In an evaluation run by an institution or a media outlet, both method and specimens are chosen by the evaluator, and credibility depends largely on how much methodological detail is published. All three can be useful, but you need to know which one you are holding.
The responsibility boundary for commissioned testing is explicit in Chinese regulation. The Order of the State Administration for Market Regulation No. 39, the Measures for the Supervision and Administration of Inspection and Testing Institutions, states that where an institution tests samples submitted by a client, the report is responsible for whether those samples comply with the inspected items, while “the representativeness and authenticity of the submitted samples are the client’s responsibility.” Two things are being separated: the lab owns the result for “these items, these criteria,” and the client owns the question of whether those items represent the production batch. Collapsing the two is the most common overreach in citation.
Check one: who decided the samples, and does the report say so?
This check answers one question: does the conclusion cover a population, or just the items on the bench? The answer is usually readable from the report itself.
- Provenance. Does the report state whether samples were submitted by the client, purchased independently by the evaluator, or drawn on site by a sampler?
- Description. Are name, model, batch or serial number, quantity and received condition complete? Without quantity, no “pass rate” statement later in your material can be recomputed.
- Sampling plan. A sampling inspection should state the sampling basis, location and executing party. A report with a result but no account of how items were drawn cannot have its coverage evaluated.
The rule is blunt: coverage never exceeds the samples described. If the report covers three items, what you may say publicly is that those three items met the inspected criteria, not that the product meets them. That distinction matters most when the claim travels to customers, platforms or regulators, because the claim is where liability actually lives.
How much risk does “everything passed” rule out?
An all-pass result can be quantified directly with the binomial distribution. If all n tested items pass, the exact two-sided one-sided upper 95% confidence limit for the true failure rate behaves as follows:
| Items tested, all passing (n) | Upper 95% limit on the true failure rate |
|---|---|
| 3 | 63% |
| 10 | 26% |
| 20 | 14% |
| 30 | 9.5% |
| 59 | 4.9% |
| 100 | 3.0% |
| 299 | 1.0% |
Read the table backwards. Decide the risk ceiling you can accept, then ask whether the report tested enough items to support it. If you want to state that the failure rate is no higher than 5%, an all-pass result needs roughly 59 items. If the report tested three, its upper limit is still 63%: it has barely moved the ceiling at all. The familiar “rule of three,” where the all-pass upper limit is about 3/n, works well at larger n and is optimistic at small n. For 0/3 the approximation returns 100%, while the exact limit is 63%. At the other end, an all-pass report on 100 items looks much more solid and still only reaches about 3%.
That yields a hard rule for wording: the report should say that no failures were found among the n items tested. “Not found” is not “does not exist.” If the sample size was set by someone else and cannot be changed, the qualifier must survive into your citation. When you do get to set the sample size yourself, the logic matches how many respondents a survey needs, except that the event here is a defect rather than a survey option; and if a proportion is reported at the end, the interpretation in confidence intervals for survey results applies as well.
Check two: whose interests is the testing body independent from?
Regulation requires independence: inspection and testing institutions and their personnel should be independent of the interested parties involved in the report, free from factors that could interfere with technical judgment, and obliged to publicly declare their compliance, independence and impartiality. But “the lab is independent” does not mean “the conclusion is unaffected by the client.” Within the rules, a client still chooses which item to submit, in what condition, to which institution, at what moment, and those choices alone can change the answer. Systematic review evidence on industry sponsorship and research outcomes finds an association between funding source and the direction of conclusions. That is not an accusation of fraud against any institution; it means “who pays, and who submits the samples” is itself information you must record.
The list to verify is short but must be documented: whether the institution has published a declaration on acting independently and impartially; whether the report discloses the client and whether any items were subcontracted and to whom (the regulation requires the client’s prior consent and an explicit note in the report); whether the cited accreditation and scope of capability cover the items you care about; and whether the samples came from the client or were purchased by the institution. If any of the four is missing, downgrade the citation.
Avoid the mirror-image error as well: independence is necessary information, not proof that a conclusion is correct. A genuinely independent institution can still produce a report that cannot support your decision, because the samples were unrepresentative, the method was unsuitable, or the measurement uncertainty was too large.
Check three: can the method and raw records be reviewed?
Reviewability is what turns a third-party conclusion into evidence. Two provisions can be borrowed directly from regulation. First, reports must use compliant wording and state the standards and other technical bases; raw records and reports must be archived and retained for at least six years. Second, “data or results that are erroneous or cannot be reviewed” is itself listed as a form of untruthful report. A report that cannot be reviewed already falls into a category regulators treat as problematic.
At minimum you should be able to verify: the report number and signatory information, the method standard and its version, the sample description and received condition, the data or reviewable attachments behind the conclusion, and, where relevant, ambient conditions and equipment calibration status. Whatever is missing should be named as unchecked in your citation, rather than assumed to be fine.
Software and algorithm evaluations carry an extra layer of uncertainty, because results move with data splits, parameter initialization and hyperparameter choices. Benchmarking research shows this variance alone is large enough to flip which algorithm appears better. So when you see a single-run comparison, ask for the number of repetitions and the spread before crediting words like “leading” or “superior.”
Reviewing does not always require redoing the test. More often it means requesting a minimum set of verifiable information, and when the stakes justify it, arranging an independent repeat. The large-scale multi-lab replication effort in psychology is cited so often because it wrote a common-sense rule into evidence: independent replication is a routine step in confirming a result, not a statement of distrust. Applied to commercial reports, the question becomes whether you could see a similar result again. When such conclusions go into the methods section of a research report, this is precisely where sources and limitations belong.
Check four: who does the conclusion generalize to?
The first two checks ask whether a result is credible; this one asks who it is about. Standard evidence-grading frameworks treat indirectness and imprecision as separate reasons to downgrade: the tested object differs from the object you actually have to decide about (different batch, model, version, population, usage scenario or time point), or the estimate itself is too imprecise. The framework was built for medical evidence, but both judgments hold for commercial reports.
In practice it is three consecutive questions: what scope does the report itself claim; where do your batch, version or population differ from it; and how large is that gap relative to the threshold you have to decide against. If the gap is small and the threshold forgiving, cite the report with its conditions attached. If the conclusion will enter a public commitment, touch safety or compliance, or land right at the threshold, the third-party report is not enough and a small self-run verification is warranted.
When you do verify, the key is to state the target as a number: fix the risk level you want to rule out, derive the sample size from the table above, and write down the sampling, judging and recording rules before running anything. Whether sampling is random, whether judging is blinded, and how nonconforming items are handled determine whether the result can be reviewed later. For screening decisions in a population, the logic in controlling false inclusions and false exclusions in screeners applies: every decision rule creates both kinds of error at once.
Four sentences that carry the limitations into your report
What you write after checking is easier to distort than the checking itself. The four sentences below can be reused, and the order is worth keeping.
- Source. “The data come from [report name and number] issued by [institution] on [date]; samples were [submitted by the client / drawn by a sampler / purchased by the institution], [n] items in total.”
- Scope. “The report covers [samples / batch / model / criteria]; this citation is used only for [specific purpose].”
- Unverified. “[Sampling plan / raw data / subcontracting information] was not provided in the report and has not been independently reviewed here.”
- Strength. “With these limitations, this conclusion is [directly citable / citable with conditions / indicative only], and should be used publicly only after [follow-up action].”
The strength level can be fixed by three rules so that judgment does not depend on the day: citable directly when sample provenance, method basis and data attachments are present and the tested object matches your target; citable with conditions when one of the three is missing; indicative only when provenance and method basis are both absent and nothing but a conclusion and a cover page remains. The point of the rules is that the same report receives the same rating from different people, instead of the discussion drifting into whether the institution “feels” trustworthy.
When this appraisal does not apply
- Statutory mandatory inspection. Sampling plan, items and acceptance rules are set by regulation or standard, so the check becomes authenticity and completeness of the report rather than whether to believe it.
- Leads in an exploratory phase. Early on, an unappraised report is enough to generate hypotheses. The mistake is writing the hypothesis as a conclusion, not using the report.
- Low-stakes internal decisions. When only the team uses the result and errors can be corrected immediately, the check can be lighter, but source, sample size and limitations still need a written trace, or nobody can recompute it months later.
One case deserves separate mention: conclusions touching personal safety, medicine, law or regulatory compliance are not solved by grading citation strength. They require a qualified professional judgment. This checklist only helps decide whether a dataset may enter your material; it is not professional advice and is not a judgment about any specific institution or case.
Where WeJot fits when you need your own evidence
A third-party report answers whether the items on the bench complied at that moment. Most decisions actually want to know what users encountered, and those are different kinds of evidence. WeJot covers the second: running a self-collected study with the target population, using conditional logic so different groups see different question sets, and quotas to hold demographic or version strata stable so subgroup comparisons rest on a common sample structure. For respondents who need to explain what happened, an AI interview can capture the specific sequence and failure context, after which open text can be coded into countable categories.
Its limits should be as clear as its use. A self-collected study yields reported experiences and evaluations, which can cross-check how many people in a population encountered a problem; it cannot replace physical testing of a batch and cannot establish whether a given item passed. When both kinds of evidence appear in one document, label source, sample size, target object and limitations separately; following writing research into an actionable insight report, state what the evidence supports before stating the conclusion.
Sources
- Order No. 39 of the State Administration for Market Regulation, Measures for the Supervision and Administration of Inspection and Testing Institutions (State Council Gazette 2021 No. 16, effective 2021-06-01, accessed 2026-10-17) (basis for sample responsibility for submitted items; independence from interested parties and public self-declaration; client consent and disclosure for subcontracting; report wording and technical basis; retention of raw records and reports for at least six years; “data or results that are erroneous or cannot be reviewed” counted as untruthful reports.)
- Hanley & Lippman-Hand (1983), JAMA 249(13):1743–1745 — If nothing goes wrong, is everything all right? Interpreting zero numerators (origin of the rule of three for all-pass results and the caveat that it behaves poorly at very small n.)
- Clopper & Pearson (1934), Biometrika 26(4):404–413 — The use of confidence or fiducial limits illustrated in the case of the binomial (exact binomial interval used for the limits in the table: 63.2% for 0/3, 4.9% for 0/59, 1.0% for 0/299.)
- Lundh, Lexchin, Mintzes, Schroll & Bero (2017), Cochrane Database of Systematic Reviews 2:MR000033 — Industry sponsorship and research outcome (systematic review finding an association between sponsorship and the direction of conclusions; used here to argue that funding and sample submission are information to record, not an accusation.)
- Guyatt, Oxman, Vist, Kunz et al. (2008), BMJ 336(7650):924–926 — GRADE: an emerging consensus on rating quality of evidence and strength of recommendations (evidence-grading framework treating indirectness and imprecision as separate reasons to downgrade, borrowed here to judge generalization.)
- Bouthillier, Delaunay, Bronzi et al. (2021), arXiv:2103.03098 — Accounting for Variance in Machine Learning Benchmarks (variance from data sampling, parameter initialization and hyperparameters materially affects benchmark comparisons; basis for asking about repetitions and spread in software and algorithm evaluations.)
- Open Science Collaboration (2015), Science 349(6251):aac4716 — Estimating the reproducibility of psychological science (large-scale multi-lab replication; used to frame independent replication as a routine step rather than a sign of distrust.)