In this article

How much evidence supports this pair?

Imagine a study with 200 registered participants and 100 comparison pairs. It would be easy to describe it as a large examiner study. Yet if most participants receive ten pairs, and completion is uneven, one pair might attract forty submitted examinations while another receives three. The participant total tells us little about either item.

This hypothetical example exposes a practical question: what is the denominator behind a statement of agreement? We might be counting people, assignments, completed comparisons, decisions or pairs of responses. Each can be useful, but each describes a different part of the study. Multiplying participants by the available tasks does not reveal how many examinations actually took place.

A useful account starts at the item. Which examiners assessed this material, under which instructions, and how far did they proceed? It then distinguishes an allocation from work begun, a submitted stopping decision from a completed evaluation, and a first encounter from later work on familiar material.

The headline sample is not the per-item sample

Ulery and colleagues’ 2011 fingerprint study involved 169 latent-print examiners and a pool of 744 latent–exemplar pairs. Each examiner was assigned approximately 100 pairs. Each pair was examined by an average of 23 participants. These numbers describe three different levels of the design; the study did not provide 169 assessments of every pair.[1]

The later white-box markup study gives another useful example. Its 170 examiners contributed 3,730 responses, with a median of twelve examiners assigned to each pair. Comparison-phase results came from the subset in which neither impression was assessed as having no value. The supporting data report a median of ten comparison markups per pair. The available denominator changed with the stage and the question. These studies used selected material and controlled tasks. Their results do not estimate the complete operational process, with all of its casework context and safeguards.[3][1]

This is one reason that the Ulery research family has been foundational to PROFAS. Studying the examination requires a connection between the shared material and the actual contributions made to it. A participant count cannot substitute for that connection, however substantial the overall study may be.

A tyre-study technical report encountered during the current PROFAS project illustrates the distinction in another discipline. Of 39 registered examiners, 17 completed at least one comparison. Seventy-seven of the 80 comparison sets received responses, ranging from one to six per set. The report describes 238 returned trials across those sets. Participant uptake, coverage per set and total responses describe different aspects of the collection.[4]

Keep the routes through the task visible

No-value and inconclusive decisions belong in that account. In the 2011 study, a no-value decision ended the task before the exemplar was presented. A comparison could instead lead to individualization, exclusion or an inconclusive outcome. The authors distinguished rates based on all presentations from rates based on particular subsets of comparisons. Removing a category changes what the resulting rate describes.[1]

A low error rate also needs its conclusion rate alongside it. An examiner who rarely reaches a conclusion may make few errors while providing limited resolution of the assigned material. This does not make every inconclusive decision inappropriate. It means that accuracy, willingness to conclude and the suitability of the material need to be considered together.[1]

An unfinished allocation is different again. It does not establish that the examiner considered the material inconclusive, unsuitable or difficult. A study should retain what is known about missing work and explain its handling, rather than silently adding it to a decision category. The denominator should follow the recorded event, with uncertainty about absence kept explicit.

A second encounter answers a different question

The 2012 retest study distinguishes repeatability, meaning consistency within an examiner, from reproducibility, meaning agreement between examiners. Seventy-two participants from the original study were reassigned 25 previously encountered pairs after approximately seven months. That retest was designed to investigate repeated decisions; it was not a new independent sample of examiners encountering entirely unfamiliar material.[2]

The distinction carries into verification. Two people can work separately while still responding to the same difficult item. In the 2011 study, false exclusions varied by image pair, so the authors rejected simply multiplying individual error rates to estimate joint false exclusions. Independent working conditions do not make outcomes statistically independent of the material.[1]

A research dataset therefore needs encounter history as well as examiner identity. Revisions, repeated attempts and reused verifier baselines should remain linked to their earlier work. Counting them as additional independent opinions would change the apparent weight of the evidence without adding the intended independent observation.

What PROFAS preserves for the analysis

A PROFAS study connects a defined protocol, selected material and individual assignments. The submitted examination retains its material and protocol basis, accepted records and stage. A reasoned stop retains its disposition and explanation. Later successor work remains linked to the original encounter, while declared prior exposure is carried into the research record.

Structured extraction groups contributions by recorded conditions, including material, protocol, module, stage, exposure and whether the encounter is an original or successor. These groupings make distinctions available for analysis. They do not certify that participants were free from every outside influence, or decide which observations belong in a particular statistical model.

Exports contain bounded selections of submitted records and their histories. A longitudinal study needs a collection and analysis plan that retains those packages and reconciles contributions across them under explicit comparison rules.

Accepted contributions remain attached to an examination and its recorded development. Research counting should preserve that encounter history rather than treating every revision as a new assessment.

Turn the research aim into a collection plan

The website’s study dashboard demonstrates an aggregate coverage concept: how a collection might look with a target of 100 independent assessments per pair over six to twelve months. Its figures are illustrative. It is not a completed study or a live dashboard of examination-app participant records.

Such a target requires recruitment, examiner time, suitable material and an agreed protocol. Software can retain allocated and submitted work; it cannot supply the people or establish that an ambitious collection window will be met. Pair-level progress matters because a high overall total can coexist with sparse coverage of particular material.

The final study report should make those differences legible. Readers need the per-item contributions, decision definitions, repeat conditions and missingness that support the analysis, together with the limits of the selected material. Agreement becomes informative when its denominator and conditions can be inspected.

FOLLOW THE EVIDENCE

Sources & further reading

The original research behind this article.

  1. Ulery, B. T., Hicklin, R. A., Buscaglia, J., and Roberts, M. A. (2011). Accuracy and reliability of forensic latent fingerprint decisions. Proceedings of the National Academy of Sciences, 108(19), 7733–7738.(opens in a new tab) Back to the article
  2. Ulery, B. T., Hicklin, R. A., Buscaglia, J., and Roberts, M. A. (2012). Repeatability and reproducibility of decisions by latent fingerprint examiners. PLOS ONE, 7(3), e32800.(opens in a new tab) Back to the article
  3. Ulery, B. T., Hicklin, R. A., Roberts, M. A., and Buscaglia, J. (2016). Interexaminer variation of minutia markup on latent fingerprints. Forensic Science International, 264, 89–99.(opens in a new tab)

    Accepted manuscript and supporting data.

    Back to the article
  4. Richetelli, N., LeMay, J., Dunagan, K. M., and Chapman, W. J. (2023). A Black Box Study of the Accuracy and Reproducibility of Tire Evidence Examiners’ Conclusions. Final technical research report, NIJ award 2020-DQ-BX-0026; OJP 310751, received July 2025.(opens in a new tab)

    Technical research report received by NIJ in July 2025; distinct from the journal article.

    Back to the article