RESEARCH INTO PRACTICE / The examination record
The same conclusion can conceal a different examination
Two examiners can reach the same opinion through different observations. Preserving those differences is central to understanding the examination behind the answer.
In this article
What does agreement actually tell us?
Imagine two examiners assessing the same fingermark and reference print. Both conclude that the impressions came from the same source. Both record twelve corresponding features. A table containing their conclusions and feature counts would show agreement.
Now suppose that only seven of those features overlap. One examiner has concentrated on a clear central area; the other has included several less certain features near a distorted edge. Their conclusions and totals match, while their recorded examinations differ. This is a hypothetical example, but the distinction is an empirical research problem.
To understand that difference, we need to know which features each examiner selected, where they placed them, what they considered clear and how they interpreted correspondence. A final opinion alone cannot recover that information. The question is how much of the examination remains available for another person to inspect.
A count loses the identity of a feature
Ulery and colleagues’ white-box research examined the annotations supporting fingerprint decisions. Unlike an outcome-only experiment, it retained feature positions, clarity assessments and comparison information. Their 2016 markup study found substantial differences in which minutiae examiners marked, including among examiners who reached the same determination. Differences in totals understated the variation in the selected features.[1]
The associated sufficiency study makes a further distinction: an examiner’s marked count is a record of their selection and interpretation. It should not be treated as an objective inventory of everything present in the impression. Similar counts can describe different selections, and different selections can reflect both interpretation and documentation habits.[2]
Even “feature agreement” needs a definition. Asking how often another examiner reproduces a marked feature differs from asking how widely each candidate location attracts agreement. Restricting a comparison to features both examiners have already marked answers another question again: it leaves out disagreement about whether to select that feature in the first place.[1]
These results also require context. The study material deliberately explored boundaries of sufficiency, and the annotation procedures were unfamiliar to some participants. The observed variation is a reason to investigate how examinations are recorded; it is not a universal estimate of disagreement in routine casework.[1][2]
Completeness and agreement are different achievements
Needham and colleagues approached the problem through documentation. Their study compared routine unprompted notes with a later prompted approach using PiAnoS, examining how the resulting records aligned with an ACE-V documentation checklist. Structured prompts produced greater consistency in elements of the documentation.[3]
Yet conformity in recording a category did not remove differences in what participants recorded within it. Feature positions, selected characteristics and total counts still varied. A question can be answered consistently as a matter of procedure while the assessments it elicits remain different.[3]
That distinction matters when evaluating software. Completion measures whether the requested assessment was recorded. Observational agreement concerns the substance of those assessments. Correctness requires an appropriate reference basis. Improving one of these measures does not establish improvement in the others. The study’s fixed order—unprompted work followed by prompted work—also means that familiarity, order and interface effects were not isolated.[3]
Why PROFAS preserves the examination behind the answer
I began building PROFAS in 2017. Ulery’s work and the wider white-box research were among its foundations, alongside Megan Needham’s developing research, which I knew before publication. The recurring design question was how to retain enough of an examination for someone else to understand the basis of its conclusion.
In PROFAS, a feature annotation has a position on its source material and an associated interpretation. Quality assessments, comparison relationships and explanatory notes have their own roles in the record. This allows a reader to move beyond a total and inspect the material to which an observation refers.
The record also develops over time. Accepted revisions retain earlier contributions; submitted stage records preserve a defined examination basis. A later view of the work should therefore be read with its sequence, rather than allowing the final arrangement of annotations to stand in for everything that preceded it.
This structure aims to make examinations easier to inspect and differences easier to study. It does not require examiners to produce identical observations, and the existence of a detailed record does not establish that the observations are correct. The design preserves the information needed to investigate those questions.
What can researchers ask of that record?
With participants examining the same material under a defined workflow, researchers can investigate where annotations overlap, where quality assessments diverge and which features enter the comparison. They can examine whether agreement in the conclusion accompanies agreement in its recorded basis.
The comparison rules still need to be explicit. Two nearby marks might represent the same feature placed slightly differently, or two genuinely different selections. Ulery’s analysis used spatial clustering to address this problem and showed that the grouping parameters affected the measured agreement. Saving coordinates enables a comparison; it does not choose a scientifically appropriate matching rule.[1]
An absent annotation also needs care. It might reflect an unobserved feature, a feature considered unimportant, an instruction to mark only selected detail, or a recording choice. Shared instructions and contemporaneous explanation can help researchers interpret that absence. The record provides evidence about what was documented, rather than direct access to everything an examiner perceived.[1][2]
A platform for investigating the differences
The research helped establish what PROFAS should retain: observations connected to material, interpretation recorded in context and an examination history that remains inspectable. It also supplies questions for evaluating the platform itself. Does the workflow capture useful distinctions? Can another examiner understand the record? What recording burden does it introduce?
Those are empirical questions for appropriately designed studies. The aim is to give examiners, educators and researchers a common way to preserve the detail needed to answer them. Agreement about an outcome is valuable information. The examination supporting it deserves the same attention.
FOLLOW THE EVIDENCE
Sources & further reading
The original research behind this article.
- Ulery, B. T., Hicklin, R. A., Roberts, M. A., and Buscaglia, J. (2016). Interexaminer variation of minutia markup on latent fingerprints. Forensic Science International, 264, 89–99.(opens in a new tab)
Feature selection, spatial correspondence and the meaning of examiner agreement.
Back to the article - Ulery, B. T., Hicklin, R. A., Roberts, M. A., and Buscaglia, J. (2014). Measuring what latent fingerprint examiners consider sufficient information for individualization determinations. PLOS ONE, 9(11), e110179.(opens in a new tab) Back to the article
- Needham, M., Fieldhouse, S., Morris, W., Wheeler, J., and Nicholls, G. (2022). Collaborative practise in forensic science and academia: The development of a documentation strategy for fingerprint examinations in an English fingerprint bureau in the ISO 17025 era. Science & Justice, 62(3), 336–348.(opens in a new tab) Back to the article