A medical AI accuracy figure may describe findings, images, examinations or patients. Leaders should establish what it actually measures before relying on it for a clinical or procurement decision.
Executive takeaways
- Name the metric, evaluation unit and denominator. Overall classification accuracy, sensitivity, specificity and detection/localisation performance measure different properties. A favourable finding-level score is not automatically evidence of adequate patient-level decision performance.
- Match the metric to the intended workflow. Clinical leaders should approve the decision endpoint; statisticians should address repeated observations and uncertainty.
- Trace the claim to evidence. Independent examination can test consistency of reported metrics and records, but cannot establish patient benefit, legal liability or insurance coverage by itself.
A technically correct score can answer the wrong decision
A hospital evaluating an artificial intelligence (AI) imaging tool may be presented with a high sensitivity estimate. Before comparing vendors or adopting the tool, its clinical leadership needs to know whether that score counts individual lesions, images, examinations or patients—and whether that level matches the decision clinicians will make with the output.
A study can have many images per patient and many findings per image. Evaluation unit is what is scored: a lesion, image, examination or patient. Aggregation rule is how those observations are combined.
Pooled lesion-level sensitivity, image-level classification accuracy and patient-level detection sensitivity measure different properties and may use different denominators. Even when the same metric name is used, changing the evaluation unit or aggregation rule may change what the result means. None of these measures can be substituted for another without a defined endpoint, appropriate statistical analysis and a defensible relationship to the intended clinical use.
None can be substituted for another without definitions, appropriate aggregation and a defensible link to the intended use.
The clinical task matters. A lesion-localisation aid may reasonably report lesion-level sensitivity as one performance measure. A tool claiming to support patient triage needs evidence for the triage decision. Patient-level measurement does not by itself establish clinical benefit either: workflow, human interpretation and downstream outcomes remain separate questions.
A decision-level assessment should also examine the operating threshold at which the reported performance was obtained. Sensitivity and specificity, false-positive and false-negative consequences, and the prevalence of the target condition may affect clinical usefulness differently across screening, triage and diagnosis. A favourable summary metric at one operating point does not establish acceptable performance at another. Threshold selection, reference-standard limitations and relevant uncertainty should therefore remain traceable to the intended workflow.
Accuracy, detection and clinical decisions measure different things
The term accuracy is often used broadly when describing medical artificial intelligence (AI), but it does not identify a single performance measure. The meaning of a reported score depends on the prediction task, reference standard, evaluation unit, denominator and operating threshold.
For a binary classification task, overall classification accuracy is the proportion of evaluated observations classified correctly: (true positives + true negatives) divided by all evaluated observations. This measure can be misleading when the target condition is uncommon or the consequences of false positives and false negatives differ. It should not be treated as a substitute for reporting clinically relevant error rates.
Sensitivity measures the proportion of observations with the target condition that the test correctly identifies as positive. Specificity measures the proportion without the target condition that it correctly identifies as negative. Both require a defined target condition, an appropriate reference standard and a stated evaluation unit. Where comparison is only against a non-reference standard, positive and negative percent agreement may be more appropriate terminology than diagnostic sensitivity and specificity.
Detection and localisation performance concerns whether relevant findings are identified and, where required, correctly located. A lesion-detection system may be evaluated using lesion-level sensitivity together with false-positive findings per image or examination, localisation criteria or other task-appropriate measures. These results do not automatically establish whether the system correctly classifies a patient or supports a particular clinical action.
Decision-level performance evaluates the predefined decision the system is intended to support, such as whether a patient should be escalated for further examination. The endpoint may require patient-level classification, examination-level triage, time-to-action measures or assessment of the clinician–AI workflow. It must be specified rather than inferred from a technical performance score. Even a valid decision-level result does not, by itself, demonstrate improved patient outcomes.
Accordingly, a medical AI performance claim should identify both the metric being reported and the unit at which it was measured. The evaluation protocol should also document the reference standard, aggregation rules, applicable operating threshold, uncertainty and relationship to the intended clinical decision. These distinctions support examination of the claim; they do not independently establish clinical safety, effectiveness or regulatory conformity.
Diagnostic-statistics methods address clustered observations
Established diagnostic-research methods provide relevant support for this distinction. A Radiology tutorial on sensitivity and specificity with clustered observations explains that the appropriate unit and analysis depend on the clinical decision, and that correlation between observations within patients requires statistical attention [1]. A Biostatistics paper on multiple lesions per patient proposes models that do not assume lesions in the same patient are independent [2]. Both papers address diagnostic tests generally and pre-date today’s AI tools, so applying them to medical AI is ODA3 Institute’s analysis, not a finding of either paper.
A separate concern is data leakage, when information from development data contaminates evaluation data. Even where training and test patients are properly separated, reporting multiple observations from one patient as if they were independent patient observations can misstate uncertainty or fail to answer the operational endpoint. ODA3 Institute’s analysis is that neither independence of test cohorts nor a large observation count resolves a mismatch between the measured unit and the intended decision.
Evaluation unit, statistical independence and clinical endpoint are distinct questions. A study may correctly prevent patient-level overlap between development and test datasets while still reporting multiple lesions or images as observations. Within-patient correlation must be addressed when estimating uncertainty, and the aggregation method must be justified separately. Neither correct clustering adjustments nor a large number of observations establishes that the reported metric answers the intended clinical decision.
What the 2025 medical-device guidance establishes
The International Medical Device Regulators Forum (IMDRF) records 29 January 2025 as the publication date for its final Good Machine Learning Practice for Medical Device Development: Guiding Principles (N88 FINAL:2025) [3]. The 2025 document addresses intended use, clinically meaningful performance goals, representative data, independent testing, human–AI team performance and communication of evaluation limitations, among other subjects [3]. The US Food and Drug Administration (FDA) identifies these as guiding principles informing medical-device development, building on a joint October 2021 statement [4][6]. They do not prescribe one universal unit of evaluation or validate any device-specific endpoint. Applicable regulatory evidence remains jurisdiction- and device-dependent.
A related but different evidence gap concerns patient outcomes.
An August 2026 PLOS Digital Health study examined 1,357 AI/ML-enabled medical devices identified as having received US FDA clearance or approval through 5 December 2025. The authors linked 34 devices (2.5%) to registered prospective trials and three (approximately 0.2%) to trials evaluating patient-centred outcomes. These are findings from the researchers’ public-evidence linkage methodology, not a comprehensive determination that the remaining devices had never undergone validation or generated clinical benefit evidence.
Illustrative scenario — synthetic, not based on an examined system
A chest-imaging assistant flags several incidental abnormalities but misses the lesion relevant to a predefined escalation decision. A per-finding performance summary might remain favourable even though the escalation decision fails in that examination. Whether the tool has performed inadequately depends on its documented intended use, the adjudicated reference standard, the operating threshold and approved endpoint. No real device, accuracy rate, patient harm or assurance finding is implied.
Evidence must flow between suppliers, deployers and reviewers
The supplier should provide the evaluation protocol, intended use, unit definitions, reference-standard rules, patient/examination/image/finding linkage, aggregation method, exclusion handling, confidence intervals and model version. The supplier should identify the population, observation window and limits on transferring the results across sites and workflows.
The healthcare organisation should specify the actual decision and human–AI workflow: whether AI output triggers escalation, prioritises review, informs a diagnosis or operates only as an advisory signal. Clinical and statistical specialists should decide what endpoint and analysis are adequate before procurement reliance; an approval or questionnaire alone is not such an assessment.
A qualified internal statistician or authorised external examiner could reconcile sampled identifiers with reported denominators, recompute predefined metrics, inspect exclusions and test the handling of repeated observations. Any external examination would need a defined system and question, approved criteria, an observation period, recorded source provenance, a verification method, exceptions and explicit reliance limits. Under the stated criteria, evidence that cannot be obtained is Unknown, not evidence of failure. A Gap requires a demonstrated contradiction or a required element shown to be missing. Such work supports only a bounded conclusion about metric traceability, not clinical benefit, long-term effectiveness, regulatory acceptance or insurance coverage. Management retains the deployment decision. ODA3 Institute has not examined a system for this article.
| Executive decision | Evidence to examine | What it can establish | What it cannot establish alone |
|---|---|---|---|
| Does the headline metric fit the intended use? | Intended-use statement, clinical endpoint, metric definition | Whether the claimed unit addresses the identified decision | Whether using the tool improves patient outcomes |
| Are numerator and denominator reproducible? | Source identifiers, code or documented aggregation, exclusions | Traceability of reported results | Completeness of unseen records or future reliability |
| Has within-patient dependence been addressed? | Statistical analysis plan and interval estimation | Whether clustering was considered under the stated method | That the chosen method is optimal without specialist review |
| Is the result applicable locally? | Population, sites, device/model version, workflow, monitoring records | Boundary of tested conditions and available deployment observations | Transferability to every hospital or patient group |
| Is the evidence suitable for a risk-transfer question? | Exposures, usage context, controls, incidents, policy wording | What the performance evidence says about one possible failure mode | Insurability, legal liability, premium, coverage or claim payment |
The limited insurance implication
A potential pathway runs from an AI-supported clinical decision through reliance, possible error, patient harm, alleged responsibility and financial loss. None of those steps follows automatically from the previous one. If a healthcare organisation submits medical AI validation evidence to an insurer, one plausible information request is whether the scoring unit matches the clinical failure under consideration. The organisation could substantiate its response using the intended-use statement, validation protocol, deployment controls and documented exceptions. This is ODA3 Institute analysis, not verified insurer practice.
Professional indemnity, product liability or technology errors-and-omissions policies may be relevant for examination, depending on facts, jurisdiction and contract; no policy response has been established. Evaluation-unit disclosure does not demonstrate lower error rates, insurability, premium changes or coverage. This concerns insuring AI-related risk, not an insurer using AI to underwrite.
Decisions for clinical and enterprise leadership
| Action | Accountable owner | Required decision/evidence | When to reconsider |
|---|---|---|---|
| Require evaluation-unit, denominator and aggregation disclosure for material vendor metrics | Clinical evaluation lead and procurement | Product-specific protocol and a comparable decision-level endpoint | If the claim already includes an adequate, independently reviewable endpoint |
| Approve the clinical decision and human–AI workflow before relying on the metric | Chief Medical Officer or delegated clinical governance body | Intended use, escalation logic, reference standard and monitoring responsibilities | If clinical use or model version changes materially |
| Request a bounded metrics-reconciliation exercise when stakes and access warrant it | Risk committee sponsor (accountable); qualified statistician or examiner performs the work | Agreed scope, sampling, exclusions, analysis plan and provenance | If records are inaccessible, materially incomplete or outside examiner competence |
| Separate clinical evidence, regulatory status and insurance disclosure | CRO, General Counsel and insurance lead | A documented statement of what each evidence item does—and does not—support | If actual policy wording, incident experience or clinical outcomes warrant further examination |
Notably Absent
No device-specific raw study data, adjudicated clinical endpoints, statistical analysis files, deployment telemetry or patient outcome records were examined for this article. No examined instance of misleading evaluation-unit reporting, patient injury or clinical negligence is established by the examples. No insurance wording, claims file, premium differential, underwriter acceptance criterion or verified correlated-loss dataset was examined; the reviewed sources did not establish an insurer requirement for evaluation-unit disclosure. The reviewed statistical papers establish a methodological issue; they do not quantify how frequently modern medical AI vendors make this mistake. The 2026 study’s limited public trial linkages do not establish the absence of private validation or a defect in all unlinked devices. No ODA3 Institute client examination, insurer endorsement, certification or completed assurance engagement is asserted.
Conclusion
Before clinical leadership relies on a medical AI accuracy claim, it should require a declared unit, a documented aggregation rule and an endpoint matched to the decision the tool supports. Those records can support a bounded assessment of the performance claim; whether the tool helps patients—and what any insurer or court might conclude—requires different evidence.
References
[1] Genders TSS, Spronk S, Stijnen T, Steyerberg EW, Lesaffre E, Hunink MGM. “Methods for Calculating Sensitivity and Specificity of Clustered Data: A Tutorial.” Radiology, 2012;265(3):910–916. https://doi.org/10.1148/radiol.12120509 (accessed 30 October 2026).
[2] Zwinderman AH, Glas AS, Bossuyt PM, Florie J, Bipat S, Stoker J. “Statistical models for quantifying diagnostic accuracy with multiple lesions per patient.” Biostatistics, 2008;9(3):513–522. https://doi.org/10.1093/biostatistics/kxm052 (accessed 30 October 2026).
[3] International Medical Device Regulators Forum. “Good machine learning practice for medical device development: Guiding principles.” IMDRF/AIML WG/N88 FINAL:2025; official catalogue publication date 29 January 2025 (PDF cover dated 27 January). https://www.imdrf.org/documents/good-machine-learning-practice-medical-device-development-guiding-principles ; https://www.imdrf.org/sites/default/files/2025-02/IMDRF_AIML%20WG_GMLP_N88%20Final.pdf (accessed 30 October 2026; original principle text inspected).
[4] US Food and Drug Administration. “Good Machine Learning Practice for Medical Device Development: Guiding Principles.” FDA webpage, publication date not reliably stated (page states content current as of 19 December 2025); describes January 2025 IMDRF release. https://www.fda.gov/medical-devices/artificial-intelligence-enabled-medical-devices/good-machine-learning-practice-medical-device-development-guiding-principles (accessed 30 October 2026).
[5] Abulibdeh R, et al. “1,357 AI medical devices cleared, 3 actually tested on patient outcomes.” PLOS Digital Health, 19 August 2026, 5(8):e0001597. https://doi.org/10.1371/journal.pdig.0001597 (accessed 30 October 2026; full article inspected on publisher website).
[6] US Food and Drug Administration, Health Canada and UK Medicines and Healthcare products Regulatory Agency. “Good Machine Learning Practice for Medical Device Development: Guiding Principles.” October 2021. https://fda.gov/media/153486/download (accessed 30 October 2026; read in part).
[7] US Food and Drug Administration. Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests.
[8] US Food and Drug Administration. Evaluation Methods for AI-Enabled Medical Devices: Performance Assessment and Uncertainty Quantification.
Methodology note
Editorial evidence review of published sources; no device examination. Findings bounded to claims supported by reviewed public materials. Missing evidence is Unknown, not a demonstrated Gap.
Notably Absent
No raw device-specific data, patient outcome records, operational examination, insurer acceptance criteria, pricing effects, or clinical/statistical specialist sign-off. No certification or completed assurance engagement.
