The Radiologist Who Reviews What AI Sees — and What It Misses

By Dr. James Hartley — 2025-11-01

Radiology is the specialty AI has targeted most aggressively. Chest X-ray analysis, CT interpretation, mammography screening, MRI lesion detection — the imaging domain has attracted more AI investment than almost any other area of medicine. Dr. James Hartley, eighteen years into his consultancy at a leading academic medical center, has watched this unfold with a mixture of genuine enthusiasm and professional caution.

The gap between benchmarks and clinical reality

"The diagnostic accuracy numbers are impressive in controlled studies," he says. "But controlled studies don't reflect clinical reality. Real imaging has motion artifact, positioning variations, incidental findings, prior studies to compare. Real patients have comorbidities that change what a finding means. Real radiologists know to call a surgeon when something doesn't quite fit the pattern."

Hartley joined IXO to work specifically on the gap between benchmark performance and clinical performance. His projects involve reviewing AI outputs on imaging cases — not just rating whether the primary finding was identified correctly, but evaluating the full report for completeness, incidental finding recognition, appropriate uncertainty communication, and clinical correlation.

The incidental finding problem

"The incidental finding problem is underappreciated," he says. "A model focused on detecting pulmonary nodules can miss an aortic aneurysm that's sitting right there in the field of view. That's not a theoretical failure — that's something I see in AI evaluations regularly. Annotating those misses creates the training signal the model needs."

Teaching AI the language of uncertainty

He's also contributed to projects on uncertainty calibration — training models to say "this finding is uncertain, clinical correlation recommended" rather than generating false confidence on equivocal cases. "Radiologists have a language for uncertainty. AI often doesn't. Teaching it that language is one of the most valuable things human evaluators can do."

Read The Radiologist Who Reviews What AI Sees — and What It Misses on the IXO blog