Detecting Cancer Recurrence from Radiology Imaging Reports Using Supervised Deep Learning and Large Language Models

Music City Center 

Cancer recurrences are critical events that negatively impacts patients' quality of life, often leading to mortality. Yet they are not systematically recorded in population-based cancer registries. Studies have shown that there are potentials in using traditional Natural Language Processing (NLP) techniques to extract the presence and timing of recurrence from clinical notes, pathology reports and radiology reports. Recent developments in large language models (LLMs) within the realm of generative Artificial Intelligence (AI) have shown greater promise in information extraction from narrative texts. Regardless, research is still lacking regarding the use of deep learning methods and LLMs for extracting recurrence information from population-based electronic radiology imaging reports (E-RAD).

In this study, we aim to bridge this gap by comparing various AI frameworks to detect cancer recurrence from population-based E-RAD. This study includes 13,703 radiology reports from 4,498 patients diagnosed between 2011 and 2018 with 12 primary cancer types (e.g., Breast, Prostate, Lung, etc.), and linked to the Surveillance, Epidemiology, and End Results (SEER) program. We developed a deep learning-based framework with a rule-based negation detection algorithm to identify recurrence or metastasis instances and then compared it with the performance of a traditional keyword-searching algorithm and various general and medical-specific LLMs. Our proposed approach outperformed the best-performing comparison models on a validation dataset, achieving higher accuracy (0.99 vs. 0.97), sensitivity (0.88 vs. 0.50), precision (0.78 vs. 0.60), and F1 score (0.82 vs. 0.55), while maintaining a similar specificity (0.99 vs. 0.99).