Statistical Innovations in Harnessing Large Language Models and EHR Data for Rare Disease Research

Rare diseases are frequently associated with prolonged diagnostic delays and complex differential diagnosis pathways, driven by heterogeneous clinical presentations and overlapping symptomatology with more common phenotypes. We propose a knowledge-guided graph co-training framework that integrates longitudinal electronic health record (EHR) data with external biomedical knowledge to improve rare disease detection and characterize diagnostic trajectories. Our approach jointly trains across multiple graph modalities, including patient-derived EHR co-occurrence graphs, general biomedical knowledge graphs, and disease-specific knowledge graphs. Through co-training, the model leverages complementary information across graph sources and related phenotypes to learn clinically informed patient representations from both codified and narrative data. By incorporating relationships among diagnoses, medications, procedures, and symptoms, the framework captures evolving patterns of disease presentation and competing differential diagnoses that often precede definitive diagnosis. We demonstrate that multi-graph co-training can identify clinically meaningful diagnostic patterns associated with delayed recognition and misclassification across rare disease populations. These findings highlight the potential of knowledge-guided graph learning to support earlier rare disease detection and improve understanding of diagnostic complexity in high-dimensional EHR data. 

Phenotyping is fundamental to rare disease diagnosis, but manual curation of structured phenotypes from clinical notes is labor-intensive and difficult to scale. Existing artificial intelligence approaches typically optimize individual components of phenotyping but do not operationalize the full clinical workflow of extracting features from clinical text, standardizing them to Human Phenotype Ontology (HPO) terms, and prioritizing diagnostically informative HPO terms. We developed RARE-PHENIX, an end-to-end AI framework for rare disease phenotyping that integrates large language model-based phenotype extraction, ontology-grounded standardization to HPO terms, and supervised ranking of diagnostically informative phenotypes. We trained RARE-PHENIX using data from 2,671 patients across 11 Undiagnosed Diseases Network clinical sites, and externally validated it on 16,357 real-world clinical notes from Vanderbilt University Medical Center. Using clinician-curated HPO terms as the gold standard, RARE-PHENIX consistently outperformed a state-of-the-art deep learning baseline (PhenoBERT) across ontology-based similarity and precision-recall-F1 metrics in end-to-end evaluation (i.e., ontology-based similarity of 0.70 vs. 0.58). Ablation analyses demonstrated performance improvements with the addition of each module in RARE-PHENIX (extraction, standardization, and prioritization), supporting the value of modeling the full clinical phenotyping workflow. By modeling phenotyping as a clinically aligned workflow rather than a single extraction task, RARE-PHENIX provides structured, ranked phenotypes that are more concordant with clinician curation and has the potential to support human-in-the-loop rare disease diagnosis in real-world settings. 

Our understanding of rare diseases remains limited. However, the universal adoption of electronic health record (EHR) systems has enabled high-fidelity characterization of clinical manifestations of diseases with accurate temporality—data that is vital for enhancing clinical decision support and patient outcomes. In this talk, I will present our work leveraging large language models (LLMs) to extract and normalize rare disease phenotype concepts and identify accurate onset information. By supporting phenotype-driven genetic test recommendations and diagnosis, our results demonstrate that nuanced phenotypes derived from EHR narratives effectively accelerate knowledge discovery and application for rare diseases. 

Rare disease knowledge evolves from clinical observations into expert-curated knowledge resources and must be continuously re-evaluated as new real-world evidence becomes available. Effective use of large language models (LLMs) and EHR data for rare disease research therefore requires high-quality knowledge resources beyond generic knowledge bases (KBs) such as OMIM and Orphanet, which are often incomplete, inconsistent, and misaligned with the clinical language used in patient notes. Building and maintaining such resources requires substantial expert effort for knowledge alignment, organization, and disease-specific curation. We investigate how LLM can support this process and how their performance can be rigorously evaluated when the expert reference standard is itself a purpose-built, single-annotator resource.
Using four inherited metabolic diseases (IMDs) spanning the major groups of IMDs and characterized by substantial diagnostic delay, we integrate three complementary knowledge sources: generic knowledge bases, expert-designed diagnostic fact sheets developed by 67 IMD experts, and real-world EHR cohorts from Necker Children's Hospital (Paris, France). We evaluate LLM performance at three stages: (1) aligning phenotype annotations across KBs; (2) aligning expert fact sheets with generic KBs and (3) reproducing physician curation on
1,886 clinical concepts extracted from patient notes for normalization, categorization and diagnostic relevance curation.
Performance is assessed using Cohen's and Fleiss' κ, bootstrap confidence intervals, McNemar's tests, and standard classification metrics.
Across the four IMDs, LLM-assisted phenotype alignment showed only 24.3% exact phenotype agreement between OMIM and Orphanet, while ontology-based semantic relationships explained only 30.4% of the discordant phenotype pairs. LLM-assisted alignment further identified
10 of 104 (9.6%) expert-defined diagnostic phenotypes were absent from both generic KBs. For EHR-derived concepts, the LLM matched physician organ-system categorization for 80.6% of concepts versus 74.9% for an ontology-only approach (p < 0.01), while successfully categorizing the 34% of concepts that could not be mapped by the ontology. For disease-specific relevance curation, a clinical reasoning-grounded LLM reproduced physician keep/drop decisions substantially better than a KB-membership strategy (κ = 0.59 vs. 0.38) and recovered 46% versus 5% of diagnostically relevant phenotypes absent from both generic KBs and expert fact sheets.
These results demonstrate that LLMs are most valuable where existing knowledge resources are incomplete and provide a statistically grounded framework for evaluating and refining rare disease knowledge using complementary evidence from expert resources and real-world EHR data.