18: Evaluating the Generalizability of Commercial Healthcare Claims Data

Music City Center 

Large healthcare claims databases, which aggregate claims from commercial insurers, are increasingly being used to generate real-world evidence in medical research. Nearly 10,000 manuscripts have been published, and the pace of output is accelerating. Despite their widespread use, these databases have not been rigorously vetted against ground-truth data. Representation in such datasets has been found to be systematically biased along racial and socioeconomic lines. These same factors are known to be effect modifiers for a myriad of conditions and treatments in medicine, and the combination of inconsistent sampling and effect modification can give rise to external validity bias. In [Dahlen Deng & Charu 2024], we undertook the most detailed empirical analysis of external validity bias in healthcare claims data to date, focusing on the rates of a comprehensive set of inpatient procedures, for which a unique ground-truth dataset exists. We found large variation in the extent of the bias across procedures, including 22.8% that were underestimated by more than a factor of 2. Further, we found a significant relationship between social determinants of health and the magnitude of bias.

external validity bias

healthcare claims databases 

Section on Statistics in Epidemiology