Applicability of clinical prediction models: lessons from traumatic brain injury

Music City Center 

Moderate to severe traumatic brain injury carries a poor prognosis, with mortality in 20% of patients within 6 months after injury. The IMPACT and CRASH investigators have developed prognostic calculators for the probability of poor 6 month outcome in adult patients with Glasgow Coma Scale <=12 on admission (https://crash.lshtm.ac.uk/Risk%20calculator/index.html; http://www.tbi-impact.org/?p=impact/calc). By entering clinical characteristics into the calculators, absolute risk predictions are provided for individual patients. We aim to discuss the validity and applicability of these predictions at the population and individual level.
A literature review identified over 50 external validation studies for these models. Standard performance measures such as the area under the ROC curve showed satisfactory results (typical range: 0.7 – 0.85 at external validation). Calibration was more variable, with systematic overestimation of risks.
We conclude that the adequate discrimination supports the application of the calculators and underlying prediction for purposes of classification and characterization of large cohorts of patients. Extreme caution is required when applying the estimated prognosis to individual patients. Other lessons from the field of traumatic brain injury will also be discussed, relating to the importance of study design (sample size, setting); the irrelevance of specific methods for model development (regression versus machine learning); and implementation aspects (presentation of risk, uncertainty, and guidance on medical decision making).

External validation

area under the ROC curve

Calibration

Individualized risk