Adjusting for verification and selection bias to improve clinical risk model validation

Clinical risk prediction tools are frequently developed from large studies in order to improve public health monitoring, doctor-patient decision-making and clinical trial management. Posting such tools online facilitates rapid external validation across heterogeneous patient populations. While external model validation is extremely important, variation of validation performance even on seemingly similar patient populations can lead to confusion over the utility of the tool, thus discouraging its use. In Pfeiffer et al, Statistics in Medicine, 2022 we formalize the concepts of reproducibility versus transportability of clinical risk tools, as well as differential selection and verification bias between training data for a risk tool and validation data used for performance evaluation. When individual level information from both the training and validation data sets is available, we propose weighted versions of the validation metrics that adjust for differences in the risk factor distributions (selection bias) and in probability of outcome verification (verification bias) between the training and validation data. We suggest that validation studies report both the weighted and unweighted performance measures to provide comprehensive evaluation of risk tools. We illustrate the methods by developing and validating a model for predicting prostate cancer risk using data from two large North American prostate cancer prevention trials, the SELECT and PLCO trials.

risk model

validation

selection bias

verification bias

prostate cancer

online tool