08: A Unified Framework for Inference with General Missingness Patterns and Machine Learning Imputation

Thomas M. Menino Convention & Exhibition Center 

Predictions from machine learning (ML) models are increasingly used to impute missing data, but their naive use risks biased inference. Existing methods provide valid inference with ML imputations regardless of prediction quality to enhance efficiency, while they are limited to missing outcomes under a missing-completely-at-random assumption. We develop a novel framework for valid statistical inference in Z-estimation problems using ML imputations under a missing-at-random assumption and for general missingness patterns. The method stratifies data by distinct missingness patterns and constructs an estimator by appropriately weighting and aggregating pattern-specific information. We establish the asymptotic theory and provide a theoretical guarantee on efficiency dominance over weighted complete-case analyses (WCCA). Practically, the method affords simple implementations by leveraging existing WCCA software. Extensive simulations are carried out to validate theoretical results. An analysis of \textit{All of Us} data further shows the practical utility of the method. The paper concludes with a brief discussion on practical implications, and potential future directions.

All of Us

Machine learning imputation

Missing data

Prediction-based inference

Z-estimation 

Biometrics Section