Cross-fitting model evaluation for small area estimation using complex survey data.

Oregon Convention Center 

Model checking, evaluation, or comparison in Small Area Estimation (SAE) with limited data is difficult. A generic problem is that given a survey dataset D, what is a good metric to score a model M? Considering cluster sampling for the national surveys, we would like to achieve two goals: 1) to score models based on their ability to estimate subpopulation prevalence at different administrative levels. 2) to decide if a given model M can be accepted (or not rejected under a hypothesis testing framework). Focusing on a scenario where there is one level of spatial unit, we want to score models based on their ability to produce national estimates. We evaluate models using score rules such as mean square error (MSE), continuous ranked probability score (CRPS), and distribution-free score from conformal prediction, based on leave-one-region out, leave-one-cluster-out, or other splitting methods, and we use design-based estimates as a reference.

Cross validation

Small Area Estimation

Complex survey data 

Survey Research Methods Section