A framework for estimating and reporting participation rates in large-scale educational assessments

Thomas M. Menino Convention & Exhibition Center 

Agencies conducting large-scale educational assessments recognize that the problem of student non-response is a fundamental threat to the validity of the reported findings. Response rates are thus often reported as quantitative measures of data quality. These studies, however, typically do not distinguish between the statistical and the identification problems arising from non-response when estimating and reporting response rates. The statistical problem is a consequence of a reduced sample size and manifests itself as sampling error. The identification problem is a consequence of the unobservability of some students in the population, which forces analysts to make untestable distributional assumptions. The identification problem manifests as non-sampling error.

We develop a coherent strategy for estimating and reporting response rates that considers an orderly decomposition of these two inferential problems. Importantly, our study accounts for the subtleties arising from the multi-stage sample design governing the data collection, which makes this decomposition non-trivial. We exemplify our strategy with application to the International Computer and Information Literacy Study

Survey methodology

Non-response in stratified, mult-stage random sample designs

Probability samples

Large-scale educational assessments 

Survey Research Methods Section