The Anatomy and Evolution of Survey Error

Thomas M. Menino Convention & Exhibition Center 

The survey research literature has documented error in surveys from four broad sources, survey coverage, unit non-response, item non-response, and measurement error. We provide new methods to analyze and decompose survey error in an empirical Total Survey Error (TSE) framework that measures error in terms of bias, further subdividing the four sources into false positives, false negatives, and errors in amounts. We apply our approach to comprehensively measure error in the CPS ASEC, the official source of income and poverty statistics in the U.S., using linked administrative records for ten income sources over two decades. Our results show severe and rising underreporting of income and program receipt, driven primarily by measurement error among survey respondents on the extensive margin (false negatives). As a result, improving reporting among respondents may be the most promising margin for remedy. Non-respondent imputations are particularly noisy, with positive and negative errors that tend to offset in net terms. Moreover, we find strong evidence that absolute error (noise) has risen over time. Our approach clarifies the sources of bias and where improvements could be greatest.

Total Survey Error

Measurement Error

Survey Bias 

Government Statistics Section