Sunday, Aug 4: 4:00 PM - 5:50 PM
5028
Contributed Papers
Oregon Convention Center
Room: CC-C125
Main Sponsor
Survey Research Methods Section
Presentations
The covid-19 pandemic has accelerated a trend in survey research to use online data collection for general population samples. High quality web surveys recently achieved response rates comparable to or even exceeding those of telephone surveys. However, selection bias with respect to education is often more pronounced. Most web surveys offer weights to adjust for education bias that rely on the assumption that nonresponse is random conditional on the variables in the model. In 2023, the Institute for Employment Research in Germany launched a new online panel survey of the German workforce (IAB-OPAL) using a push-to-web approach. Addresses were sampled from a powerful database comprising compulsory social insurance notifications by employers as well as unemployment insurance and welfare benefit records. We utilize this unique opportunity of a sampling frame containing detailed individual level information on complete employment biographies. This allows us to assess not only how education bias develops over the recruitment process, but whether response propensities within education strata differ by usually unobserved attributes like benefit receipt experience, occupations or wages.
Keywords
nonresponse bias
web survey
education
Recent changes in the survey landscape due to the Covid-19 pandemic include industry-wide declining response rates and mode changes. These shifts can impact the suitability of a survey's design. The Medicare Current Beneficiary Survey (MCBS), a longitudinal survey of a nationally representative sample of the Medicare population conducted by the Centers for Medicare and Medicaid Services (CMS), is a historically in-person survey with a three-stage geographically clustered design. The sizes of the secondary sampling units (SSUs) were set in 2014. Since then, declines in response rates and a switch to mostly phone interviewing in 2020 have required larger sample sizes than anticipated, straining the effectiveness of the existing SSU design. This paper describes these challenges, our methodology for evaluating alternative strategies for expanding the size and reach of SSUs, and the impacts of the size and structure of the SSUs on sample selection and estimation. We cover the role of SSUs in the MCBS design, options for increasing the size of the SSUs and the available sample within them, considerations for determining the size of SSUs, and tradeoffs between the SSU expansion methods.
Keywords
sample design
data collection
survey sampling
healthcare surveys
Medicare
response rates
Web-based respondent driven sampling (W-RDS), an application of RDS, has potential to recruit hard-to-reach groups with strong social ties and a high Web access rate, such as certain racial/ethnic minority groups. However, little is known about the quality of W-RDS data and the performance of popular RDS estimators.
This study attempts to fill this gap with a national W-RDS study of Korean Americans, Health and Well-being of Koreans (HAWK). This study experimented on the seed types recruited from two sources: 1) social media; and 2) randomly selected addresses in the commercial data associated with Korean surnames. We will compare the HAWK sample estimates by seed type and against estimates for Korean Americans from the American Community Survey. In doing so, we will apply existing RDS estimators as well as model-based estimators to the HAWK data.
Keywords
Respondent Driven Sampling
Web Survey
Statistical inference
Data quality
Model-based estimators
RDS estimators
Survey researchers try to cater to sample members' mode preferences to increase response rates. Past research showed that mode familiarity and access were the strongest predictors of mode, with weaker effects for measures related to physical and cognitive demands, normative concerns, or personal safety concerns (Smyth et al. 2014). Multiple researchers have subsequently used predicted mode preference to tailor mode assignments, sometimes in adaptive or responsive designs (e.g., Coffey et al. 2019; Freedman et al. 2018; Jackson et al 2023). Much has changed in the 15+ years since the initial research was done that might change predictors of mode preference. In this paper, we use updated data collected in the 2022 Nebraska Annual Social Indicators Survey (n=1,455 RR2=18.2%) to reexamine predictors of survey mode preference. Initial analyses indicate that about 2% of Nebraska adults prefer interviewer-administered modes, 45% prefer mail, 35% prefer web on a computer, and 17% prefer mobile web. We also examine whether the mode preference models from 2022 predict mode selection in the 2023 administration of this cross-sectional omnibus survey. Practical implications will be discussed.
Keywords
Data Collection
Nonresponse
Mixed-mode surveys
Mode preference
Moving to a multimode survey design has many benefits over a face-to-face design, including making participation more convenient and reducing data collection costs. However, transitioning from a single, interviewer-administered mode to a mixed-mode design with self-administration can lead to measurement differences caused by mode effects. These changes make it hard for repeated cross-sectional surveys to maintain trends and may necessitate mode effect adjustments. This paper explores a set of mode effect adjustments for a recently transitioned mixed-mode survey.
The General Social Survey (GSS) was a face-to-face survey for nearly 50 years. Given the increasing cost of in-person collection and accelerated by the COVID-19 pandemic, the 2022 GSS was fielded as a multimode study, with respondents completing the survey via web, face-to-face, or phone. We examine the impact of various adjustments to key trend items including logistic regression, multiple imputation using chained equations, implied utility-multiple imputation, and adaptive mode adjustment. Our findings expand the mode effect literature and provide guidance to surveys with longstanding trends moving to a multimode design.
Keywords
mixed-mode
mode effects
multimode design
measurement error
nonresponse
data quality
Co-Author(s)
Sara Lafia, NORC at the University of Chicago
Martha McRoy, NORC at the University of Chicago
First Author
Brian Wells, NORC at the University of Chicago
Presenting Author
Brian Wells, NORC at the University of Chicago
Sample overlap control methods like Keyfitz have long been used in establishment surveys to solve the problem of sampling for two or more surveys for the same or different target populations from one list frame. In establishment surveys, they are commonly used to maximize overlap between two or more samples. Our application of the Keyfitz methodology, we sampled people rather than institutions, with the goal of minimizing overlap. The Ohio Pregnancy Assessment Survey (OPAS) and the Ohio Fatherhood Survey (OFS) target parents of a live born infant who is 2-6 months old at sampling to understand pre-natal, natal, and post-partum outcomes. OPAS is similar to CDC's Pregnancy Risk Assessment Monitoring System (PRAMS) as it uses many PRAMS questions and its target population is mothers of these infants. OFS, also based on the PRAMS, started collecting data from fathers in 2022. The sampling frame for both OPAS and OFS is birth certificate information from Vital Statistics, but we did not want to overly burden families by selecting both parents of an infant. We discuss the logistics of implementing sample overlap control and the potential impacts on stratum size and design weights.
Keywords
Sampling
Sample design
Frame
Overlapping frames
Survey methods
Longitudinal studies serve the purpose of measuring changes over time; however, the validity of such estimates can be threatened when the modes of data collection vary across periods, as different modes can result in different levels of measurement error. This study provides a general framework to accommodate different mixed-mode designs and thus has the potential to support mode comparisons across studies or waves. Borrowing from the causal inference literature, we treat the mode of data collection as the treatment. We employ a potential outcome framework to multiply impute the potential response status of cases if assigned to another mode, along with the associated potential outcomes. After imputation, we construct principal strata based on the observed and the predicted response status of each case to adjust for whether a participant is able to respond via a certain mode when making inference about mode effects. Next, we estimate mode effects within each principal stratum. We then combine these estimates across both the principal strata and the imputed datasets for inference. This analytical strategy is applied to the Health and Retirement Study 2016 and 2018 core surveys.
Keywords
Mode Effects
Principal Stratification
Multiple Imputation
Health and Retirement Study
Sequential Mixed-mode Design