Improving Inferences Based on Survey Data Collected Using Mixed-Mode Designs

Music City Center 

Although survey modes often have different measurement properties, standard practice is to pool mixed-mode data, neglecting the potential impacts of mode effects. This study proposes three approaches: the "Testimator" approach, a Bayesian approach, and a model averaging approach. In the "Testimator" approach, we test whether the means and variances of mixed-mode samples are the same. If the null hypothesis is not rejected, we take the average of the estimates; otherwise, we use the estimate in the preferred direction (assumed to be known). In the Bayesian approach, we estimate the effect size and the ratio of variances to determine cutoffs. We then use the probability of these two quantities falling into different cutoff regions as weights to combine estimates. In the model averaging approach, we combine estimates from four models—each assuming either the same or different means and variances across modes—using marginal posteriors as weights. Compared to existing methods, our proposed approaches incorporate testing procedures into inference, leading to more robust results. We evaluate these methods through a simulation study and apply them to the Arab Barometer Survey wave 6 data.