Mobile voice surveys: Evaluating response quality using text and voice features

Music City Center 

An increasing smartphone usage in web surveys paves the way for new answer collection methods, as smartphones are equipped with numerous sensors. More specifically, smartphones contain built-in microphones that facilitate the collection of voice responses to open questions that resemble the voice input functions of popular instant messaging apps, such as WhatsApp and WeChat. Even though voice responses potentially trigger open narrations resulting in nuanced and in-depth information from respondents, they usually require complex data processing for which no best practices exist yet. In this study, we contribute to an expansion of the data analysis toolkit in mobile web survey research leveraging acoustic features and Large Language Models (LLMs). In doing so, we explore the potential of integrating LLMs with acoustic feature analysis to automate and improve the evaluation of voice response quality. Our work draws on a smartphone survey (N = 501) which includes two open comprehension probing questions with requests for voice answers. Voice responses are categorized into five different quality types (1) uninterpretable responses, (2) probe-response mismatches, (3) soft nonresponses, (4) hard nonresponses, and (5) substantive responses. Our ongoing analytical methods employ three approaches: (1) encoded linguistic information generated by LLMs, (2) acoustic features derived from speech characteristics, and (3) a multi-modal fusion of linguistic and acoustic features. Our study evaluates the effectiveness of each analytical approach individually and assesses the synergistic benefits of combining them for voice response quality classification. Potentially, our findings will have important implications for social science research in general and mobile web survey research in particular, offering scalable and automated tools for response quality analysis and evaluation. Importantly, the methods developed can be applied to other research domains reliant on voice input, including interview analysis, customer feedback evaluation, and conversational AI systems.

Data Quality

Machine Learning

Speech Processing