LLM-driven Bots in Web Surveys: Predicting Robotic Language in Open Narrative Answers

Music City Center 

Web survey data is key for social and political decision-making, including official statistics. However, respondents are often recruited through online access panels or social media platforms, making it difficult to verify that answers come from humans. As a consequence, bots – programs that autonomously interact with systems – may shift web survey outcomes and social and political decisions. Bot and human answers often differ regarding word choice and lexical structure. This may allow researchers to identify bots by predicting robotic language in open narrative answers. In this study, we therefore investigate the following research question: Can we predict robotic language in open narrative answers? We conducted a web survey on equal gender partnerships, including three open narrative questions. We recruited 1,512 respondents through Facebook ads. We also programmed two LLM-driven bots that each ran through our web survey 200 times: The first bot is linked to the LLM Gemini Pro, and the second bot additionally includes a memory feature and adopts personas (e.g. age and gender). Using a transformer model (BERT) we attempt to predict robotic language in the open narrative answers.