Automating Quality Control in Recorded Interviews with Machine Learning

2930 

Contributed Abstract 

Paper 

Peter Baumgartner (1), Kirsty Weitzel (2), Jerry Timbrook (2)

(1) RTI International, N/A, (2) N/A, N/A

Kirsty Weitzel  
N/A

Jerry Timbrook  
N/A

Peter Baumgartner  
RTI International

Kirsty Weitzel  
N/A

Interviewer-administered surveys can suffer from quality issues when question wording differs from the questionnaire. Manual review is required to identify discrepancies and ensure survey quality. RTI QUINTET, a machine learning tool suite, automates quality checks by comparing AI-automated transcripts to the questionnaire. Discrepancies between interviewer administration and the questionnaire are identified, and potentially problematic cases are prioritized for human review. This enhances data quality by identifying re-training opportunities and problematic questionnaire items. We evaluated QUINTET on a telephone healthcare survey with 923 recorded interviews. We compared a random subset of 21 cases that were manually transcribed to transcripts generated by QUINTET, assuming the manual transcripts as ground truth. Preliminary results indicate 90% accuracy for QUINTET. We explore reasons for differences between human and automated transcripts, suggesting future improvements. We also transcribed all interviews to calculate similarity between transcripts to the questionnaire, manually validating low similarity cases. We conclude with discussion of implications for surveys.

Machine Learning|CATI|CARI|Automated Transcription|Survey Administration|Automated Quality Control

Survey Research Methods Section

Miscellaneous