Automating Quality Control in Recorded Interviews with Machine Learning

Oregon Convention Center 

Interviewer-administered surveys can suffer from quality issues when question wording differs from the questionnaire. Manual review is required to identify discrepancies and ensure survey quality. RTI QUINTET, a machine learning tool suite, automates quality checks by comparing AI-automated transcripts to the questionnaire. Discrepancies between interviewer administration and the questionnaire are identified, and potentially problematic cases are prioritized for human review. This enhances data quality by identifying re-training opportunities and problematic questionnaire items. We evaluated QUINTET on a telephone healthcare survey with 923 recorded interviews. We compared a random subset of 21 cases that were manually transcribed to transcripts generated by QUINTET, assuming the manual transcripts as ground truth. Preliminary results indicate 90% accuracy for QUINTET. We explore reasons for differences between human and automated transcripts, suggesting future improvements. We also transcribed all interviews to calculate similarity between transcripts to the questionnaire, manually validating low similarity cases. We conclude with discussion of implications for surveys.

Machine Learning

CATI

CARI

Automated Transcription

Survey Administration

Automated Quality Control 

Survey Research Methods Section