Print  Close

A Retrospective Analysis of the SmartFind COVID-19 Vaccine Chatbot: Statistical Insights

Abstract Number:

2412 

Submission Type:

Contributed Abstract 

Contributed Abstract Type:

Paper 

Participants:

Yi Mu (1), Suchita A. Patel (2), Faisal Reza (2), Cynthia Knighton (2), Angela Marie Chambliss (2)

Institutions:

(1) N/A, N/A, (2) CDC, Atlanta, GA

Co-Author(s):

Suchita A. Patel  
CDC
Faisal Reza  
CDC
Cynthia Knighton  
CDC
Angela Marie Chambliss  
CDC

First Author:

Yi Mu  
N/A

Presenting Author:

Yi Mu  
N/A

Abstract Text:

In 2021, the Centers for Disease Control and Prevention (CDC) implemented a cloud-based chatbot called SmartFind COVID-19 Vaccine Chatbot. This Chatbot employed natural language processing (NLP) to automatically address vaccination-related inquiries. It provided high-confidence responses matched to CDC's COVID-19 frequently asked questions and answers (FAQ&As), or a "Sorry, the ChatBot couldn't find a good match" response for low-confidence matches. An analysis of system logs from August 30, 2021 to March 16, 2023 examined 64,884 visitor questions (of which about three-fifths received an NLP matched response) and 3,925 visitor feedback entries. The goal of this project is to use NLP statistical methods, including tokenization and feature extraction, to analyze question text to determine topics that the chatbot was not able to provide a matched response for, including by design for clinical and disease-related questions, for example. The results can guide improvement of vaccination content by including FAQ&As on CDC's webpages and informing development of future chatbots using more powerful large language models.

Keywords:

Chatbot|Natural Language Processing|COVID-19 Vaccination| | |

Sponsors:

Section on Text Analysis

Tracks:

Miscellaneous

Can this be considered for alternate subtype?

Yes

Are you interested in volunteering to serve as a session chair?

Yes

I have read and understand that JSM participants must abide by the Participant Guidelines.

Yes

I understand that JSM participants must register and pay the appropriate registration fee by June 1, 2024. The registration fee is non-refundable.

I understand