Using Natural Language Processing (NLP) in Data Linkage

Music City Center 

The National Center for Health Statistics (NCHS) has a data linkage program that combines national survey data with key sources of health outcomes and health care utilization. The overall accuracy and quality of a data linkage depends on the quality of the data fields. This applies in a variety of data linkage methods, including clear text and PPRL. Data pre-processing and cleaning are essential to address data quality issues in most linkage tasks. Automating pre-processing tasks can reduce time-consuming manual reviews particularly when linkages involve a large number of records. For some data fields, cleaning and pre-processing are relatively straightforward. For example, dates typically have a limited number of plausible values that make checking and cleaning relatively easy. Unique identifiers (e.g., social security number) often conform to some set format or have restrictions on the values that would be expected. Other data fields, such as first name and last name, present greater challenges with respect to automating the cleaning process. The use of NLP to identify valid names and automate identification and removal of non-name text in name fields will be discussed. In addition, the results from an evaluation of artificial-intelligence-based large language model (LLM) and a simple rule-based algorithm to identify non-name text in name fields will be presented.