From Naive Bayes to Large Language Models: A New Era of Text-Based Classification

Music City Center 

Text classification has been central to data science, with models like Naive Bayes serving as foundational tools. While effective for short, structured data, traditional classifiers struggled with long-form text due to methods like TF-IDF, which failed to capture context and long-term dependencies. Tasks like sentiment analysis of reports or categorizing legal documents often led to inefficiencies. Large language models (LLMs) have transformed this landscape. Transformer architectures excel at understanding context within text, enabling accurate classification for complex tasks such as identifying topics in policy documents or classifying unstructured medical records. Further, smaller, fine-tuned LLMs like BERT provide scalable, cost-effective solutions without sacrificing performance compared to larger general-purpose LLMs like GPT-4o which involve higher costs for both training and inference. This talk focuses on how such smaller LLMs can be utilized to solve classification problems at a low cost with high accuracy. It'll highlight strategies for deploying such models efficiently so that they empower data scientists and statisticians to tackle challenges in modern text analytics.

Text classification

Large language models (LLMs)

Transformer architectures

BERT

Cost-effective solutions


Modern text analytics 

Section on Statistics and Data Science Education