21: Enhancing SQL Code Efficiency in Insurance Data with LLMs: A Repeated Measures Approach

Music City Center 

This project evaluates the effectiveness of an LLM-driven (Large Language Model) tool for SQL documentation and programming language conversion/SQL code generation. The experiment tests the LLM tool with code samples at three complexity levels-beginner, intermediate, and advanced-under three prompt conditions: minimally defined, moderately defined, and extremely defined. Raters will assess the LLM-generated outputs using a pre-set rubric. The statistical analysis will employ a Repeated Measures ANOVA to determine the impact of the experimental conditions on the tool's performance. Inter-rater reliability will be measured using Shrout and Fleiss intraclass correlations to measure evaluation consistency.

LLM (Large Language Models)

SQL Code

Documentation

Quality Evaluation

Repeated Measures ANOVA

Inter-Rater Reliability Statistics 

Quality and Productivity Section