From Screening to Analysis: Harnessing Large Language Models in Clinical Development

The integration of generative artificial intelligence (AI), particularly large language models (LLMs), presents transformative opportunities to enhance clinical trial matching—a traditionally labor-intensive process limiting patient access to potential therapies. Recent advancements demonstrate that both proprietary models (e.g., GPT-3.5, GPT-4) and fine-tuned open-source LLMs (e.g., LLAMA-based models like Trial-LLAMA and OncoLLM) can effectively interpret complex eligibility criteria and match patients using real-world electronic health records (EHRs), achieving performance comparable to medical professionals. These developments promise increased efficiency, reduced costs, and improved privacy and reproducibility in patient-trial matching.

However, the adoption of LLMs in healthcare raises critical equity considerations. Studies reveal that LLMs could potentially exacerbate existing health disparities for clinical trial matching. Furthermore, establishing comprehensive ethical principles is essential for the responsible deployment of generative AI in healthcare, and robust human evaluation methodologies of LLM applications in healthcare are crucial for ensuring safety and effectiveness.

In this talk, we will share recent clinical trial matching system empowered by LLMs developed by our team in collaboration with industry partners, and discuss our recent research on the "GREAT PLEA" ethical principles, the EquityGuard framework, and the QUEST human evaluation framework for LLM applications in healthcare.  

Clinical trials are fundamental in developing new drugs, medical devices, and treatments. However, they are often time-consuming and have low success rates. Although there have been initial attempts to create large language models (LLMs) for clinical trial design and patient-trial matching, these models remain task-specific and not adaptable to diverse clinical trial tasks. To address this challenge, we propose a clinical trial foundation model named Panacea, designed to handle multiple tasks, including trial search, trial summarization, trial design, and patient-trial matching. We also assemble a large-scale dataset, named TrialAlign, of 793,279 trial documents and 1,113,207 trial-related scientific papers, to infuse clinical knowledge into the model by pre-training. We further curate TrialInstruct, which has 200,866 of instruction data for fine-tuning. These resources enable Panacea to be widely applicable for a range of clinical trial tasks based on user requirements.

We evaluated Panacea on a new benchmark, named TrialPanorama, which covers eight clinical trial tasks. Our method performed the best on seven of the eight tasks compared to six cutting-edge generic or medicine-specific LLMs. Specifically, Panacea showed great potential to collaborate with human experts in crafting the design of eligibility criteria, study arms, and outcome measures, in multi-round conversations. In addition, Panacea achieved 14.42% improvement in patient-trial matching, 41.78% to 52.02% improvement in trial search, and consistently ranked at the top for five aspects of trial summarization. Our approach demonstrates the effectiveness of Panacea in clinical trials and establishes a comprehensive resource, including training data, model, and benchmark, for developing clinical trial foundation models, paving the path for AI-based clinical trial development. 

With generative AI setting in motion upon the release of OpenAI's GPT3 in 2000 and ChatGPT in 2022, researchers from cross-disciplines have been actively seeking innovative applications of generative AI tools such as large language models (LLMs) across industries. Such exploratory efforts gradually propagate to clinical development applications, though wide applications in production are still relatively few. On the one hand, LLM research is blooming. On the other hand, many clinical trialists are still seeking good use cases and jumping through hurdles to implement LLM tools in production environment. In this talk, I will share some examples of generative AI initiatives in drug development process and discuss statistician's role in these projects.