SysChat: A Human-Expert Guided Retrieval Augmented Generation (RAG) Chatbot for Complex System Question Answering

Music City Center 

SysChat is a Retrieval Augmented Generation (RAG) tool that combines information retrieval, black-box large language models (LLMs), and expert feedback to answer user questions on mechanical systems. In retrieval-augmented generation, an embedding model first stores all documents in a vector database—in our case, tens of thousands of pages of complex systems documents. These embeddings are then used to identify relevant information for each query, guiding black-box LLM responses with improved factual accuracy and traceable information sources. Experts were then given access to this tool, and their feedback was used to train auxiliary methods that guide LLM outputs towards expert-preferred responses. This talk will discuss SysChat's architecture, highlighting classical and modern RAG techniques, LLM enhancements to improve reasoning capabilities, and the integration of expert feedback to guide black-box LLM generation.

LLM

NLP

Information Retrieval

Generative AI