Representing a Collection of Large Language Models as a Gaussian Mixture

Music City Center 

Motivated by the prevalence of black-box large language models (LLMs), we aim to understand the statistical properties of LLMs via response embeddings. We consider prompt augmentation, with each augmentation corresponding to one augmented LLM. Statistical consistency of the response embeddings is established. We define a measure of dissimilarity between response embeddings for a collection of augmented LLMs, which leads to a matrix of comparative dissimilarities. We consider, via multidimensional scaling (MDS), representing this dissimilarity matrix in low-dimensional Euclidean space. Under regularity conditions, we prove a row-wise central limit theorem for the MDS representation associated with the collection of augmented LLMs. That is, for a given query set, the MDS embedding of each augmented LLM asymptotically follows a Gaussian mixture model distribution when the augmentations are drawn from a mixture distribution.

Gaussian Mixture Model

Multidimensional Scaling

Central Limit Theorem

Prompt Augmentation

Large Language Model 

Section on Statistical Learning and Data Science