Bridging Statistics and Modern AI: Foundations for Deep Learning and Generative Models

Large language models are capable of in-context learning, the ability to perform new tasks at test time using a handful of input-output examples, without parameter updates. We develop a universal approximation theory to elucidate how transformers enable in-context learning. For a general class of functions (each representing a distinct task), we demonstrate how to construct a transformer that, without any further weight updates, can predict based on a few noisy in-context examples with vanishingly small risk. Unlike prior work that frames transformers as approximators of optimization algorithms (e.g., gradient descent) for statistical learning tasks, we integrate Barron's universal function approximation theory with the algorithm approximator viewpoint. Our approach yields approximation guarantees that are not constrained by the effectiveness of the optimization algorithms being mimicked, extending far beyond convex problems like linear regression. The key is to show that (i) any target function can be nearly linearly represented, with small L1-norm, over a set of universal features, and (ii) a transformer can be constructed to find the linear representation -- akin to solving Lasso -- at test time. This is joint work with Gen Li, Yuchen Jiao, Yu Huang, and Yuting Wei.
 

Convolutional Neural Networks (CNNs) are foundational in modern image analysis due to their ability to efficiently learn feature representations. However, theoretical understanding of their efficiency remains limited, largely due to inadequate modeling of image structures and their interaction with CNNs. To address this, we introduce novel statistical generative models (SGMs) that decompose images into task-relevant signals and noise, capturing the complexities of natural image data. Based on these SGMs, we propose a feature mapping approach (FMA) to characterize the transformation from raw image data to feature vectors. We analyze CNNs' approximation capabilities, their adaptation to low-dimensional structures, and their efficiency in vision tasks, ultimately developing statistical learning theories for CNN-based image analysis. Our findings reveal the challenges inherent in vision tasks and highlight CNNs' remarkable efficiency in addressing them, providing new insights into their theoretical and practical capabilities. This is based on the joint work with Dr. Guohao Shen.  

Artificial neural networks are inspired by the functioning of the brain but differ in several key aspects. In biological neural networks, information is encoded in the spiking times of neurons. Furthermore, it is implausible that biological learning is based on gradient descent. This has prompted researchers to propose various biologically inspired learning procedures. However, these methods lack a solid theoretical foundation. While statistical theory for artificial neural networks has been developed over the past years, the aim now is to extend this theory to biological neural networks, as the future of AI is likely to draw even more inspiration from biology. In this talk, we will explore the challenges and present some statistical risk bounds for different biologically inspired learning rules. 

Deep neural networks often acquire their capabilities through qualitatively distinct training phases, and characterizing these phases sheds light on how models learn and generalize. We frame training as a dynamic statistical process and study task-agnostic scalar measures as general-purpose indicators of phase transitions. These include norm-based measures and the Local Learning Coefficient from Singular Learning Theory.
To compare these measures, we introduce a multilingual variant of modular addition in which per-language data fractions control how much consecutive phase transitions overlap. This overlapping regime arises naturally when models acquire capabilities in close succession, yet has lacked simple, controlled benchmarks. Systematically comparing 53 such measures, we find that overlapping transitions pose a substantial challenge for recovering the underlying phase structure. The setting also opens questions of compositionality and representation sharing: whether models reuse circuits across languages rather than relearning the task, which we hope makes it a useful testbed beyond this work.