Complexity measures and generalization in deep learning

Thomas M. Menino Convention & Exhibition Center 

Deep neural networks often acquire their capabilities through qualitatively distinct training phases, and characterizing these phases sheds light on how models learn and generalize. We frame training as a dynamic statistical process and study task-agnostic scalar measures as general-purpose indicators of phase transitions. These include norm-based measures and the Local Learning Coefficient from Singular Learning Theory.
To compare these measures, we introduce a multilingual variant of modular addition in which per-language data fractions control how much consecutive phase transitions overlap. This overlapping regime arises naturally when models acquire capabilities in close succession, yet has lacked simple, controlled benchmarks. Systematically comparing 53 such measures, we find that overlapping transitions pose a substantial challenge for recovering the underlying phase structure. The setting also opens questions of compositionality and representation sharing: whether models reuse circuits across languages rather than relearning the task, which we hope makes it a useful testbed beyond this work.

deep learning

generalization

complexity measures

feature learning

phase transitions

compositionality