Complexity measures and generalization in deep learning

Over-parameterized deep neural networks necessitate new statistical complexity measures to accurately capture generalization behavior. We analyze training as a dynamic statistical process characterized by distinct phases of feature learning and structural evolution. Our approach leverages tools from Singular Learning Theory (SLT), particularly the Local Learning Coefficient (LLC), providing singularity-aware measures of effective statistical capacity. A systematic investigation correlates the evolution of various complexity metrics (LLC, norm-based) with feature learning and generalization performance. Crucially, we introduce stabilized complexity measures, which are invariant across function-equivalent parameter sets. This ensures a statistically reliable, geometry-aware estimate of the model's true generalization capacity.

deep learning

generalization

complexity measures

feature learning