The Predictive Degrees of Freedom of LASSO

Music City Center 

The double descent phenomenon observed in overparameterized machine learning models appears to defy classical prediction risk theory and has spurred considerable research. Recently, a notion of predictive model degrees of freedom (PDOF) has been proposed as an alternative measure of model complexity to explain the double descent phenomenon with a focus on linear modeling procedures. We extend PDOF to the nonlinear case by first studying the lasso model. The PDOF for lasso involves the covariance matrix of the lasso estimator, for which no closed-form expression exists. Furthermore, existing covariance matrix estimators only work in the under-parameterized case. To fill this gap, we explore two estimators: one based on the iterative soft-thresholding algorithm, and the other based on the infinitesimal jackknife. In a simulation study, we compare these estimators with bootstrap and other covariance matrix estimators based on approximate lasso solutions. Beyond lasso, the infinitesimal jackknife approach can be used to quantify the PDOF of other algorithmic models such as random forests and neural networks.

Double Descent

Infinitesimal Jackknife

Lasso

Model Degrees of Freedom

Variance Estimation 

Section on Statistical Learning and Data Science