On the risk of a cross-validated shrinkage estimator in the linear model

Oregon Convention Center 

We study a new shrinkage estimator in the Gaussian model. Unlike the classical James-Stein estimator motivated by the maximization of the marginal likelihood, this estimator is based on the so-called expected log predictive density, a quantity that we estimate with cross validation. We conduct an in-depth risk analysis, and show that its risk is comparable to the one of the celebrated James-Stein estimator. In particular, this estimator outperforms the no-shrinkage baseline if the dimension is greater than 4.
The study of this estimator is motivated by the practice of Bayesian statistics: marginal likelihood maximization for hyperparameter tuning is usually prohibitively expensive even in moderately complex Bayesian models. To deal with this issue practitioners have advocated the use of more tractable surrogates such as the expected log likelihood. Thus, our risk analysis provides theoretical support for this common practice. We apply our shrinkage methodology on an epidemiology application, showing that it can be used to optimally combine information from one small but unbiased sample (a serosurvey) with a large but biased sample (a non-representative survey)

Shrinkage

Expected log predictive density

Cross validation

James-Stein

Gaussian Model 

IMS