Hyper-Parameter Selection for Robust & Non-Convex Estimators via Information Sharing

Music City Center 

Robust estimators for regression use non-convex objective functions to shield against adverse affects of outliers. The non-convexity brings challenges, particularly in combination with penalization. Selecting hyper-parameters for the penalty is a critical task, with cross-validation (CV) the prevalent strategy in practice and good performance for convex estimators. Applied to robust estimators, however, CV often gives poor results due to the interplay between multiple local minima and the penalty. The best local minimum attained on the full sample may not be the minimum with the desired statistical properties. Furthermore, there may be a mismatch between this minimum and the minima attained in the CV folds. We introduce a novel adaptive CV strategy that tracks multiple minima for each combination of hyper-parameters and subsets of the data. A matching scheme is presented for correctly evaluating minima computed on the full sample using the corresponding minima from the CV folds. We show that the proposed strategy reduces the variability of the estimated performance metric, leads to smoother CV curves, and hence substantially increases the reliability of robust penalized estimators.

Robust regression

Hyper-parameter tuning

Cross-validation

Non-convexity estimator

Penalized regression 

Section on Statistical Learning and Data Science