Random-projection ensemble dimension reduction

Music City Center 

We propose a framework for dimension reduction in high-dimensional regression, by aggregating an ensemble of random projections selected based on empirical regression performance. Specifically, we consider disjoint groups of independent random projections, apply a base regression method after each projection is appied to the covariates, and retain the best-performing projection in each group. The selected projections are aggregated by taking the SVD of their empirical average, yielding the leading singular vectors. Notably, the singular values indicate the importance of the corresponding projection directions, aiding in selecting the final projection dimension. We provide recommendations on aspects of our framework, including the projection distribution, base regression method, and the number of random projections. Additionally, we explore further dimension reduction by applying our algorithm twice when the initially recommended dimension is too large. Our theoretical results show that the error of algorithm stabilises as the number of projection groups increases. We demonstrate our proposal's strong empirical performance through an extensive study using simulated and real data.

High-dimensional

mean central subspace

random projection

singular value decomposition

sufficient dimension reduction 

Section on Statistical Learning and Data Science