Sparse Boosting

Peter Bühlmann; Bin Yu

Sparse Boosting

Peter Bühlmann, Bin Yu; 7(36):1001−1024, 2006.

Abstract

We propose Sparse Boosting (the SparseL₂Boost algorithm), a variant on boosting with the squared error loss. SparseL₂Boost yields sparser solutions than the previously proposed L₂Boosting by minimizing some penalized L₂-loss functions, the FPE model selection criteria, through small-step gradient descent. Although boosting may give already relatively sparse solutions, for example corresponding to the soft-thresholding estimator in orthogonal linear models, there is sometimes a desire for more sparseness to increase prediction accuracy and ability for better variable selection: such goals can be achieved with SparseL₂Boost.

We prove an equivalence of SparseL₂Boost to Breiman's nonnegative garrote estimator for orthogonal linear models and demonstrate the generic nature of SparseL₂Boost for nonparametric interaction modeling. For an automatic selection of the tuning parameter in SparseL₂Boost we propose to employ the gMDL model selection criterion which can also be used for early stopping of L₂Boosting. Consequently, we can select between SparseL₂Boost and L₂Boosting by comparing their gMDL scores.