User:Sbowen209/sandbox

For other uses, see User:Sbowen209/sandbox (disambiguation).

In statistics, the LASSO (Least Absolute Shrinkage and Selection Operator), is a regularized regression method that is used for variable shrinkage and selection in linear or logistic regression models.

History

The LASSO method was created by Robert Tibshirani after being influenced by Leo Breiman's work on subset selection. The LASSO method has grown in popularity due to the importance of variable selection in big data problems and advances in computational technologies.

The LASSO

Given a linear regression with a response variable $y\in R^{n}$ and a matrix $X\in R^{nxp}$ of explanatory variables, the LASSO method produces the estimate for the unknown parameter $\beta$:

${\hat {\beta }}^{lasso}={\underset {\beta }{\operatorname {argmin} }}(\|y-X\beta \|^{2}){\mbox{ s.t. }}\sum _{j=1}^{p}|\beta _{j}|\leq t.$

This is equivalent to the following loss function:

${\hat {\beta }}^{lasso}={\underset {\beta }{\operatorname {argmin} }}(\|y-X\beta \|^{2}+\lambda \|\beta \|_{1}),$

where $\|\beta \|_{1}$ is an L1 penalty and the tuning parameter $\lambda$ has a one-to-one correspondence with $t$ and governs the strength of this penalty. Like other types of regularization, such as ridge regression, the size of the coefficients shrink as $\lambda$ is increased. However, unlike ridge regression, some of the coefficients are shrunken identically to zero. In this manner, the LASSO performs continuous variable selection.

Elastic Net Regularization

Main article: Elastic Net

The LASSO may saturate quickly in the case of data with a large number of predictors but only few observations. Additionally, it will ignore variables if they are highly correlated to another. Variations such as the elastic net method can overcome these limitations by employing penalties from both the LASSO and ridge regression.

Computation

Because the LASSO utilizes an absolute value operation in its penalty, the loss function is not differentiable and in general does not have a closed form. Therefore, convex optimization algorithms are required to find a solution. In the original paper, solutions were found through quadratic programming, but more recent methods such as least-angle regression and coordinate descent have proven more efficient.

Software

"Glmnet: Lasso and elastic-net regularized generalized linear models" is software which is implemented as an R source package. This includes fast algorithms for estimation of generalized linear models with ℓ₁ (the lasso), ℓ₂ (ridge regression) and mixtures of the two penalties (the elastic net) using cyclical coordinate descent, computed along a regularization path.

See also

• Ordinary Least Squares
• Ridge regression

References

• Hastie, Trevor; Tibshirani, Robert; Friedman, Jerome H. "The Elements of Statistical Learning". Retrieved 3 March 2014.

Category:Mathematical analysis Category:Inverse problems