De-sparsified lasso explained

De-sparsified lasso contributes to construct confidence intervals and statistical tests for single or low-dimensional components of a large parameter vector in high-dimensional model.^[1]

High-dimensional linear model

Y = X\beta ^{0} + \epsilon with n x p design matrix X = : [X _{1,...,} X _{p]} (n x p vectors X _{j} ), \epsilon \sim N _{n(0,} \sigma ^{2} _{\epsilon} I) independent of X and unknown regression p x 1 vector \beta ^{0} .

The usual method to find the parameter is by Lasso:

\hat{\beta} ^{n(λ)} = \underset{\beta \in R ^{p}{argmin}} (1)/(2 n) \left\| Y - X \beta \right\| ^{2} ^{2} + λ \left\| \beta \right\| _{1}

The de-sparsified lasso is a method modified from the Lasso estimator which fulfills the Karush–Kuhn–Tucker conditions^[2] is as follows:

\hat{\beta} ^{n(λ,M)} = \hat{\beta} ^{n(λ)} + (1)/(n) M X ^{T(Y-} X \hat{\beta} ^{n} ( λ ))

where

M \in R ^{p x} is an arbitrary matrix. The matrix M is generated using a surrogate inverse covariance matrix.

Generalized linear model

Desparsifying

l _{1} -norm penalized estimators and corresponding theory can also be applied to models with convex loss functions such as generalized linear models.

Consider the following

1 x p vectors of covariables x _{i} \in \chi\subset R ^{p} and univariate responses y _{i} \in Y \subset R for i = 1 ,...,n

we have a loss function

\rho _{\beta(y,x)} = \rho(y, x \beta) (\beta \in R ^{p)} which is assumed to be strictly convex function in \beta \in R ^{p}

The

l _{1} -norm regularized estimator is \hat{\beta} = \underset{\beta}{argmin}(P _{n} \rho _{\beta} + λ \left\| \beta \right\| _{1)}

Similarly, the Lasso for node wise regression with matrix input is defined as follows:Denote by

\hat{\Sigma} a matrix which we want to approximately invert using nodewise lasso.

The de-sparsified

l _{1} -norm regularized estimator is as follows:\hat{\gamma _{j}:=} \underset{\gamma \in R ^{p-1} }(\hat_ - 2 \hat_ \gamma + \gamma^T \hat_ \gamma + 2 \lambda_j\left\|\gamma\right\|_1

where

\hat{\Sigma} _{j,/j} denotes the j th row of \hat{\Sigma} without the diagonal element (j,j) , and \hat{\Sigma} _{/j,/j} is the sub matrix without the j th row and j th column.

Notes and References

1. Geer . Sara van de . Buhlmann . Peter . Ritov . Ya'acov . Dezeure . Ruben . On Asymptotically Optimal Confidence Regions and Tests for High-Dimensional Models . The Annals of Statistics . 2014 . 42 . 3 . 1162–1202 . 10.1214/14-AOS1221 . 1303.0518 . 9663766 .
2. Web site: Tibshirani. Ryan. Gordon. Geoff. Karush-Kuhn-Tucker conditions.