Matrix regularization explained

In the field of statistical learning theory, matrix regularization generalizes notions of vector regularization to cases where the object to be learned is a matrix. The purpose of regularization is to enforce conditions, for example sparsity or smoothness, that can produce stable predictive functions. For example, in the more common vector framework, Tikhonov regularization optimizes over$$\backslash min\_x\; \backslash left\backslash |A\; x\; -\; y\backslash right\backslash |^2\; +\; \backslash lambda\; \backslash left\backslash |x\backslash right\backslash |^2$$to find a vector

x that is a stable solution to the regression problem. When the system is described by a matrix rather than a vector, this problem can be written as$$\backslash min\_X\; \backslash left\backslash |A\; X\; -\; Y\backslash right\backslash |^2\; +\; \backslash lambda\; \backslash left\backslash |X\backslash right\backslash |^2,$$where the vector norm enforcing a regularization penalty on x has been extended to a matrix norm on X .

Matrix regularization has applications in matrix completion, multivariate regression, and multi-task learning. Ideas of feature and group selection can also be extended to matrices, and these can be generalized to the nonparametric case of multiple kernel learning.

Basic definition

Consider a matrix

W to be learned from a set of examples, S = (X ^{t)} _{i} , where i goes from 1 to n , and t goes from 1 to T . Let each input matrix X _{i} be \in R ^{DT} , and let W be of size D x T . A general model for the output y can be posed as$$y\_i^t\; =\; \backslash left\backslash langle\; W,\; X\_i^t\; \backslash right\backslash rangle\_F,$$where the inner product is the Frobenius inner product. For different applications the matrices X _{i} will have different forms,^[1] but for each of these the optimization problem to infer W can be written as$$\backslash min\_\; E(W)\; +\; R(W),$$where E defines the empirical error for a given W , and R(W) is a matrix regularization penalty. The function R(W) is typically chosen to be convex and is often selected to enforce sparsity (using \ell ^{1} -norms) and/or smoothness (using \ell ^{2} -norms). Finally, W is in the space of matrices l{H} with Frobenius inner product \langle ... \rangle _{F} .

General applications

Matrix completion

In the problem of matrix completion, the matrix

X ^{t} _{i} takes the form$$X\_i^t\; =\; e\_t\; \backslash otimes\; e\_i\text{'},$$where (e _{t)} _{t} and (e _{i')} _{i} are the canonical basis in R ^{T} and R ^{D} . In this case the role of the Frobenius inner product is to select individual elements w ^{t} _{i} from the matrix W . Thus, the output y is a sampling of entries from the matrix W .

The problem of reconstructing

W from a small set of sampled entries is possible only under certain restrictions on the matrix, and these restrictions can be enforced by a regularization function. For example, it might be assumed that W is low-rank, in which case the regularization penalty can take the form of a nuclear norm.^[2] $$R(W)\; =\; \backslash lambda\; \backslash left\backslash |W\backslash right\backslash |\_*\; =\; \backslash lambda\; \backslash sum\_i\; \backslash left|\backslash sigma\_i\backslash right|,$$where \sigma _{i} , with i from 1 to min D, T , are the singular values of W .

Multivariate regression

Models used in multivariate regression are parameterized by a matrix of coefficients. In the Frobenius inner product above, each matrix

X is$$X\_i^t\; =\; e\_t\; \backslash otimes\; x\_i$$such that the output of the inner product is the dot product of one row of the input with one column of the coefficient matrix. The familiar form of such models is$$Y\; =\; X\; W\; +\; b$$

Many of the vector norms used in single variable regression can be extended to the multivariate case. One example is the squared Frobenius norm, which can be viewed as an

\ell ^{2} -norm acting either entrywise, or on the singular values of the matrix:$$R(W)\; =\; \backslash lambda\; \backslash left\backslash |W\backslash right\backslash |\_F^2\; =\; \backslash lambda\; \backslash sum\_i\; \backslash sum\_j\; \backslash left|w\_\backslash right|^2\; =\; \backslash lambda\; \backslash operatorname\backslash left(W^*W\backslash right)\; =\; \backslash lambda\; \backslash sum\_i\; \backslash sigma\_i^2.$$

In the multivariate case the effect of regularizing with the Frobenius norm is the same as the vector case; very complex models will have larger norms, and, thus, will be penalized more.

Multi-task learning

The setup for multi-task learning is almost the same as the setup for multivariate regression. The primary difference is that the input variables are also indexed by task (columns of

Y ). The representation with the Frobenius inner product is then$$X\_i^t\; =\; e\_t\; \backslash otimes\; x\_i^t.$$

The role of matrix regularization in this setting can be the same as in multivariate regression, but matrix norms can also be used to couple learning problems across tasks. In particular, note that for the optimization problem$$\backslash min\_W\; \backslash left\backslash |X\; W\; -\; Y\backslash right\backslash |\_2^2\; +\; \backslash lambda\; \backslash left\backslash |W\backslash right\backslash |\_2^2$$the solutions corresponding to each column of

Y are decoupled. That is, the same solution can be found by solving the joint problem, or by solving an isolated regression problem for each column. The problems can be coupled by adding an additional regularization penalty on the covariance of solutions$$\backslash min\_\; \backslash left\backslash |X\; W\; -\; Y\backslash right\backslash |\_2^2\; +\; \backslash lambda\_1\; \backslash left\backslash |W\backslash right\backslash |\_2^2\; +\; \backslash lambda\_2\; \backslash operatorname\backslash left(W^T\; \backslash Omega^\; W\backslash right)$$where \Omega models the relationship between tasks. This scheme can be used to both enforce similarity of solutions across tasks, and to learn the specific structure of task similarity by alternating between optimizations of W and \Omega .^[3] When the relationship between tasks is known to lie on a graph, the Laplacian matrix of the graph can be used to couple the learning problems.

Spectral regularization

Regularization by spectral filtering has been used to find stable solutions to problems such as those discussed above by addressing ill-posed matrix inversions (see for example Filter function for Tikhonov regularization). In many cases the regularization function acts on the input (or kernel) to ensure a bounded inverse by eliminating small singular values, but it can also be useful to have spectral norms that act on the matrix that is to be learned.

There are a number of matrix norms that act on the singular values of the matrix. Frequently used examples include the Schatten p-norms, with p = 1 or 2. For example, matrix regularization with a Schatten 1-norm, also called the nuclear norm, can be used to enforce sparsity in the spectrum of a matrix. This has been used in the context of matrix completion when the matrix in question is believed to have a restricted rank.^[2] In this case the optimization problem becomes:$$\backslash min\; \backslash left\backslash |W\backslash right\backslash |\_*\; ~~\; \backslash text\; ~~\; W\_\; =\; Y\_.$$

Spectral Regularization is also used to enforce a reduced rank coefficient matrix in multivariate regression.^[4] In this setting, a reduced rank coefficient matrix can be found by keeping just the top

n singular values, but this can be extended to keep any reduced set of singular values and vectors.

Structured sparsity

Sparse optimization has become the focus of much research interest as a way to find solutions that depend on a small number of variables (see e.g. the Lasso method). In principle, entry-wise sparsity can be enforced by penalizing the entry-wise

\ell ^{0} -norm of the matrix, but the \ell ^{0} -norm is not convex. In practice this can be implemented by convex relaxation to the \ell ^{1} -norm. While entry-wise regularization with an \ell ^{1} -norm will find solutions with a small number of nonzero elements, applying an \ell ^{1} -norm to different groups of variables can enforce structure in the sparsity of solutions.^[5]

The most straightforward example of structured sparsity uses the

\ell _{p,q} norm with p = 2 and q = 1 :$$\backslash left\backslash |W\backslash right\backslash |\_\; =\; \backslash sum\_i\; \backslash left\backslash |w\_i\backslash right\backslash |\_2.$$

For example, the

\ell _{2,1} norm is used in multi-task learning to group features across tasks, such that all the elements in a given row of the coefficient matrix can be forced to zero as a group.^[6] The grouping effect is achieved by taking the \ell ^{2} -norm of each row, and then taking the total penalty to be the sum of these row-wise norms. This regularization results in rows that will tend to be all zeros, or dense. The same type of regularization can be used to enforce sparsity column-wise by taking the \ell ^{2} -norms of each column.

More generally, the

\ell _{2,1} norm can be applied to arbitrary groups of variables:$$R(W)\; =\; \backslash lambda\; \backslash sum\_g^G\; \backslash sqrt\; =\; \backslash lambda\; \backslash sum\_g^G\; \backslash left\backslash |w\_g\backslash right\backslash |\_g$$where the index g is across groups of variables, and |G _{g|} indicates the cardinality of group g .

Algorithms for solving these group sparsity problems extend the more well-known Lasso and group Lasso methods by allowing overlapping groups, for example, and have been implemented via matching pursuit:^[7] and proximal gradient methods.^[8] By writing the proximal gradient with respect to a given coefficient,

w ^{i} _{g} , it can be seen that this norm enforces a group-wise soft threshold$$\backslash operatorname\_\backslash left(w\_g\backslash right)^i\; =\; \backslash left(w\_g^i\; -\; \backslash lambda\; \backslash frac\backslash right)\; \backslash mathbf\_.$$where 1 _{\|w _{g\|} _{g} \geq λ} is the indicator function for group norms \geq λ .

Thus, using

\ell _{2,1} norms it is straightforward to enforce structure in the sparsity of a matrix either row-wise, column-wise, or in arbitrary blocks. By enforcing group norms on blocks in multivariate or multi-task regression, for example, it is possible to find groups of input and output variables, such that defined subsets of output variables (columns in the matrix Y ) will depend on the same sparse set of input variables.

Multiple kernel selection

The ideas of structured sparsity and feature selection can be extended to the nonparametric case of multiple kernel learning.^[9] This can be useful when there are multiple types of input data (color and texture, for example) with different appropriate kernels for each, or when the appropriate kernel is unknown. If there are two kernels, for example, with feature maps

A and B that lie in corresponding reproducing kernel Hilbert spaces l{H _{A},l{H} _{B}} , then a larger space, l{H _{D}} , can be created as the sum of two spaces:$$\backslash mathcal:\; f=h+h\text{'};\; h\backslash in\; \backslash mathcal,\; h\text{'}\backslash in\; \backslash mathcal$$assuming linear independence in A and B . In this case the \ell _{2,1} -norm is again the sum of norms:$$\backslash left\backslash |f\backslash right\backslash |\_\; =\; \backslash left\backslash |h\backslash right\backslash |\_\; +\; \backslash left\backslash |h\text{'}\backslash right\backslash |\_$$

Thus, by choosing a matrix regularization function as this type of norm, it is possible to find a solution that is sparse in terms of which kernels are used, but dense in the coefficient of each used kernel. Multiple kernel learning can also be used as a form of nonlinear variable selection, or as a model aggregation technique (e.g. by taking the sum of squared norms and relaxing sparsity constraints). For example, each kernel can be taken to be the Gaussian kernel with a different width.

See also

• Regularization (mathematics)

Notes and References

1. Book: Lorenzo . Rosasco . Tomaso . Poggio . Tomaso Poggio . A Regularization Tour of Machine Learning . MIT-9.520 Lectures Notes . Manuscript . December 2014 .
2. Exact Matrix Completion via Convex Optimization . Candès . Emmanuel J. . Emmanuel Candès . Recht . Benjamin . 2009 . Foundations of Computational Mathematics . 9 . 6 . 717–772 . 10.1007/s10208-009-9045-5 . free .
3. Zhang . Yeung . 2012 . 1203.3536 . A Convex Formulation for Learning Task Relationships in Multi-Task Learning . Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence (UAI2010) . 2012arXiv1203.3536Z .
4. Alan J. . Izenman . Reduced Rank Regression for the Multivariate Linear Model . . 5 . 2 . 248–264 . 1975 . 10.1016/0047-259X(75)90042-1 . free .
5. Kakade . Shalev-Shwartz . Tewari . Regularization Techniques for Learning with Matrices . Journal of Machine Learning Research . 13 . 2012 . 1865–1890 .
6. A. . Argyriou . T. . Evgeniou . M. . Pontil . 10.1007/s10994-007-5040-8 . Convex multi-task feature learning . . 73 . 3 . 243–272 . 2008 . free .
7. Huang . Zhang . Metaxas . Learning with Structured Sparsity . Journal of Machine Learning Research . 12 . 2011 . 3371–3412 .
8. Chen . Xi . Qihang . Lin . Seyoung . Kim . Jaime G. . Carbonell . Eric P. . Xing . 1 . Smoothing Proximal Gradient Method for General Structured Sparse Regression . Annals of Applied Statistics . 2012 . 6 . 2 . 719–752 . 10.1214/11-AOAS514 . free . 1005.4717 .
9. Sonnenburg . Ratsch . Schafer . Scholkopf . Large Scale Multiple Kernel Learning . Journal of Machine Learning Research . 7 . 2006 . 1531–1565 .