Vectorization (mathematics) explained

In mathematics, especially in linear algebra and matrix theory, the vectorization of a matrix is a linear transformation which converts the matrix into a vector. Specifically, the vectorization of a matrix A, denoted vec(A), is the column vector obtained by stacking the columns of the matrix A on top of one another:$$\backslash operatorname(A)\; =\; [a\_\{1,1\},\; \backslash ldots,\; a\_\{m,1\},\; a\_\{1,2\},\; \backslash ldots,\; a\_\{m,2\},\; \backslash ldots,\; a\_\{1,n\},\; \backslash ldots,\; a\_\{m,n\}]^\backslash mathrm$$Here,

a _{i,j} represents the element in the i-th row and j-th column of A, and the superscript {} ^{T} denotes the transpose. Vectorization expresses, through coordinates, the isomorphism R ^{m} : = R ^{m} ⊗ R ^{n} \cong R ^{mn} between these (i.e., of matrices and vectors) as vector spaces.

For example, for the 2×2 matrix

A = \begin{bmatrix} a & b \ c & d \end{bmatrix} , the vectorization is \operatorname{vec}(A) = \begin{bmatrix} a \ c \ b \ d \end{bmatrix} .

The connection between the vectorization of A and the vectorization of its transpose is given by the commutation matrix.

Compatibility with Kronecker products

The vectorization is frequently used together with the Kronecker product to express matrix multiplication as a linear transformation on matrices. In particular,$$\backslash operatorname(ABC)\; =\; (C^\backslash mathrm\backslash otimes\; A)\; \backslash operatorname(B)$$for matrices A, B, and C of dimensions k×l, l×m, and m×n.^[1] For example, if

\operatorname{ad} _{A(X)} = AX - XA (the adjoint endomorphism of the Lie algebra of all n×n matrices with complex entries), then \operatorname{vec}(\operatorname{ad} _{A(X))} = (I _{n ⊗} A - A ^{T} ⊗ I _{n} ) vec (X) , where I _{n} is the n×n identity matrix.

There are two other useful formulations:

More generally, it has been shown that vectorization is a self-adjunction in the monoidal closed structure of any category of matrices.^[2]

Compatibility with Hadamard products

Vectorization is an algebra homomorphism from the space of matrices with the Hadamard (entrywise) product to Cⁿ² with its Hadamard product:$$\backslash operatorname(A\; \backslash circ\; B)\; =\; \backslash operatorname(A)\; \backslash circ\; \backslash operatorname(B)\; .$$

Compatibility with inner products

Vectorization is a unitary transformation from the space of n×n matrices with the Frobenius (or Hilbert–Schmidt) inner product to Cⁿ²:$$\backslash operatorname(A^\backslash dagger\; B)\; =\; \backslash operatorname(A)^\backslash dagger\; \backslash operatorname(B),$$where the superscript ^† denotes the conjugate transpose.

Vectorization as a linear sum

The matrix vectorization operation can be written in terms of a linear sum. Let X be an matrix that we want to vectorize, and let e_i be the i-th canonical basis vector for the n-dimensional space, that is $\backslash mathbf\_i=\backslash left[0,\backslash dots,0,1,0,\backslash dots,0\backslash right]^\backslash mathrm$. Let B_i be a block matrix defined as follows:$$\backslash mathbf\_i\; =\; \backslash begin\backslash mathbf\; \backslash \backslash \backslash vdots\; \backslash \backslash \backslash mathbf\; \backslash \backslash \backslash mathbf\_m\; \backslash \backslash \backslash mathbf\; \backslash \backslash \backslash vdots\; \backslash \backslash \backslash mathbf\backslash end=\; \backslash mathbf\_i\; \backslash otimes\; \backslash mathbf\_m$$

B_i consists of n block matrices of size, stacked column-wise, and all these matrices are all-zero except for the i-th one, which is a identity matrix I_m.

Then the vectorized version of X can be expressed as follows:$$\backslash operatorname(\backslash mathbf)\; =\; \backslash sum\_^n\; \backslash mathbf\_i\; \backslash mathbf\; \backslash mathbf\_i$$

Multiplication of X by e_i extracts the i-th column, while multiplication by B_i puts it into the desired position in the final vector.

Alternatively, the linear sum can be expressed using the Kronecker product:$$\backslash operatorname(\backslash mathbf)\; =\; \backslash sum\_^n\; \backslash mathbf\_i\; \backslash otimes\; \backslash mathbf\; \backslash mathbf\_i$$

Half-vectorization

For a symmetric matrix A, the vector vec(A) contains more information than is strictly necessary, since the matrix is completely determined by the symmetry together with the lower triangular portion, that is, the entries on and below the main diagonal. For such matrices, the half-vectorization is sometimes more useful than the vectorization. The half-vectorization, vech(A), of a symmetric matrix A is the column vector obtained by vectorizing only the lower triangular part of A:$$\backslash operatorname(A)\; =\; [A\_\{1,1\},\; \backslash ldots,\; A\_\{n,1\},\; A\_\{2,2\},\; \backslash ldots,\; A\_\{n,2\},\; \backslash ldots,\; A\_\{n-1,n-1\},\; A\_\{n,n-1\},\; A\_\{n,n\}]^\backslash mathrm.$$

For example, for the 2×2 matrix

A = \begin{bmatrix} a & b \ b & d \end{bmatrix} , the half-vectorization is \operatorname{vech}(A) = \begin{bmatrix} a \ b \ d \end{bmatrix} .

There exist unique matrices transforming the half-vectorization of a matrix to its vectorization and vice versa called, respectively, the duplication matrix and the elimination matrix.

Programming language

Programming languages that implement matrices may have easy means for vectorization.In Matlab/GNU Octave a matrix A can be vectorized by A(:).GNU Octave also allows vectorization and half-vectorization with vec(A) and vech(A) respectively. Julia has the vec(A) function as well.In Python NumPy arrays implement the flatten method,^[1] while in R the desired effect can be achieved via the c or as.vector functions. In R, function vec of package 'ks' allows vectorization and function vech implemented in both packages 'ks' and 'sn' allows half-vectorization.^{[3] [4] [5]}

Applications

Vectorization is used in matrix calculus and its applications in establishing e.g., moments of random vectors and matrices, asymptotics, as well as Jacobian and Hessian matrices.^[6] It is also used in local sensitivity and statistical diagnostics.^[7]

See also

• Duplication and elimination matrices
• Voigt notation
• Packed storage matrix
• Column-major order
• Matricization

Notes and References

1. The identity for row-major vectorization is \operatorname{vec}(ABC) = (A ⊗ C ^{T)\operatorname{vec}(B)} .
2. H. D. . Macedo . J. N. . Oliveira . Typing Linear Algebra: A Biproduct-oriented Approach . Science of Computer Programming . 78 . 11 . 2013 . 2160–2191 . 10.1016/j.scico.2012.07.012 . 1312.4818 . 9846072 .
3. Web site: Tarn . Duong . 2018 . ks: Kernel Smoothing . R package version 1.11.0 .
4. Web site: Adelchi . Azzalini . 2017 . The R package 'sn': The Skew-Normal and Related Distributions such as the Skew-t . R package version 1.5.1 .
5. Book: Vinod, Hrishikesh D. . Hands-on Matrix Algebra Using R: Active and Motivated Learning with Applications . Singapore . World Scientific . 2011 . 978-981-4313-69-8 . Simultaneous Reduction and Vec Stacking . 233–248 . . https://books.google.com/books?id=oXzkJwutS1UC&pg=PA233 .
6. Book: Magnus . Jan . Neudecker . Heinz . Matrix differential calculus with applications in statistics and econometrics . John Wiley . New York . 2019 . 9781119541202 .
7. Liu. Shuangzhe. Leiva. Victor. Zhuang. Dan. Ma. Tiefeng. Figueroa-Zúñiga. Jorge I.. March 2022. Matrix differential calculus with applications in the multivariate linear model and its diagnostics . Journal of Multivariate Analysis. en. 188. 104849. 10.1016/j.jmva.2021.104849 . free.