Tensor reshaping explained

In multilinear algebra, a reshaping of tensors is any bijection between the set of indices of an order-

M tensor and the set of indices of an order-L tensor, where L < M . The use of indices presupposes tensors in coordinate representation with respect to a basis. The coordinate representation of a tensor can be regarded as a multi-dimensional array, and a bijection from one set of indices to another therefore amounts to a rearrangement of the array elements into an array of a different shape. Such a rearrangement constitutes a particular kind of linear map between the vector space of order-M tensors and the vector space of order-L tensors.

Definition

Given a positive integer

M , the notation [M] refers to the set \{ 1 , ... , M \} of the first positive integers.

For each integer

m where 1 \le m \le M for a positive integer M , let V _{m} denote an I _{m} -dimensional vector space over a field F . Then there are vector space isomorphisms (linear maps)

where

\pi \in ak{S} _{M} is any permutation and ak{S} _{M} is the symmetric group on M elements. Via these (and other) vector space isomorphisms, a tensor can be interpreted in several ways as an order-L tensor where L \le M .

Coordinate representation

The first vector space isomorphism on the list above,

V _{1} ⊗ … ⊗ V _{M} \simeq F ^{I _{1}} ⊗ … ⊗ F ^{I _{M}} , gives the coordinate representation of an abstract tensor. Assume that each of the M vector spaces V _{m} has a basis \{ v ^{m,} _{1} v ^{m,} _{2} \ldots, v ^{m} _{I _{m}} \} . The expression of a tensor with respect to this basis has the form $$\backslash mathcal\; =\; \backslash sum\_^\backslash ldots\backslash sum\_^\; a\_\; v\_^1\; \backslash otimes\; v\_^2\; \backslash otimes\; \backslash cdots\; \backslash otimes\; v\_^,$$ where the coefficients a _{i _{1,i} _{2,\ldots,i} _{M}} are elements of F . The coordinate representation of l{A} is $$\backslash sum\_^\backslash ldots\backslash sum\_^\; a\_\; \backslash mathbf\_^1\; \backslash otimes\; \backslash mathbf\_^2\; \backslash otimes\; \backslash cdots\; \backslash otimes\; \backslash mathbf\_^M,$$where e ^{m} _{i} is the i ^{th} standard basis vector of F ^{I _{m}} . This can be regarded as a M-way array whose elements are the coefficients a _{i _{1,i} _{2,\ldots,i} _{M}} .

General flattenings

For any permutation

\pi \in ak{S} _{M} there is a canonical isomorphism between the two tensor products of vector spaces V _{1} ⊗ V _{2} ⊗ … ⊗ V _{M} and V _{\pi(1)} ⊗ V _{\pi(2)} ⊗ … ⊗ V _{\pi(M)} . Parentheses are usually omitted from such products due to the natural isomorphism between V _{i ⊗ (V} _{j ⊗} V _{k)} and (V _{i ⊗} V _{j) ⊗} V _{k} , but may, of course, be reintroduced to emphasize a particular grouping of factors. In the grouping,$$(V\_\; \backslash otimes\; \backslash cdots\; \backslash otimes\; V\_)\backslash otimes(V\_\; \backslash otimes\; \backslash cdots\; \backslash otimes\; V\_)\backslash otimes\backslash cdots\backslash otimes(V\_\; \backslash otimes\; \backslash cdots\; \backslash otimes\; V\_),$$there are L groups with r _{l-r} _{l-1} factors in the l ^{th} group (where r _{0=0} and r _{L=M} ).

Letting

S _{l=(\pi(r} _{l-1} +1 ),\pi(r _{l-1} +2 ),\ldots,\pi(r _{l))} for each l satisfying 1 \le l\le L , an (S _{1,S} _{2,\ldots,S} _{L)} -flattening of a tensor l{A} , denoted l{A} _{(S _{1,S} _{2,\ldots,S} _{L)}} , is obtained by applying the two processes above within each of the L groups of factors. That is, the coordinate representation of the l ^{th} group of factors is obtained using the isomorphism (V _{\pi(r _{l-1} +1 )} ⊗ V _{\pi(r _{l-1} +2 )} ⊗ … ⊗ V _{\pi(r _{l)}} )\simeq(F ^{I _{\pi(r _{l-1} +1 )}} ⊗ F ^{I _{\pi(r _{l-1} +2 )}} ⊗ … ⊗ F ^{I _{\pi(r _{l)}}} ) , which requires specifying bases for all of the vector spaces V _{k} . The result is then vectorized using a bijection \mu _{l:[I} ^{\pi(r _{l-1} +1 )} ] x [I _{\pi(r _{l-1} +2 )} ] x … x [I _{\pi(r _{l)}} ]\to[I _{S _{l}} ] to obtain an element of F ^{I _{S _{l}}} , where $I\_\; :=\; \backslash prod\_^\; I\_$, the product of the dimensions of the vector spaces in the l ^{th} group of factors. The result of applying these isomorphisms within each group of factors is an element of F ^{I _{S _{1}}} ⊗ … ⊗ F ^{I _{S _{L}}} , which is a tensor of order L .

Vectorization

By means of a bijective map

\mu : [I _{1]} x … x [I _{M]} \to [I _{1 …} I _{M]} , a vector space isomorphism between F ^{I _{1}} ⊗ … ⊗ F ^{I _{M}} and F ^{I _{1} … I _{M}} is constructed via the mapping e ^{1} _{i _{1}} ⊗ … e ^{m} _{i _{m}} ⊗ … ⊗ e ^{M} _{i _{M}} \mapsto e _{\mu(i _{1,i} _{2,\ldots,i} _{M)}} , where for every natural number i such that 1 \le i \le I _{1} … I _{M} , the vector e _{i} denotes the ith standard basis vector of F ^{i _{1} … i _{M}} . In such a reshaping, the tensor is simply interpreted as a vector in F ^{I _{1} … I _{M}} . This is known as vectorization, and is analogous to vectorization of matrices. A standard choice of bijection \mu is such that

which is consistent with the way in which the colon operator in Matlab and GNU Octave reshapes a higher-order tensor into a vector. In general, the vectorization of

l{A} is the vector [ a _{\mu ^{-1} (i)} ] ^{I _{1} … I _{M}} _{i =1} .

The vectorization of

l{A} denoted with vec( l{A}) or l{A} _{[:]} is an [S _{1,S} _{2]} -reshaping where S _{1} = ( 1 , 2 ,\ldots,M) and S _{2=\empty} .

Mode-m Flattening / Mode-m Matrixization

Let

l{A} \in F ^{I _{1}} ⊗ F ^{I _{2}} ⊗ … ⊗ F ^{I _{M}} be the coordinate representation of an abstract tensor with respect to a basis.Mode-m matrixizing (a.k.a. flattening) of l{A} is an [S _{1,} S _{2]} -reshaping in which S _{1} = (m) and S _{2} = ( 1 , 2 ,\ldots,m -1 ,m +1 ,\ldots,M) . Usually, a standard matrixizing is denoted by

$$\_\; =\; \backslash mathcal\_$$

This reshaping is sometimes called matrixizing, matricizing, flattening or unfolding in the literature. A standard choice for the bijections

\mu _{1, \mu} _{2} is the one that is consistent with the reshape function in Matlab and GNU Octave, namely

Definition Mode-m Matrixizing:$$[\{\backslash mathbf\; A\}\_\{[m]\}]\_\; =\; a\_,\; \backslash ;\backslash ;\; \backslash text\; j\; =\; i\_m\; \backslash text\; k=1+\backslash sum\_^M(i\_n\; -\; 1)\; \backslash prod\_^\; I\_l.$$The mode-m matrixizing of a tensor

{ l A} \in F ^{I _{1 x ...I} _{M}} , is defined as the matrix { A} _{[m]} \in F ^{I _{m} x (I _{1} ... I _{m-1} I _{m+1} ... I _{M)}} . As the parenthetical ordering indicates, the mode-m column vectors are arranged bysweeping all the other mode indices through their ranges,with smaller mode indexes varying more rapidly than larger ones; thu