Quadratic form (statistics) explained

In multivariate statistics, if

\varepsilon is a vector of n random variables, and Λ is an n -dimensional symmetric matrix, then the scalar quantity \varepsilon ^{TΛ\varepsilon} is known as a quadratic form in \varepsilon .

Expectation

It can be shown that^[1]

\operatorname{E}\left[\varepsilon ^{TΛ\varepsilon\right]=\operatorname{tr}\left[Λ} \Sigma\right] + \mu ^{TΛ\mu}

where

\mu and \Sigma are the expected value and variance-covariance matrix of \varepsilon , respectively, and tr denotes the trace of a matrix. This result only depends on the existence of \mu and \Sigma ; in particular, normality of \varepsilon is not required.

A book treatment of the topic of quadratic forms in random variables is that of Mathai and Provost.^[2]

Proof

Since the quadratic form is a scalar quantity,

\varepsilon ^{TΛ\varepsilon} = \operatorname{tr}(\varepsilon ^{TΛ\varepsilon)} .

Next, by the cyclic property of the trace operator,

\operatorname{E}[\operatorname{tr}(\varepsilon ^{TΛ\varepsilon)]} = \operatorname{E}[\operatorname{tr}( Λ \varepsilon\varepsilon ^{T)].}

Since the trace operator is a linear combination of the components of the matrix, it therefore follows from the linearity of the expectation operator that

\operatorname{E}[\operatorname{tr}( Λ \varepsilon\varepsilon ^{T)]} = \operatorname{tr}( Λ \operatorname{E}(\varepsilon\varepsilon ^{T)).}

A standard property of variances then tells us that this is

\operatorname{tr}( Λ (\Sigma + \mu \mu ^{T)).}

Applying the cyclic property of the trace operator again, we get

\operatorname{tr}( Λ \Sigma) + \operatorname{tr}( Λ \mu \mu ^{T)} = \operatorname{tr}( Λ \Sigma) + \operatorname{tr}(\mu ^{TΛ\mu)} = \operatorname{tr}( Λ \Sigma) + \mu ^{TΛ\mu.}

Variance in the Gaussian case

In general, the variance of a quadratic form depends greatly on the distribution of

\varepsilon . However, if \varepsilon does follow a multivariate normal distribution, the variance of the quadratic form becomes particularly tractable. Assume for the moment that Λ is a symmetric matrix. Then,\operatorname{var} \left[\varepsilon ^{TΛ\varepsilon\right]} = 2 \operatorname{tr}\left[ Λ \Sigma Λ \Sigma\right] + 4 \mu ^{TΛ\SigmaΛ\mu} .^[3]

In fact, this can be generalized to find the covariance between two quadratic forms on the same

\varepsilon (once again, Λ _{1} and Λ _{2} must both be symmetric):\operatorname{cov}\left[\varepsilon ^{T Λ} _{2 \varepsilon\right] =2 \operatorname{tr}\left[ Λ} _{1\SigmaΛ} _{2} \Sigma\right] + 4 \mu ^{T Λ} _{1 \Sigma Λ} _{2\mu} .^[4]

In addition, a quadratic form such as this follows a generalized chi-squared distribution.

Computing the variance in the non-symmetric case

The case for general

Λ can be derived by noting that\varepsilon ^{TΛ} ^{T\varepsilon=\varepsilon} ^{TΛ\varepsilon}

so

\varepsilon ^{T\tilde{Λ}\varepsilon=\varepsilon} ^{T\left(Λ+Λ} ^{T\right)\varepsilon/2}

is a quadratic form in the symmetric matrix

\tilde{ Λ } = \left( Λ+Λ ^{T\right)/2} , so the mean and variance expressions are the same, provided Λ is replaced by \tilde{ Λ } therein.

Examples of quadratic forms

In the setting where one has a set of observations

y and an operator matrix H , then the residual sum of squares can be written as a quadratic form in y :rm{RSS}=y ^{T(I-H)} ^{T} (I - H)y.

For procedures where the matrix

H is symmetric and idempotent, and the errors are Gaussian with covariance matrix \sigma ^{2I} , rm{RSS}/\sigma ^{2} has a chi-squared distribution with k degrees of freedom and noncentrality parameter λ , wherek = \operatorname{tr}\left[(I - H) ^{T(I-H)\right]} λ= \mu ^{T(I-H)} ^{T(I-H)\mu/2}

may be found by matching the first two central moments of a noncentral chi-squared random variable to the expressions given in the first two sections. If

Hy estimates \mu with no bias, then the noncentrality λ is zero and rm{RSS}/\sigma ^{2} follows a central chi-squared distribution.

See also

• Quadratic form
• Covariance matrix
• Matrix representation of conic sections

Notes and References

1. Web site: Bates. Douglas. Quadratic Forms of Random Variables. STAT 849 lectures. August 21, 2011.
2. Book: Quadratic Forms in Random Variables . CRC Press . Mathai, A. M. . Provost, Serge B. . amp . 1992 . 424 . 978-0824786915.
3. Book: Rencher, Alvin C. . Linear models in statistics . 2008 . Wiley-Interscience . Schaalje . G. Bruce. . 9780471754985 . 2nd . Hoboken, N.J. . 212120778.
4. Book: Graybill . Franklin A. . Matrices with applications in statistics . Belmont, Calif. . Wadsworth . 0534980384 . 367 . 2..