In multivariate statistics, if
ε
is a vector of
n
random variables, and
Λ
is an
n
-dimensional symmetric matrix, then the scalar quantity
ε
T
Λ
ε
is known as a quadratic form in
ε
.
It can be shown that
E
[
ε
T
Λ
ε
]
=
tr
[
Λ
Σ
]
+
μ
T
Λ
μ
where
μ
and
Σ
are the expected value and variance-covariance matrix of
ε
, respectively, and tr denotes the trace of a matrix. This result only depends on the existence of
μ
and
Σ
; in particular, normality of
ε
is not required.
A book treatment of the topic of quadratic forms in random variables is
Since the quadratic form is a scalar quantity
E
[
ε
T
Λ
ε
]
=
tr
(
E
[
ε
T
Λ
ε
]
)
. Since the trace operator is a linear combination of the components of the matrix, it therefore follows from the linearity of the expectation operator that
tr
(
E
[
ε
T
Λ
ε
]
)
=
E
[
tr
(
ε
T
Λ
ε
)
]
.
Next, by the cyclic property of the trace operator,
E
[
tr
(
ε
T
Λ
ε
)
]
=
E
[
tr
(
Λ
ε
ε
T
)
]
.
Another application of linearity of expectation tells us that
E
[
tr
(
Λ
ε
ε
T
)
]
=
tr
(
Λ
E
(
ε
ε
T
)
)
.
A standard property of variances then tells us that this is
tr
(
Λ
(
Σ
+
μ
μ
T
)
)
.
Applying the cyclic property of the trace operator again, we get
tr
(
Λ
Σ
)
+
tr
(
Λ
μ
μ
T
)
=
tr
(
Λ
Σ
)
+
tr
(
μ
T
Λ
μ
)
=
tr
(
Λ
Σ
)
+
μ
T
Λ
μ
.
In general, the variance of a quadratic form depends greatly on the distribution of
ε
. However, if
ε
does follow a multivariate normal distribution, the variance of the quadratic form becomes particularly tractable. Assume for the moment that
Λ
is a symmetric matrix. Then,
var
[
ε
T
Λ
ε
]
=
2
tr
[
Λ
Σ
Λ
Σ
]
+
4
μ
T
Λ
Σ
Λ
μ
In fact, this can be generalized to find the covariance between two quadratic forms on the same
ε
(once again,
Λ
1
and
Λ
2
must both be symmetric):
cov
[
ε
T
Λ
1
ε
,
ε
T
Λ
2
ε
]
=
2
tr
[
Λ
1
Σ
Λ
2
Σ
]
+
4
μ
T
Λ
1
Σ
Λ
2
μ
Some texts incorrectly state that the above variance or covariance results hold without requiring
Λ
to be symmetric. The case for general
Λ
can be derived by noting that
ε
T
Λ
T
ε
=
ε
T
Λ
ε
so
ε
T
Λ
~
ε
=
ε
T
(
Λ
+
Λ
T
)
ε
/
2
But this is a quadratic form in the symmetric matrix
Λ
~
=
(
Λ
+
Λ
T
)
/
2
, so the mean and variance expressions are the same, provided
Λ
is replaced by
Λ
~
therein.
In the setting where one has a set of observations
y
and an operator matrix
H
, then the residual sum of squares can be written as a quadratic form in
y
:
RSS
=
y
T
(
I
−
H
)
T
(
I
−
H
)
y
.
For procedures where the matrix
H
is symmetric and idempotent, and the errors are Gaussian with covariance matrix
σ
2
I
,
RSS
/
σ
2
has a chi-squared distribution with
k
degrees of freedom and noncentrality parameter
λ
, where
k
=
tr
[
(
I
−
H
)
T
(
I
−
H
)
]
λ
=
μ
T
(
I
−
H
)
T
(
I
−
H
)
μ
/
2
may be found by matching the first two central moments of a noncentral chi-squared random variable to the expressions given in the first two sections. If
H
y
estimates
μ
with no bias, then the noncentrality
λ
is zero and
RSS
/
σ
2
follows a central chi-squared distribution.
