Squared deviations from the mean

Introduction

An understanding of the computations involved is greatly enhanced by a study of the statistical value:

E ⁡ ( X 2 ) .

It is well known that for a random variable X with mean μ and variance σ 2 ,

σ 2 = E ⁡ ( X 2 ) − μ 2 .

Therefore,

E ⁡ ( X 2 ) = σ 2 + μ 2 .

From the above, the following are easily derived:

E ⁡ ( ∑ ( X 2 ) ) = n σ 2 + n μ 2 , E ⁡ ( ( ∑ X ) 2 ) = n σ 2 + n 2 μ 2 .

If Y ^ is a vector of n predictions, and Y is the vector of the true values, then the SSE of the predictor is

S S E = 1 2 ∑ i = 1 n ( Y i ^ − Y i ) 2 .

Sample variance

The sum of squared deviations needed to calculate sample variance (before deciding whether to divide by n or n − 1) is most easily calculated as

S = ∑ x 2 − ( ∑ x ) 2 n

From the two derived expectations above the expected value of this sum is

E ⁡ ( S ) = n σ 2 + n μ 2 − n σ 2 + n 2 μ 2 n

which implies

E ⁡ ( S ) = ( n − 1 ) σ 2 .

This effectively proves the use of the divisor n − 1 in the calculation of an unbiased sample estimate of σ².

Partition — analysis of variance

In the situation where data is available for k different treatment groups having size n_i where i varies from 1 to k, then it is assumed that the expected mean of each group is

E ⁡ ( μ i ) = μ + T i

and the variance of each treatment group is unchanged from the population variance σ 2 .

Under the Null Hypothesis that the treatments have no effect, then each of the T i will be zero.

It is now possible to calculate three sums of squares:

Individual

I = ∑ x 2 E ⁡ ( I ) = n σ 2 + n μ 2

Treatments

T = ∑ i = 1 k ( ( ∑ x ) 2 / n i ) E ⁡ ( T ) = k σ 2 + ∑ i = 1 k n i ( μ + T i ) 2 E ⁡ ( T ) = k σ 2 + n μ 2 + 2 μ ∑ i = 1 k ( n i T i ) + ∑ i = 1 k n i ( T i ) 2

Under the null hypothesis that the treatments cause no differences and all the T i are zero, the expectation simplifies to

E ⁡ ( T ) = k σ 2 + n μ 2 .

Combination

C = ( ∑ x ) 2 / n E ⁡ ( C ) = σ 2 + n μ 2

Sums of squared deviations

Under the null hypothesis, the difference of any pair of I, T, and C does not contain any dependency on μ , only σ 2 .

E ⁡ ( I − C ) = ( n − 1 ) σ 2 total squared deviations aka total sum of squares E ⁡ ( T − C ) = ( k − 1 ) σ 2 treatment squared deviations aka explained sum of squares E ⁡ ( I − T ) = ( n − k ) σ 2 residual squared deviations aka residual sum of squares

The constants (n − 1), (k − 1), and (n − k) are normally referred to as the number of degrees of freedom.

Example

In a very simple example, 5 observations arise from two treatments. The first treatment gives three values 1, 2, and 3, and the second treatment gives two values 4, and 6.

I = 1 2 1 + 2 2 1 + 3 2 1 + 4 2 1 + 6 2 1 = 66 T = ( 1 + 2 + 3 ) 2 3 + ( 4 + 6 ) 2 2 = 12 + 50 = 62 C = ( 1 + 2 + 3 + 4 + 6 ) 2 5 = 256 / 5 = 51.2

Giving

Total squared deviations = 66 − 51.2 = 14.8 with 4 degrees of freedom.Treatment squared deviations = 62 − 51.2 = 10.8 with 1 degree of freedom.Residual squared deviations = 66 − 62 = 4 with 3 degrees of freedom.

Two-way analysis of variance

The following hypothetical example gives the yields of 15 plants subject to two different environmental variations, and three different fertilisers.

Five sums of squares are calculated:

Finally, the sums of squared deviations required for the analysis of variance can be calculated.