In mathematical statistics, polynomial least squares refers to a broad range of statistical methods for estimating an underlying polynomial that describes observations. These methods include polynomial regression, curve fitting, linear regression, least squares, ordinary least squares, simple linear regression, linear least squares, approximation theory and method of moments. Polynomial least squares has applications in radar trackers, estimation theory, signal processing, statistics, and econometrics.
Contents
- Polynomial least squares estimate of a deterministic first degree polynomial corrupted with observation errors
- Definitions and assumptions
- Polynomial least squares and the orthogonality principle
- The empirically determined polynomial least squares output of a first degree polynomial corrupted with observation errors
- The weighting function describing the linear polynomial least squares system
- Empirically determined statistical moments
- Measuring or approximating the statistical variance of the random errors
- Properties of polynomial least squares modeled as a linear system
- The synergy of integrating polynomial least squares with statistical estimation theory
- References
Two common applications of polynomial least squares methods are approximating a low-degree polynomial that approximates a complicated function and estimating an assumed underlying polynomial from corrupted (also known as "noisy") observations. The former is commonly used in statistics and econometrics to fit a scatter plot with a first degree polynomial (that is, a line). The latter is commonly used in target tracking in the form of Kalman filtering, which is effectively a recursive implementation of polynomial least squares. Estimating an assumed underlying deterministic polynomial can be used in econometrics as well. In effect, both applications produce average curves as generalizations of the common average of a set of numbers, which is equivalent to zero degree polynomial least squares.
In the above applications, the term "approximate" is used when no statistical measurement or observation errors are assumed, as when fitting a scatter plot. The term "estimate", derived from statistical estimation theory, is used when assuming that measurements or observations of a polynomial are corrupted.
Polynomial least squares estimate of a deterministic first degree polynomial corrupted with observation errors
Assume the deterministic first degree polynomial equation
which is corrupted with an additive stochastic process
Given samples
Definitions and assumptions
(1) The term linearity in mathematics may be considered to take two forms that are sometimes confusing: a linear "system" or transformation (sometimes called an operator) and a linear equation. The term "function" is often used to describe both a system and an equation, which may lead to confusion. A linear system is defined by
where
(2) The error
(3) Polynomial least squares is modeled as a linear signal processing "system" which processes statistical inputs deterministically, the output being the linearly processed empirically determined statistical estimate, variance, and expected value.
(4) Polynomial least squares processing produces deterministic moments (analogous to mechanical moments), which may be considered as moments of sample statistics, but not of statistical moments.
Polynomial least squares and the orthogonality principle
Approximating a function
where hat (^) denotes the estimate and (J − 1) is the polynomial degree, can be performed by applying the orthogonality principle. The error
According to the orthogonality principle,
This can be described as the orthogonal projection of the data
The advantage of using orthogonal projection is that
The empirically determined polynomial least squares output of a first degree polynomial corrupted with observation errors
To fully determine the output of polynomial least squares, a weighting function describing the processing must first be structured and then the statistical moments can be computed.
The weighting function describing the linear polynomial least squares "system"
The weighting function
where N is the number of samples,
which represent the linear polynomial least squares "system" and describe its processing. The Greek letter
Empirically determined statistical moments
Applying
where
and
As linear functions of the random variables
Because the statistical expectation operator E[•] is a linear function and the sampled stochastic process errors
The statistical variance in
because
where
Carrying out the multiplications and summations in
Measuring or approximating the statistical variance of the random errors
In a hardware system, such as a tracking radar, the measurement noise variance
However, if polynomial least squares is used when the variance
As a result, to the first order approximation from the estimates
which goes to zero in the absence of the errors
As a result, the samples
Properties of polynomial least squares modeled as a linear "system"
(1) The empirical statistical variance
(2) The empirical statistical variance
(3) The empirical statistical variance
(4) There is an additional issue to be considered when the noise variance is not measurable: Independent of the polynomial least squares estimation, any new observations would be described by the variance
Thus, the polynomial least squares statistical estimation variance
(5) This concept also applies to higher degree polynomials. However, the weighting function
The synergy of integrating polynomial least squares with statistical estimation theory
Modeling polynomial least squares as a linear signal processing "system" creates the synergy of integrating polynomial least squares with statistical estimation theory to deterministically process samples of an assumed polynomial corrupted with a statistically described stochastic error ε. In the absence of the error ε, statistical estimation theory is irrelevant and polynomial least squares reverts back to the conventional approximation of complicated functions and scatter plots.