Good–Turing frequency estimation is a statistical technique for estimating the probability of encountering an object of a hitherto unseen species, given a set of past observations of objects from different species. (In drawing balls from an urn, the 'objects' would be balls and the 'species' would be the distinct colors of the balls (finite but unknown in number). After drawing
Contents
Historical background
Good–Turing frequency estimation was developed by Alan Turing and his assistant I. J. Good as part of their efforts at Bletchley Park to crack German ciphers for the Enigma machine during World War II. Turing at first modeled the frequencies as a multinomial distribution, but found it inaccurate. Good developed smoothing algorithms to improve the estimator's accuracy.
The discovery was recognized as significant when published by Good in 1953, but the calculations were difficult so it was not used as widely as it might have been. The method even gained some literary fame due to the Robert Harris novel Enigma.
In the 1990s, Geoffrey Sampson worked with William A. Gale of AT&T, to create and implement a simplified and easier-to-use variant of the Good–Turing method described below.
The method
First notation and some required data structures are defined:
For example,
The first step in the calculation is to find an estimate of the total probability of unseen species. This estimate is:
The next step is to find an estimate of probability for species which were seen r times. For a single species this estimate is:
To estimate a probability of encountering any species from this group (i.e., the group of species seen r times) one can use the following formula:
Here, the notation
We would like to make a plot of
and where q, r and t are consecutive subscripts having
The assumption of Good–Turing estimation is that the number of occurrence for each species follows a binomial distribution.
A simple linear regression is then fitted to the log–log plot. For small values of r it is reasonable to set