In information geometry, the Fisher information metric is a particular Riemannian metric which can be defined on a smooth statistical manifold, i.e., a smooth manifold whose points are probability measures defined on a common probability space. It can be used to calculate the informational difference between measurements.
Contents
- Definition
- Relation to the KullbackLeibler divergence
- Relation to Ruppeiner geometry
- Change in entropy
- Relation to the JensenShannon divergence
- As Euclidean metric
- As FubiniStudy metric
- Continuously valued probabilities
- References
The metric is interesting in several respects. First, it can be understood to be the infinitesimal form of the relative entropy (i.e., the Kullback–Leibler divergence); specifically, it is the Hessian of the divergence. Alternately, it can be understood as the metric induced by the flat space Euclidean metric, after appropriate changes of variable. When extended to complex projective Hilbert space, it becomes the Fubini–Study metric; when written in terms of mixed states, it is the quantum Bures metric.
Considered purely as a matrix, it is known as the Fisher information matrix. Considered as a measurement technique, where it is used to estimate hidden parameters in terms of observed random variables, it is known as the observed information.
Definition
Given a statistical manifold with coordinates
The Fisher information metric then takes the form:
The integral is performed over all values x in X. The variable
When the probability is derived from the Gibbs measure, as it would be for any Markovian process, then
Substituting
To show that the equivalent form equals the above definition note that
and apply
Relation to the Kullback–Leibler divergence
Alternately, the metric can be obtained as the second derivative of the relative entropy or Kullback–Leibler divergence. To obtain this, one considers two probability distributions
with
The symmetric matrix
Relation to Ruppeiner geometry
The Ruppeiner metric and Weinhold metric arise as the thermodynamic limit of the Fisher information metric.
Change in entropy
The action of a curve on a Riemannian manifold is given by
The path parameter here is time t; this action can be understood to give the change in entropy of a system as it is moved from time a to time b. Specifically, one has
as the change in entropy. This observation has resulted in practical applications in chemical and processing industry: in order to minimize the change in entropy of a system, one should follow the minimum geodesic path between the desired endpoints of the process. The geodesic minimizes the entropy, due to the Cauchy–Schwarz inequality, which states that the action is bounded below by the length of the curve, squared.
Relation to the Jensen–Shannon divergence
The Fisher metric also allows the action and the curve length to be related to the Jensen–Shannon divergence. Specifically, one has
where the integrand dJSD is understood to be the infinitesimal change in the Jensen–Shannon divergence along the path taken. Similarly, for the curve length, one has
That is, the square root of the Jensen–Shannon divergence is just the Fisher metric (divided by the square root of 8).
As Euclidean metric
For a discrete probability space, that is, a probability space on a finite set of objects, the Fisher metric can be understood to simply be the Euclidean metric restricted to a positive "quadrant" of a unit sphere, after appropriate changes of variable.
Consider a flat, Euclidean space, of dimension N+1, paramterized by points
where the
the Euclidean metric may be written as
The superscript 'flat' is there to remind that, when written in coordinate form, this metric is with respect to the flat-space coordinate
An N-dimensional unit sphere embedded in (N + 1)-dimensional Euclidean space may be defined as
This embedding induces a metric on the sphere, it is inherited directly from the Euclidean metric on the ambient space. It takes exactly the same form as the above, taking care to ensure that the coordinates are constrained to lie on the surface of the sphere. This can be done, e.g. with the technique of Lagrange multipliers.
Consider now the change of variable
while the metric becomes
The last can be recognized as one-fourth of the Fisher information metric. To complete the process, recall that the probabilities are parametric functions of the manifold variables
or, in coordinate form, the Fisher information metric is:
where, as before,
The superscript 'fisher' is present to remind that this expression is applicable for the coordinates
When the random variable
As Fubini–Study metric
The above manipulations deriving the Fisher metric from the Euclidean metric can be extended to complex projective Hilbert spaces. In this case, one obtains the Fubini–Study metric. This should perhaps be no surprise, as the Fubini–Study metric provides the means of measuring information in quantum mechanics. The Bures metric, also known as the Helstrom metric, is identical to the Fubini–Study metric, although the latter is usually written in terms of pure states, as below, whereas the Bures metric is written for mixed states. By setting the phase of the complex coordinate to zero, one obtains exactly one-fourth of the Fisher information metric, exactly as above.
One begins with the same trick, of constructing a probability amplitude, written in polar coordinates, so:
Here,
is equivalently expressed by the idea the square amplitude be normalized:
When
The Fubini–Study metric, written in infinitesimal form, using quantum-mechanical bra–ket notation, is
In this notation, one has that
The expression
Inserting the above into the Fubini–Study metric gives:
Setting
The imaginary term is a symplectic form, it is the Berry phase or geometric phase. In index notation, the metric is:
Again, the first term can be clearly seen to be (one fourth of) the Fisher information metric, by setting
Continuously-valued probabilities
A slightly more formal, abstract definition can be given, as follows.
Let X be an orientable manifold, and let
The statistical manifold S(X) of X is defined as the space of all measures
Pick a point
Here,
In order for the integral to be well-defined, the space S(X) must have the Radon–Nikodym property, and more specifically, the tangent space is restricted to those vectors that are square-integrable. Square integrability is equivalent to saying that a Cauchy sequence converges to a finite value under the weak topology: the space contains it's limit points. Note that Hilbert spaces possess this property.
This definition of the metric can be seen to be equivalent to the previous, in several steps. First, one selects a submanifold of S(X) by considering only those measures
With some additional abuse of language, one notes that the exponential map provides a map from vectors in a tangent space to points in an underlying manifold. Thus, if