In statistical classification the Bayes classifier minimizes the probability of misclassification.
Suppose a pair ( X , Y ) takes values in R d × { 1 , 2 , … , K } , where Y is the class label of X . This means that the conditional distribution of X, given that the label Y takes the value r is given by
X ∣ Y = r ∼ P r for
r = 1 , 2 , … , K where " ∼ " means "is distributed as", and where P r denotes a probability distribution.
A classifier is a rule that assigns to an observation X=x a guess or estimate of what the unobserved label Y=r actually was. In theoretical terms, a classifier is a measurable function C : R d → { 1 , 2 , … , K } , with the interpretation that C classifies the point x to the class C(x). The probability of misclassification, or risk, of a classifier C is defined as
R ( C ) = P { C ( X ) ≠ Y } . The Bayes classifier is
C Bayes ( x ) = argmax r ∈ { 1 , 2 , … , K } P ( Y = r ∣ X = x ) . In practice, as in most of statistics, the difficulties and subtleties are associated with modeling the probability distributions effectively—in this case, P ( Y = r ∣ X = x ) . The Bayes classifier is a useful benchmark in statistical classification.
The excess risk of a general classifier C (possibly depending on some training data) is defined as R ( C ) − R ( C Bayes ) . Thus this non-negative quantity is important for assessing the performance of different classification techniques. A classifier is said to be consistent if the excess risk converges to zero as the size of the training data set tends to infinity.
