In statistics, the Bayesian information criterion (BIC) or Schwarz criterion (also SBC, SBIC) is a criterion for model selection among a finite set of models; the model with the lowest BIC is preferred. It is based, in part, on the likelihood function and it is closely related to the Akaike information criterion (AIC).
Contents
When fitting models, it is possible to increase the likelihood by adding parameters, but doing so may result in overfitting. Both BIC and AIC attempt to resolve this problem by introducing a penalty term for the number of parameters in the model; the penalty term is larger in BIC than in AIC.
The BIC was developed by Gideon E. Schwarz and published in a 1978 paper, where he gave a Bayesian argument for adopting it.
Definition
The BIC is formally defined as
where
The BIC is an asymptotic result derived under the assumptions that the data distribution is in an exponential family. That is, the integral of the likelihood function
For large
Properties
Limitations
The BIC criterion suffers from two main limitations
- the above approximation is only valid for sample size
n much larger than the numberk of parameters in the model. - the BIC cannot handle complex collections of models as in the variable selection (or feature selection) problem in high-dimension.
Gaussian special case
Under the assumption that the model errors or disturbances are independent and identically distributed according to a normal distribution and that the boundary condition that the derivative of the log likelihood with respect to the true variance is zero, this becomes (up to an additive constant, which depends only on n and not on the model):
where
which is a biased estimator for the true variance. In terms of the residual sum of squares (RSS) the BIC is
When testing multiple linear models against a saturated model, the BIC can be rewritten in terms of the deviance
where
When picking from several models, the one with the lowest BIC is preferred. The BIC is an increasing function of the error variance
The BIC generally penalizes free parameters more strongly than the Akaike information criterion, though it depends on the size of n and relative magnitude of n and k.
It is important to keep in mind that the BIC can be used to compare estimated models only when the numerical values of the dependent variable are identical for all estimates being compared. The models being compared need not be nested, unlike the case when models are being compared using an F-test or a likelihood ratio test.