Holm–Bonferroni method

Motivation

When considering several hypotheses, the problem of multiplicity arises: the more hypotheses we check, the higher the probability of a Type I error (false positive). The Holm–Bonferroni method is one of many approaches that control the family-wise error rate (the probability that one or more Type I errors will occur) by adjusting the rejection criteria of each of the individual hypotheses or comparisons.

Formulation

The method is as follows:

The Holm–Bonferroni method ensures that this method will control the F W E R ≤ α , where F W E R is the familywise error rate

Proof

Holm-Bonferroni controls the FWER as follows. Let H ( 1 ) … H ( m ) be a family of hypotheses, and P ( 1 ) ≤ P ( 2 ) ≤ … ≤ P ( m ) be the sorted p-values. Let I 0 be the set of indices corresponding to the (unknown) true null hypotheses, having m 0 members.

Let us assume that we wrongly reject a true hypothesis. We have to prove that the probability of this event is at most α . Let h be the first rejected true hypothesis (first in the ordering given by the Bonferroni–Holm test). So h − 1 is the last false hypothesis rejected and h − 1 + m 0 ≤ m . From there, we get 1 m − h + 1 ≤ 1 m 0 (1). Since h is rejected we have P ( h ) ≤ α m + 1 − h by definition of the test. Using (1), the right hand side is at most α m 0 . Thus, if we wrongly reject a true hypothesis, there has to be a true hypothesis with P-value at most α m 0 .

So let us define A = { P i ≤ α m 0  for some  i ∈ I 0 } . Whatever the (unknown) set of true hypotheses I 0 is, we have Pr ( A ) ≤ α (by the Bonferroni inequalities). Therefore, the probability to reject a true hypothesis is at most α .

Alternative proof

The Holm–Bonferroni method can be viewed as closed testing procedure, with Bonferroni method applied locally on each of the intersections of null hypotheses. As such, it controls the familywise error rate for all the k hypotheses at level α in the strong sense. Each intersection is tested using the simple Bonferroni test.

It is a shortcut procedure since practically the number of comparisons to be made equal to m or less, while the number of all intersections of null hypotheses to be tested is of order 2 m .

The closure principle states that a hypothesis H i in a family of hypotheses H 1 , . . . , H m is rejected - while controlling the family-wise error rate of α - if and only if all the sub-families of the intersections with H i are controlled at level of family-wise error rate of α .

In Holm-Bonferroni procedure, we first test H ( 1 ) . If it is not rejected then the intersection of all null hypotheses ⋂ i = 1 m H i is not rejected too, such that there exist at least one intersection hypothesis for each of elementary hypotheses H 1 , . . . , H m that is not rejected, thus we reject none of the elementary hypotheses.

If H ( 1 ) is rejected at level α / m then all the intersection sub-families that contain it are rejected too, thus H ( 1 ) is rejected. This is because P ( 1 ) is the smallest in each one of the intersection sub-families and the size of the sub-families is the most m , such that the Bonferroni threshold larger than α / m .

The same rationale applies for H ( 2 ) . However, since H ( 1 ) already rejected, it sufficient to reject all the intersection sub-families of H ( 2 ) without H ( 1 ) . Once P ( 2 ) ≤ α / ( m − 1 ) holds all the intersections that contains H ( 2 ) are rejected.

The same applies for each 1 ≤ i ≤ m .

Example

Consider four null hypotheses H 1 , . . . , H 4 with unadjusted p-values p 1 = 0.01 , p 2 = 0.04 , p 3 = 0.03 and p 4 = 0.005 , to be tested at significance level α = 0.05 . Since the procedure is step-down, we first test H 4 = H ( 1 ) , which has the smallest p-value p 4 = p ( 1 ) = 0.005 . The p-value is compared to α / 4 = 0.0125 , the null hypothesis is rejected and we continue to the next one. Since p 1 = p ( 2 ) = 0.01 < 0.0167 = α / 3 we reject H 1 = H ( 2 ) as well and continue. The next hypothesis H 3 is not rejected since p 3 = p ( 3 ) = 0.03 > 0.025 = α / 2 . We stop testing and conclude that H 1 and H 4 are rejected and H 2 and H 3 are not rejected while controlling the familywise error rate at level α = 0.05 . Note that even though p 2 = p ( 4 ) = 0.04 < 0.05 = α applies, H 2 is not rejected. This is because the testing procedure stops once a failure to reject occurs.

Holm–Šidák method

When the hypothesis tests are not negatively dependent, it is possible to replace α m , α m − 1 , . . . , α 1 with:

resulting in a slightly more powerful test.

Weighted version

Let P ( 1 ) , . . . , P ( m ) be the ordered unadjusted p-values. Let H ( i ) , 0 ≤ w ( i ) correspond to P ( i ) . Reject H ( i ) as long as

Adjusted p-values

The adjusted p-values for Holm–Bonferroni method are:

In the earlier example, the adjusted p-values are p ~ 1 = 0.03 , p ~ 2 = 0.06 , p ~ 3 = 0.06 and p ~ 4 = 0.02 . Only hypotheses H 1 and H 4 are rejected at level α = 0.05 .

The weighted adjusted p-values are:

A hypothesis is rejected at level α if and only if its adjusted p-value is less than α. In the earlier example using equal weights, the adjusted p-values are 0.03, 0.06, 0.06, and 0.02. This is another way to see that using α = 0.05, only hypotheses one and four are rejected by this procedure.

Alternatives and usage

The Holm–Bonferroni method is uniformly more powerful than the classic Bonferroni correction. There are other methods for controlling the family-wise error rate that are more powerful than Holm-Bonferroni.

In the Hochberg procedure, rejection of H ( 1 ) … H ( k ) is made after finding the maximal index k such that P ( k ) ≤ α m + 1 − k . Thus, The Hochberg procedure is more powerful by construction. However, the Hochberg procedure requires the hypotheses to be independent or under certain forms of positive dependence, whereas Holm-Bonferroni can be applied without such assumptions.

A similar step-up procedure is the Hommel procedure.

Naming

Carlo Emilio Bonferroni did not take part in inventing the method described here. Holm originally called the method the "sequentially rejective Bonferroni test", and it became known as Holm-Bonferroni only after some time. Holm's motives for naming his method after Bonferroni are explained in the original paper: "The use of the Boole inequality within multiple inference theory is usually called the Bonferroni technique, and for this reason we will call our test the sequentially rejective Bonferroni test."