Bonferroni Correction Calculator
Correct a family of p values for the number of tests, with the Bonferroni threshold and the step down Holm version.
What it does
Run twenty tests at the 5 percent level and the odds of at least one false positive are not 5 percent, they are much more than that. The correction for that is one of the oldest ideas in statistics, and it is still the first thing you reach for when a study reports a long list of p values.
The University of Chicago lecture notes behind this page set out the problem and the fix. A familywise error is when one of several null hypotheses is wrongly rejected, and the familywise error rate is the chance of that happening. By the Bonferroni inequality the rate is at most the sum of the individual error rates, so if each test is run at alpha over k, the family as a whole stays at alpha.
The practical form of that is what the notes call the comparisonwise p value: a test is rejected when its p value is below alpha over k. The same notes point out the weakness, that Bonferroni works well when k is small and gets too conservative beyond about ten tests, and that the actual error rate is usually much lower than the bound.
The step down version, due to Holm, fixes most of that, and the Penn State notes give the recipe. Sort the p values, compare the smallest with alpha over m, then the next with alpha over m minus one, and keep going until one fails. The University of Dundee notes add the step up version, the Benjamini-Hochberg procedure, which controls a different error rate altogether.
The limits are worth stating. The correction does not make the tests more independent, it only controls the chance of at least one false positive across the family. And it says nothing about the many other ways a study with dozens of comparisons can mislead, which is why the planning stage matters more than the correction.
How to use
- Paste in the p values from the family of tests, separated by commas.
- Set the significance level for the family as a whole, which is 0.05 in most work.
- Read the Bonferroni threshold, which is the level each individual test has to beat.
- Read the adjusted p values, which are the ones you can compare directly with your level.
- Read the Holm step down values, which are the ones to use when you want more power without losing control of the familywise error.
- Count how many tests fall below the threshold and report that count, rather than picking out the ones you like.
Formula
- k
- Number of tests in the family.
- alpha
- Significance level wanted for the family as a whole.
- p
- Raw p value from one test.
- FWER
- Familywise error rate, the chance of at least one false positive across the whole family.
The Bonferroni adjustment is the crude version and it is always at least as conservative as the Holm step down procedure, which uses the fact that once a test has been rejected the remaining ones no longer all have to be protected. The Benjamini-Hochberg procedure controls the false discovery rate instead, the expected share of the rejected hypotheses that are wrong.
Worked example
Example: six comparisons with a level of 0.05
Defaults: six p values of 0.13356, 0.00159, 1.91e-8, 0.0854, 1.19e-5 and 0.00486, with a significance level of 0.05.
- There are 6 tests, so the Bonferroni threshold is 0.05 divided by 6, which is 0.008333.
- The raw p values are 0.13356, 0.00159, 1.91e-8, 0.0854, 1.19e-5 and 0.00486.
- Four of them are below 0.008333, namely the second, the third, the fifth and the sixth, which is what the University of Chicago notes find in their worked example.
- The Bonferroni adjusted values are those p values times 6, capped at 1, so 0.80136, 0.00954, 1.146e-7, 0.5124, 7.14e-5 and 0.02916.
- The Holm step down values start with the smallest p value times 6, then the next times 5, and take a running maximum, which gives 0.1708, 0.00636, 1.146e-7, 0.1708, 5.95e-5 and 0.01458.
- Holm rejects one more result than plain Bonferroni, at 0.01458 against 0.02916, which is the extra power that the step down version buys.
- The tool shows a threshold of 0.008333, the two sets of adjusted values, 4 of 6 tests significant and a smallest raw p value of 1.91e-8.
Reference table
| Number of tests | Threshold | Of the six results, below the threshold | Cost in power |
|---|---|---|---|
| 1 | 0.050000 | 4 of 6 | none |
| 3 | 0.016667 | 4 of 6 | small |
| 6 | 0.008333 | 4 of 6 | moderate |
| 20 | 0.002500 | 2 of 6 | large |
Limits
- The correction controls the chance of at least one false positive across the family and nothing else, so it does not make the tests independent.
- It is the crudest of the methods and it is the most conservative, so it throws away power that a better method would keep.
- It is only as good as the p values it is given, and a p value from a badly chosen test is not repaired by adjusting it.
- Nothing here covers the fact that unplanned comparisons, or looking at the data and then deciding what to test, inflate the error rate in a way no correction can undo.
Common mistakes
- Applying the correction to a family that was planned in advance, where it is usually not needed.
- Adjusting the p values and then also dividing the level by k, which corrects twice.
- Using the Bonferroni threshold and then reporting the raw p values, which is the same mistake in a different place.
- Treating a family of 50 tests as 50 independent chances, when the correction is about the family as a whole.
- Forgetting that a correction cannot rescue a test that was not worth doing in the first place.
Try it yourself
Quick checks. Answers sit tucked under each question.
How many of the six results survive the correction?
Four of the six. The second, third, fifth and sixth p values are below the 0.008333 threshold, which is the same set the University of Chicago notes identify in their worked example, and the first and fourth are left as they were.
What does the Holm step down version buy me?
One extra result in this example, because the sixth p value of 0.00486 has a Holm adjusted value of 0.01458 against a Bonferroni value of 0.02916. That is the extra power the step down version gets from not having to protect the tests that have already been rejected.
What happens if I run 1000 tests?
The threshold falls to 0.00005, and the Penn State notes show the power collapsing from 99 percent to 29 percent for an effect size of 2 with 10 observations per group. That is the price of the protection, and it is why the number of tests has to be planned rather than discovered.
Questions
What does the Bonferroni correction do?
It divides the significance level by the number of tests, so the chance of at least one false positive across the whole family stays at the level you asked for. The University of Chicago notes describe it as rejecting a null when the comparisonwise p value is less than alpha over k.
How do I report a corrected result?
Report the adjusted p value rather than the raw one, and say how many tests were in the family. Reporting only the results that survived the correction is what the correction is meant to stop.
What is the Holm step down procedure?
A more powerful version of the same idea. The Penn State notes give the algorithm: sort the p values, compare the smallest with alpha over m, then the next with alpha over m minus one, and keep going until one fails. The adjusted p values are the running maximum of the multiplied values.
What is the Benjamini-Hochberg procedure?
A different target. Instead of controlling the chance of any false positive it controls the false discovery rate, the expected share of the rejected results that are wrong, which is a looser target and much more powerful. The Dundee notes give a table comparing the two.
Do I need a correction at all?
If the tests were planned before the data was seen and the family is small, many researchers argue that no correction is needed. If the family was decided after looking at the data, the correction is the least you can do.
Does the correction make the p values more honest?
No. It only controls one kind of error. A p value that came from a badly designed test stays bad, and no adjustment of its size changes what went wrong.
Sources
- University of Chicago statistics lecture notes, chapter 5: multiple comparisons and Bonferroni's method
- Penn State University STAT 555 notes: controlling the familywise error rate
- University of Dundee lecture notes: multiple test corrections, Holm and Benjamini-Hochberg