Fisher's Exact Test Calculator
Test a 2 by 2 table of counts exactly, and get the p value, the odds ratio and the chance of the table you saw.
What it does
The chi-square test of independence is the workhorse for a table of counts, but it is only an approximation and it needs a decent sample size. Fisher's exact test is the alternative: it works out the probability of every table that could have produced the same row and column totals and adds up the ones at least as unlikely as the one you saw.
The NIST Dataplot documentation describes the test and the reason it exists. The test is based on the probability of obtaining a table more extreme than the observed table, it compares the frequency of the top left cell with the hypergeometric distribution, and it is typically used when the row and column totals are small. When they are large the chi-square test is accurate enough, and the exact test becomes slow.
The SciPy documentation spells out the mechanics with a worked example that this page reproduces. The null hypothesis is that the table came from the hypergeometric distribution with the same margins, the probability of each candidate table is the hypergeometric mass function, and the two sided p value adds up every table whose probability does not exceed the probability of the observed one. For the table 6, 2, 1, 4 that gives a p value of 0.10256.
This page runs that calculation on a table you type in. Enter the four counts, choose whether you want a two sided or a one sided test, and it returns the exact p value, the odds ratio, the probability of the table you entered and the total number of observations.
The odds ratio is the measure most people actually want from a table like this, so the page reports it alongside the test. It is the ratio of the odds of the outcome in the first row to the odds in the second row, and it is a descriptive number rather than a hypothesis test, so read it on its own terms.
How to use
- Lay out your table as two rows and two columns and enter the four counts.
- Choose the alternative hypothesis. Use the two sided test unless you set up the question in advance as a one sided one.
- Read the exact p value and compare it with the level you care about, usually 0.05.
- Read the odds ratio, which says how much more likely the outcome is in the first row than in the second.
- Read the probability of the table you entered, which is how likely that exact table is if the rows and columns are independent.
- Check the total observations. Very small tables give large p values, and that is the test working correctly rather than failing.
Formula
- a, b, c, d
- The four observed counts, read row by row.
- M
- Total number of observations.
- n and N
- Row and column totals that stay fixed across the null distribution.
- x
- The top left cell of a candidate table.
- p value
- Probability of a table at least as unlikely as the observed one.
The two sided p value depends on how the tails are defined, and the rule used here is the one in the SciPy documentation: every table whose probability is at most the probability of the observed table is counted, with a small tolerance for floating point. A table with a zero cell gives an infinite odds ratio, and a table with two zero cells in the same row or column gives one that cannot be estimated, which the page reports rather than guessing.
Worked example
Example: the table 6, 2, 1, 4
Defaults: a two sided test on the table with 6, 2, 1 and 4 in it.
- The margins are M = 13 observations, n = 8 in the top row and N = 7 in the left column.
- The observed table has a top left cell of 6, so its probability is C(7, 6) x C(6, 2) divided by C(13, 8) = 105 / 1287 = 0.0816.
- The candidate tables have top left cells from 2 to 7, with probabilities 0.0163, 0.1632, 0.4079, 0.3263, 0.0816 and 0.0047.
- The tables at least as unlikely as the observed one are x = 2, x = 6 and x = 7.
- The two sided p value is 0.0163 + 0.0816 + 0.0047 = 0.1026.
- The one sided p value for a larger top left cell is 0.0816 + 0.0047 = 0.0862, and for a smaller one it is 0.9953.
- The odds ratio is (6 x 4) / (2 x 1) = 12, and the tool shows 0.1026, 12.0000, 0.0816 and 13 observations.
Reference table
| Top left cell | Table | Probability |
|---|---|---|
| 2 | 2, 6, 5, 0 | 0.0163 |
| 3 | 3, 5, 4, 1 | 0.1632 |
| 4 | 4, 4, 3, 2 | 0.4079 |
| 5 | 5, 3, 2, 3 | 0.3263 |
| 6 | 6, 2, 1, 4 | 0.0816 |
| 7 | 7, 1, 0, 5 | 0.0047 |
Limits
- The test works on counts, so the cells have to be whole numbers of observations.
- Very large tables slow the calculation down, and the chi-square test is accurate enough above a few dozen observations anyway.
- The p value depends on the null hypothesis of independence, so it says nothing about causation.
- The odds ratio describes the table as it is, without any adjustment for confounding or for how the sample was taken.
Common mistakes
- Reading a p value above 0.05 as proof that there is no effect. It only means the data are compatible with independence.
- Choosing the one sided test after seeing the table. The side has to be fixed before the data are looked at.
- Using Fisher's exact test on percentages or measurements instead of counts.
- Forgetting that the odds ratio is not a risk ratio. With 6 and 1 against 2 and 4 the odds ratio is 12 while the risk ratio is 6.
- Expecting a small table to give a small p value. With few observations the test has little power and p values stay large.
Try it yourself
Quick checks. Answers sit tucked under each question.
What is the p value for the table 6, 2, 1, 4?
0.1026 for a two sided test, which is not significant at the 5 % level. The tables at least as unlikely are the ones with a top left cell of 2, 6 and 7.
What is the one sided p value if the top left cell is larger than expected?
0.0862, which is the sum of the probabilities of the tables with a top left cell of 6 and 7.
What is the odds ratio of that table?
12, from 6 x 4 divided by 2 x 1. The odds of the outcome in the top row are six times one, and in the bottom row two times one, so the ratio is 12.
Questions
When do I need Fisher's exact test?
When the cell counts are small, usually below five in a row or column total. Above that the chi-square test is accurate and much faster.
How is the exact p value different from the chi-square p value?
The exact test enumerates the tables that could have happened and adds up their probabilities, so it makes no approximation. The chi-square test substitutes a continuous distribution for that enumeration, which is why it needs a large sample.
What is the odds ratio?
The ratio of the odds of the outcome in the first row to the odds in the second row, which is (a x d) / (b x c). A value above 1 means the outcome is more likely in the first row.
What does a two sided p value include?
Every table with the same margins whose probability is at most the probability of the table you observed, whether it is more extreme in the upper tail or in the lower one.
Does a small p value mean the two variables are related?
It means the data are unlikely under the assumption that they are independent. Whether that reflects a real relationship is a separate question that depends on how the data were collected.
What if a cell holds zero?
The odds ratio becomes infinite if the zero is in the top row, because the odds in that row are infinite, and it cannot be estimated at all if the zero is in the same row or column as another zero.
Sources
- NIST Dataplot: Fisher exact test of independence for a two way table
- SciPy documentation: scipy.stats.fisher_exact and the worked example
- R manual: fisher.test for count data