Five p-values and a false discovery rate

Idea. Benjamini-Hochberg compares the sorted p-values with a rising line of thresholds $\frac{k}{m}q$. It rejects everything up to the last p-value that falls below its threshold.

Step 1: thresholds. With $m=5$ and $q=0.05$, the thresholds are $0.01,0.02,0.03,0.04,0.05$.

Step 2: compare.

• $p_{(1)}=0.001\le 0.01$: below.
• $p_{(2)}=0.008\le 0.02$: below.
• $p_{(3)}=0.039>0.03$: above.
• $p_{(4)}=0.041>0.04$: above.
• $p_{(5)}=0.22>0.05$: above.

Step 3. The largest $k$ with $p_{(k)}$ below its threshold is $k=2$, so 2 hypotheses are rejected.

The trap. Four p-values are below $0.05$, but testing each at 5% ignores that five tests were run. Bonferroni, which tests each at $\frac{0.05}{5}=0.01$, also rejects 2 here. Benjamini-Hochberg is less strict than Bonferroni in general: it controls the expected share of false rejections among the rejections, not the chance of any false rejection.