Twenty tests, one false alarm
You test trading signals, each with a test at the level. In reality none of the signals works (every null hypothesis is true) and the tests are independent.
What is the probability that at least one signal comes out significant?
Fractions, decimals with at least 3 significant digits, and expressions such as sqrt(2)/2, e^-1 or pi/4.
Compute the probability that no test rejects.
h
s Shown after a correct answer. Revealing it first marks the problem as seen, not solved.
Idea. Each test has a chance of a false alarm. The chance that all stay quiet shrinks geometrically.
Step 1. Under a true null, a test rejects with probability . By independence, no test rejects with probability .
Step 2.
The lesson. Try enough ideas and something will look significant: with useless signals you find a “discovery” almost two times out of three, and you expect exactly false positive. The Bonferroni correction tests each signal at , which keeps the probability of any false alarm below : here .