Type I (false positive) and Type II (false negative) errors

I've seen a lot of explanations of Type I and Type II errors, but this photo collage is my favorite.

When we set a confidence level, what we are doing is setting the probability of Type I error. The null hypothesis can always be interpreted as "nothing special is happening" and Type I errors mean we think something special is happening when it isn't. In many cases, false positive are more disruptive than false negatives and we are more interested in limiting Type I errors than we are in limiting Type II errors. There will be more discussion of this in class on Oct. 15 and the note will also be posted here.

Test results and errors

We do the hypothesis testing because we cannot truly know what reality is, only the test result. If we reject the null hypothesis Ho, we did so because of strong evidence. If there is an error, it is a Type I error. If we set the error threshold at 90% confidence, we expect such errors about 10% of the time. If it is set at 95% confidence, then Type I errors should happen about 5% of the time and at 99% confidence, Type I errors should only happen about 1% of the time.

If we fail to reject H0, the only type of error we can make is called Type II error. The probability of such errors is trickier to compute and we will not work on this problem during this class.