Showing posts with label p-values. Show all posts
Showing posts with label p-values. Show all posts
Tuesday, October 6, 2015
Notes for October 6 and 8.
The posts about the confidence intervals for proportions.
The explanation of Fisher's idea, the hypothesis test.
The differences between p, p-hat and p-value.
Let's consider the lady tasting tea and how we would test her using a z-score. This is a reasonably simply calculation, though it isn't as precise when n is small as using the numbers we get from looking at the problem as sampling with replacement.
In this case, the null hypothesis is given by the equation
H0: p = 0.5
What this states is that she is "just guessing" between milk-in-tea and tea-in-milk, so she should have a 50-50 chance to be right (or wrong) each time.
Let's say we set our confidence level at 99% and tested her six times, and she went six for six. That means p-hat = 6/6 = 1. We now have all the numbers we need to get our z-score. using sqrt to stand in for square root, in our calculator we would type
(1 - .5)/sqrt(.5*.5/6) = 2.4494897...,
which is 2.45 when rounded to the nearest hundredth. We use this to look up the p-value on the goldenrod sheet and get .9929. This is the p-value, the test statistic we use to make out decision whether or not to reject H0 or fail to reject. Because this is a high tailed test and .9929 >= .99, we reject the null hypothesis, which means we think she is not just guessing given the results of this test.
Translated into English, we are 99% convinced she actually has some talent at telling the difference between the two ways of making tea, but we still hold out the 1% possibility that she was just a very lucky guesser.
As for the three numbers that all use the letter p, once again:
p is the probability we get by the definition of the test. In this case it was 0.5, but we will do other tests where the number can be something else.
p-hat is the proportion of answers she got right if we do the high tailed test, or the proportion of answers she got wrong if we use the more accurate low tailed test.
The p-value is the proportion we get from the test statistic, and this is what we use to decided whether we will reject or fail to reject the null hypothesis H0.
Wednesday, March 26, 2014
Notes for March 18 and 20
The topic for the next few weeks is hypothesis testing. The main idea
is that experiments must be conducted to test the validity of an idea,
which is called a hypothesis. There are always two hypotheses
available, the null hypothesis H0 (pronounced "H zero" or "H nought") and the alternate hypothesis HA (pronounced "H A").
The standard is to assume the null hypothesis is true, which says that
nothing special is happening, which in most cases means that two things
we can measure should be equal or close to it. The alternate
hypothesis says the two measurements are different. We can have one
tailed high tests, where we want "large" positive test statistics. In
one tailed low tests, only negative test statistics with "large"
absolute value will do. In two tailed test, "large" absolute value for
positive or negative numbers will work. We will only accept the
alternate hypothesis if the experiment produces impressive results given
our particular criteria for that test.

The basics of hypothesis testing are similar to the ideals of the English legal system, which is also the system used in United States courts, that a defendant is presumed innocent until proven guilty. There are different levels of proof of guilt in different trials, whether it is beyond a reasonable doubt or the less rigorous standard of preponderance of evidence.
In a case involving an alleged crime, there is the reality of what the defendant did and the result of the trial. If the defendant did the illegal act, then being found guilty is the correct result under the law. If the reality is that the defendant didn't do the act, the correct result would be a not guilty verdict.
The reasonable doubt standard is put in place in theory to make sending an innocent person to jail unlikely, and this is called a Type I error. The best known Type I result in legal history is Jesus Christ.
It is also possible that a person who did a crime will be found not guilty. This is called a Type II error. When I ask my students for an example of Type II error, O.J. Simpson's name still rings out the loudest.

In hypothesis testing, there is the reality and the result of the experiment. If H0 is true, the two things measured are equal or pretty close to equal. If HA is true, they are significantly different.
If the experiment produces a test statistic that is beyond the threshold we set for it, and "beyond" could mean lower if it is a one-tailed low test, or higher if it is a one-tailed high test, or either lower or higher in a two-tailed test, then we reject H0. If the test statistic fails to get "beyond" the threshold, we fail to reject H0.
Rejecting a true null hypothesis is a Type I error. Rejecting a false alternate hypothesis is a Type II error.
In class, we discussed Sir Ronald Fisher and his hypothesis testing of
the lady who said she could tell the difference in taste between tea
poured into milk or milk poured into tea.
Here are the things that have to be done to make such an experiment work.
#1 Define the null hypothesis. In modern experiments, the null hypothesis is always defined as an equation. In a proportion test, the equation will be concerning p, the true probability of success. In the lady tasting tea, we would assume if nothing special is happening, then she is just guessing whether the tea or milk was poured first, and the probability of being correct on any given trial is 50% or .5. We write this as follows.
H0: p = .5
#2 Pick a threshold. The trials we are going to perform are taste tests where the lady cannot see the tea-milk mixture being poured. We have to decide on how high a test statistic we will consider impressive. The three standard choices are 90% confidence, 95% confidence or 99% confidence. For experiments in the medical field, where the decision is whether or not to bring a new drug to market, the 99% confidence level is common. For an experiment like this, where the result is not truly earth shattering, we might decide to us the 95% confidence threshold.
The experiment will produce a z-score the thresholds for high z-scores are as follows:
90% threshold: z = 1.28
95% threshold: z = 1.645
99% threshold: z = 2.325
#3 Decide on the number of trials in an experiment. There is a tug-of-war in deciding the number of trials. More trials produces numbers we can be more confident in, but more trials is also more expensive and more time consuming. In the case of the lady tasting tea, we don't want to keep her drinking tea mixtures for hours.
Different books set different standards for the minimum number of trials based on np and nq. Some say both np > 5 and nq > 5. Others say both numbers should be greater than 10, yet others say 15. The standard that np >= 10 and nq >= 10 can be connected to the standard that says n > np + 3*sqrt(pqn) > np - 3*sqrt(pqn) > 0 by a little algebraic manipulation.
np - 3*sqrt(pqn) > 0 [add the square root to both sides]
np > 3*sqrt(pqn) [square both sides]
n^2*p^2 > 9pqn [divide both sides by np]
np > 9q
Since q must be less than 1, but can be as close to 1 as we want, set it equal to 1 and the inequality becomes
np > 9, which we can change to np >= 10.
For example, If let's look at the different possible positive z-score results if the lady were given ten trials, which would be enough if we used the lowest standard of np >= 5 and nq >= 5.
10 correct out of 10: z = 3.16227... ~= 3.16, which is above the 99% threshold.
(look-up table: .9992)
9 correct out of 10: z = 2.52982... ~= 2.53, which is above the 99% threshold.
(look-up table: .9943)
8 correct out of 10: z = 1.89737... ~= 1.90, which is above the 95% threshold, but not the 99%.
(look-up table: .9713)
7 correct out of 10: z = 1.26491... ~= 1.26, which is below the 90% threshold.
(look-up table: .8962)
If we set the bar at the 90% threshold, she could impress us by getting 8 right out of 10 or better. Likewise, at the 95% threshold, 8 of 10 will be beyond the threshold and the result would make us reject H0. At the 99% threshold, she would have to get 9 of 10 or 10 of 10 to make a z-score that breaks the threshold.
#4 Interpreting the test statistic. Let's say for the sake of argument that the lady got 8 of 10 correct. (There is a book about 20th Century statistics entitled The Lady Tasting Tea, where a witness to the experiment says he can't recall how many times she was tested, but the lady got a perfect score.) If we set the threshold at 90% confidence or 95% confidence, we would be impressed by the z score of 1.90 and we would reject H0, which says that we don't think she is "just guessing", but actually has the talent she says she has. If we set the value at 99% confidence, we would fail to reject H0.
Here's the thing. We could be wrong. If we reject H0 incorrectly, it means she was just guessing and she was very lucky during this test, a Type I error. A z-score of 1.90 corresponds to a probability of 97.13%, which is called the p-value in hypothesis testing. She can pass the test by being better than 97.13% of lucky guessers. If she is a lucky guesser, she would fool anyone who had set the threshold at 90% or 95%.
If we fail to reject H0, which we would do if we set the threshold at 99%, this could also be an error, but this time it would be a Type II error. Under this scenario, she got 8 of 10 but she should do better usually. It's more difficult math to figure out how good she actually is and how unlucky she had to be to get only 8 of 10. The probability of a Type I error is called alpha, and it is determined by the threshold. The probability of a Type II error is called beta, and it usually explained in greater detail in the class after the introduction to statistics.
A low tailed test
Let's say we want a low error rate. Unlike the lady tasting tea who needed a lot of right answers to impress us, now we need a low score to get a result that will make us reject the null hypothesis.
Now our z-score thresholds are
90% threshold: z = -1.28
95% threshold: z = -1.645
99% threshold: z = -2.325
Let's say we want to be convinced our error rate is less than 10% and we want to be convinced to the 95% confidence level.
H0: p = 0.10
HA: p < 0.10
n = 50
10% of 50 is 5, so we should check to see what happens at f = 4, 3, 2, 1 and 0 errors. Typing this into the calculator will look like
(f/50 - .1)/sqrt(.1*.9/50)
f = 4 gives a rounded z-score of -.47
(look-up table: .3192 fail to reject H0)
f = 3 gives a rounded z-score of -.94
(look-up table: .1736 fail to reject H0)
f = 2 gives a rounded z-score of -1.41
(look-up table: .0793 fail to reject H0 at 95% confidence, but reject at 90%)
f = 1 gives a rounded z-score of -1.89
(look-up table: .0294 reject H0 at 95% confidence, but fail reject at 99%)
f = 0 gives a rounded z-score of -2.36
(look-up table: .0091 reject H0 at any confidence level we use 90%, 95% or 99%)

The basics of hypothesis testing are similar to the ideals of the English legal system, which is also the system used in United States courts, that a defendant is presumed innocent until proven guilty. There are different levels of proof of guilt in different trials, whether it is beyond a reasonable doubt or the less rigorous standard of preponderance of evidence.
In a case involving an alleged crime, there is the reality of what the defendant did and the result of the trial. If the defendant did the illegal act, then being found guilty is the correct result under the law. If the reality is that the defendant didn't do the act, the correct result would be a not guilty verdict.
The reasonable doubt standard is put in place in theory to make sending an innocent person to jail unlikely, and this is called a Type I error. The best known Type I result in legal history is Jesus Christ.
It is also possible that a person who did a crime will be found not guilty. This is called a Type II error. When I ask my students for an example of Type II error, O.J. Simpson's name still rings out the loudest.

In hypothesis testing, there is the reality and the result of the experiment. If H0 is true, the two things measured are equal or pretty close to equal. If HA is true, they are significantly different.
If the experiment produces a test statistic that is beyond the threshold we set for it, and "beyond" could mean lower if it is a one-tailed low test, or higher if it is a one-tailed high test, or either lower or higher in a two-tailed test, then we reject H0. If the test statistic fails to get "beyond" the threshold, we fail to reject H0.
Rejecting a true null hypothesis is a Type I error. Rejecting a false alternate hypothesis is a Type II error.
In class, we discussed Sir Ronald Fisher and his hypothesis testing of
the lady who said she could tell the difference in taste between tea
poured into milk or milk poured into tea.Here are the things that have to be done to make such an experiment work.
#1 Define the null hypothesis. In modern experiments, the null hypothesis is always defined as an equation. In a proportion test, the equation will be concerning p, the true probability of success. In the lady tasting tea, we would assume if nothing special is happening, then she is just guessing whether the tea or milk was poured first, and the probability of being correct on any given trial is 50% or .5. We write this as follows.
H0: p = .5
#2 Pick a threshold. The trials we are going to perform are taste tests where the lady cannot see the tea-milk mixture being poured. We have to decide on how high a test statistic we will consider impressive. The three standard choices are 90% confidence, 95% confidence or 99% confidence. For experiments in the medical field, where the decision is whether or not to bring a new drug to market, the 99% confidence level is common. For an experiment like this, where the result is not truly earth shattering, we might decide to us the 95% confidence threshold.
The experiment will produce a z-score the thresholds for high z-scores are as follows:
90% threshold: z = 1.28
95% threshold: z = 1.645
99% threshold: z = 2.325
#3 Decide on the number of trials in an experiment. There is a tug-of-war in deciding the number of trials. More trials produces numbers we can be more confident in, but more trials is also more expensive and more time consuming. In the case of the lady tasting tea, we don't want to keep her drinking tea mixtures for hours.
Different books set different standards for the minimum number of trials based on np and nq. Some say both np > 5 and nq > 5. Others say both numbers should be greater than 10, yet others say 15. The standard that np >= 10 and nq >= 10 can be connected to the standard that says n > np + 3*sqrt(pqn) > np - 3*sqrt(pqn) > 0 by a little algebraic manipulation.
np - 3*sqrt(pqn) > 0 [add the square root to both sides]
np > 3*sqrt(pqn) [square both sides]
n^2*p^2 > 9pqn [divide both sides by np]
np > 9q
Since q must be less than 1, but can be as close to 1 as we want, set it equal to 1 and the inequality becomes
np > 9, which we can change to np >= 10.
For example, If let's look at the different possible positive z-score results if the lady were given ten trials, which would be enough if we used the lowest standard of np >= 5 and nq >= 5.
10 correct out of 10: z = 3.16227... ~= 3.16, which is above the 99% threshold.
(look-up table: .9992)
9 correct out of 10: z = 2.52982... ~= 2.53, which is above the 99% threshold.
(look-up table: .9943)
8 correct out of 10: z = 1.89737... ~= 1.90, which is above the 95% threshold, but not the 99%.
(look-up table: .9713)
7 correct out of 10: z = 1.26491... ~= 1.26, which is below the 90% threshold.
(look-up table: .8962)
If we set the bar at the 90% threshold, she could impress us by getting 8 right out of 10 or better. Likewise, at the 95% threshold, 8 of 10 will be beyond the threshold and the result would make us reject H0. At the 99% threshold, she would have to get 9 of 10 or 10 of 10 to make a z-score that breaks the threshold.
#4 Interpreting the test statistic. Let's say for the sake of argument that the lady got 8 of 10 correct. (There is a book about 20th Century statistics entitled The Lady Tasting Tea, where a witness to the experiment says he can't recall how many times she was tested, but the lady got a perfect score.) If we set the threshold at 90% confidence or 95% confidence, we would be impressed by the z score of 1.90 and we would reject H0, which says that we don't think she is "just guessing", but actually has the talent she says she has. If we set the value at 99% confidence, we would fail to reject H0.
Here's the thing. We could be wrong. If we reject H0 incorrectly, it means she was just guessing and she was very lucky during this test, a Type I error. A z-score of 1.90 corresponds to a probability of 97.13%, which is called the p-value in hypothesis testing. She can pass the test by being better than 97.13% of lucky guessers. If she is a lucky guesser, she would fool anyone who had set the threshold at 90% or 95%.
If we fail to reject H0, which we would do if we set the threshold at 99%, this could also be an error, but this time it would be a Type II error. Under this scenario, she got 8 of 10 but she should do better usually. It's more difficult math to figure out how good she actually is and how unlucky she had to be to get only 8 of 10. The probability of a Type I error is called alpha, and it is determined by the threshold. The probability of a Type II error is called beta, and it usually explained in greater detail in the class after the introduction to statistics.
A low tailed test
Let's say we want a low error rate. Unlike the lady tasting tea who needed a lot of right answers to impress us, now we need a low score to get a result that will make us reject the null hypothesis.
Now our z-score thresholds are
90% threshold: z = -1.28
95% threshold: z = -1.645
99% threshold: z = -2.325
Let's say we want to be convinced our error rate is less than 10% and we want to be convinced to the 95% confidence level.
H0: p = 0.10
HA: p < 0.10
n = 50
10% of 50 is 5, so we should check to see what happens at f = 4, 3, 2, 1 and 0 errors. Typing this into the calculator will look like
(f/50 - .1)/sqrt(.1*.9/50)
f = 4 gives a rounded z-score of -.47
(look-up table: .3192 fail to reject H0)
f = 3 gives a rounded z-score of -.94
(look-up table: .1736 fail to reject H0)
f = 2 gives a rounded z-score of -1.41
(look-up table: .0793 fail to reject H0 at 95% confidence, but reject at 90%)
f = 1 gives a rounded z-score of -1.89
(look-up table: .0294 reject H0 at 95% confidence, but fail reject at 99%)
f = 0 gives a rounded z-score of -2.36
(look-up table: .0091 reject H0 at any confidence level we use 90%, 95% or 99%)
Labels:
hypothesis testing,
p-values,
proportions,
test statistic,
z-scores
Tuesday, May 12, 2009
Class notes for 5/11: Hypothesis testing for the mean of a population
t-scores and p values
If we have a z-score between -3.5 and +3.5, Table A-2 lets us find the p value associated with that z-score accurate to four decimal places. For example, if z = 1.71, the p value is .9564, which is to say that z-score is higher than 95.64% of data in a normally distributed set.
The t-score table is smaller, and to read a t-score correctly, we also need n, the size of the sample, because that gives us the Degrees of Freedom, which is n-1 in this case.
If n=10, then d.f. = 9, and the t-score table reads as follows.
___________________________Area in One Tail_____________
_______0.005______0.01______0.025______0.05______0.10__________df=9___3.250_____2.821______2.262_____1.833_____1.383__________
If t = 1.71, that value isn't on our table, but because it lies between the values associated with 0.05 and 0.10, that means that score is in the top 10% of scores, but not in the top 5%.
If instead n=30 and d.f. = 29, here are the t-score values.
___________________________Area in One Tail_____________
_______0.005______0.01______0.025______0.05______0.10__________df=29__2.756_____2.462______2.045_____1.699_____1.311__________
Now a t-score of 1.71 lies between 2.045 and 1.699, which means it is in the top 5%, but not the top 2.5%.
Like the z-score table, the t-score table is symmetric about the value t=0. If d.f.=29, t=-1.71 is a score in the bottom 5%, but not the bottom 2.5%.
Hypothesis testing for the mean of a population

Hypothesis testing for the mean of a population assumes we know the population mean from some previously obtained information. Perhaps that mean has changed over time or the previous information wasn't correct to begin with, but the null hypothesis assumes we know that mean, which we call mux. If we take a sample from the population, we will get the values x-bar, sx and n, and using those values and mux, we can get the t-score.
Just like with the hypothesis test for a proportion, the test can be one-tailed high, one-tailed low or two-tailed.
For example, if we were testing people who had studied using a special method and we were checking scores on a standardized test, we would only be impressed if the average went up, so a one-tailed high test would be appropriate.
If the experiment was dealing with a cholesterol drug, we would want to see a lower average reading, and a one-tailed low test would be used.
If we assume the average duration of a pop song on the charts today is the same as the duration of pop songs in the seventies, we can't assume beforehand if the new readings will be higher or lower, and would be surprised if the new average were significantly different in either direction, so a two-tailed test would be appropriate.
Here is some data we went over in class. In most textbooks, the 'normal' human body temperature is listed at 98.6 degrees Fahrenheit, based on the work of Dr. Carl Wunderlich back in the 19th Century. If we do a test, it should be a two-tailed test, since we would be surprised if the normal temperature is significantly higher or significantly lower than this. Since this is a medical experiment, let's use the 99% level of confidence.
The size of the sample was n=103, which means the degrees of freedom are 102. Our table doesn't have a listing for d.f.=102, and the next lowest available value is d.f.=100. Here are the table values for that row of Table A-3.
___________________________Area in Two Tails____________
_________0.01______0.02_______0.05______0.10______0.20__________df=100__2.626_____2.364______1.984_____1.660_____1.290__________
With a two tailed test at the 99% confidence level, this means we want the 0.01 column. The "middle" 99% of data lies between t-scores of -2.616 and 2.616. If the t-score lies in that range, we will fail to reject H0. If it is greater or equal to 2.616 or less than or equal to -2.616, we will reject H0.
The values from this study found that x-bar = 98.2 degrees and the standard deviation sx was 0.62. Plugging into our t-score equation from above, we get
t = (98.2-98.6)/0.62*sqrt(103) = -6.547671977... ~= -6.548.
We don't get an exact p value for a number so far away from zero, but if we look at outlier z-score table, we can roughly approximate that this p value is somewhere around 1 in 1,000,000,000. We can say with 99% confidence that the average body temperature is not 98.6 degrees, but probably close to the sample average of 98.2 degrees. Our p value shows we could qualify for even greater confidence with our statement, but very rarely do tests ask for more than 99% confidence, and changing the criteria after the fact is not recommended. Still, publishing this incredibly tiny p value will convince people who can read a statistical report that the evidence is very strong indeed.
This test also changed the idea of what should constitute a fever. Instead of one temperature of 100.4 degrees Fahrenheit being the absolute gauge, the temperature will fluctuate depending on the time of day, as do the normal temperature readings.
If we have a z-score between -3.5 and +3.5, Table A-2 lets us find the p value associated with that z-score accurate to four decimal places. For example, if z = 1.71, the p value is .9564, which is to say that z-score is higher than 95.64% of data in a normally distributed set.
The t-score table is smaller, and to read a t-score correctly, we also need n, the size of the sample, because that gives us the Degrees of Freedom, which is n-1 in this case.
If n=10, then d.f. = 9, and the t-score table reads as follows.
___________________________Area in One Tail_____________
_______0.005______0.01______0.025______0.05______0.10__________df=9___3.250_____2.821______2.262_____1.833_____1.383__________
If t = 1.71, that value isn't on our table, but because it lies between the values associated with 0.05 and 0.10, that means that score is in the top 10% of scores, but not in the top 5%.
If instead n=30 and d.f. = 29, here are the t-score values.
___________________________Area in One Tail_____________
_______0.005______0.01______0.025______0.05______0.10__________df=29__2.756_____2.462______2.045_____1.699_____1.311__________
Now a t-score of 1.71 lies between 2.045 and 1.699, which means it is in the top 5%, but not the top 2.5%.
Like the z-score table, the t-score table is symmetric about the value t=0. If d.f.=29, t=-1.71 is a score in the bottom 5%, but not the bottom 2.5%.
Hypothesis testing for the mean of a population

Hypothesis testing for the mean of a population assumes we know the population mean from some previously obtained information. Perhaps that mean has changed over time or the previous information wasn't correct to begin with, but the null hypothesis assumes we know that mean, which we call mux. If we take a sample from the population, we will get the values x-bar, sx and n, and using those values and mux, we can get the t-score.
Just like with the hypothesis test for a proportion, the test can be one-tailed high, one-tailed low or two-tailed.
For example, if we were testing people who had studied using a special method and we were checking scores on a standardized test, we would only be impressed if the average went up, so a one-tailed high test would be appropriate.
If the experiment was dealing with a cholesterol drug, we would want to see a lower average reading, and a one-tailed low test would be used.
If we assume the average duration of a pop song on the charts today is the same as the duration of pop songs in the seventies, we can't assume beforehand if the new readings will be higher or lower, and would be surprised if the new average were significantly different in either direction, so a two-tailed test would be appropriate.
Here is some data we went over in class. In most textbooks, the 'normal' human body temperature is listed at 98.6 degrees Fahrenheit, based on the work of Dr. Carl Wunderlich back in the 19th Century. If we do a test, it should be a two-tailed test, since we would be surprised if the normal temperature is significantly higher or significantly lower than this. Since this is a medical experiment, let's use the 99% level of confidence.
The size of the sample was n=103, which means the degrees of freedom are 102. Our table doesn't have a listing for d.f.=102, and the next lowest available value is d.f.=100. Here are the table values for that row of Table A-3.
___________________________Area in Two Tails____________
_________0.01______0.02_______0.05______0.10______0.20__________df=100__2.626_____2.364______1.984_____1.660_____1.290__________
With a two tailed test at the 99% confidence level, this means we want the 0.01 column. The "middle" 99% of data lies between t-scores of -2.616 and 2.616. If the t-score lies in that range, we will fail to reject H0. If it is greater or equal to 2.616 or less than or equal to -2.616, we will reject H0.
The values from this study found that x-bar = 98.2 degrees and the standard deviation sx was 0.62. Plugging into our t-score equation from above, we get
t = (98.2-98.6)/0.62*sqrt(103) = -6.547671977... ~= -6.548.
We don't get an exact p value for a number so far away from zero, but if we look at outlier z-score table, we can roughly approximate that this p value is somewhere around 1 in 1,000,000,000. We can say with 99% confidence that the average body temperature is not 98.6 degrees, but probably close to the sample average of 98.2 degrees. Our p value shows we could qualify for even greater confidence with our statement, but very rarely do tests ask for more than 99% confidence, and changing the criteria after the fact is not recommended. Still, publishing this incredibly tiny p value will convince people who can read a statistical report that the evidence is very strong indeed.
This test also changed the idea of what should constitute a fever. Instead of one temperature of 100.4 degrees Fahrenheit being the absolute gauge, the temperature will fluctuate depending on the time of day, as do the normal temperature readings.
Labels:
hypothesis testing,
p-values,
Student's t-scores
Monday, May 11, 2009
Class notes for 5/11: p-values and Confidence Levels
The test statistics we will use for hypothesis testing will differ, sometimes using z-scores, other times t-scores and yet other times the chi-square table. What all these tests have in common is that they correspond to probabilities, called p-values.
The idea is that if we have a test with 90% confidence, we will only reject H0 if an event happens 10% of the time or less.
Likewise, 95% confidence means we reject H0 when the probability of the event is less than or equal to 5%.
99% confidence means we reject H0 with tests that happen 1% of the time or less.

This table shows what p-value corresponds to the threshold where we reject the null hypothesis, which technically is the same as accepting the alternate hypothesis. When it's one tailed high, the p-value threshold is easy, .9 for 90% confidence, .95 for 95% confidence and .99 for 99% confidence. For one tailed low, the pattern is that the p-value equals 100% minus the confidence level. For the two tailed test, the two tails have to add up to 100% minus the confidence level, so the 90% confidence level threshold is at the p-values of 5% (.05) and 95% (.95).
Remember, just because a test convinced us to reject the null hypothesis doesn't mean the null hypothesis is false. It could still be true, and we would be making a Type I error.
If we use the 90% confidence level, we should make Type I errors about 10% of the time.
At the 95% confidence level, the probability of Type I errors is 5%.
At 99% confidence level, the probability of Type I errors is 1%.
The lower likelihood of errors is why most tests done on medical data is done at the 99% confidence level.
The p-value is often published to show just how well the test did. Maybe you only asked to prove something to 90% confidence on a one-tailed high test, but the p-value is .9978. This shows people that read the findings that it would be strong enough data to reject H0 at even higher confidence levels.
The idea is that if we have a test with 90% confidence, we will only reject H0 if an event happens 10% of the time or less.
Likewise, 95% confidence means we reject H0 when the probability of the event is less than or equal to 5%.
99% confidence means we reject H0 with tests that happen 1% of the time or less.

This table shows what p-value corresponds to the threshold where we reject the null hypothesis, which technically is the same as accepting the alternate hypothesis. When it's one tailed high, the p-value threshold is easy, .9 for 90% confidence, .95 for 95% confidence and .99 for 99% confidence. For one tailed low, the pattern is that the p-value equals 100% minus the confidence level. For the two tailed test, the two tails have to add up to 100% minus the confidence level, so the 90% confidence level threshold is at the p-values of 5% (.05) and 95% (.95).
Remember, just because a test convinced us to reject the null hypothesis doesn't mean the null hypothesis is false. It could still be true, and we would be making a Type I error.
If we use the 90% confidence level, we should make Type I errors about 10% of the time.
At the 95% confidence level, the probability of Type I errors is 5%.
At 99% confidence level, the probability of Type I errors is 1%.
The lower likelihood of errors is why most tests done on medical data is done at the 99% confidence level.
The p-value is often published to show just how well the test did. Maybe you only asked to prove something to 90% confidence on a one-tailed high test, but the p-value is .9978. This shows people that read the findings that it would be strong enough data to reject H0 at even higher confidence levels.
Subscribe to:
Posts (Atom)

