Notes for March 11

Confidence of victory

The margin of error is the industry standard, but the people who use these numbers, most especially the news media, really don't understand them very well. Here is a different method to produce a more useful piece of mathematical information which this author has developed, called the confidence of victory method. Here is a recent poll from Public Policy Polling (PPP) on the North Carolina senate race.

n = 884
Hagan 45%
Tillis 43%


These numbers add up to 88%, which means about 12% are either undecided or prefer another candidate. What we do is effectively ignore the respondents who are either voting for third party candidates, are preferring none of the candidates or are still telling pollsters they are undecided. We now figure out how many people in the poll said they prefer Hagan and how many prefer Tillis by multiplying the percents by the size of the poll.

f(Hagan) = 884 * .45 = 397.8
f(Tillis) = 884 * .43 = 380.12

new n = 397.8+380.12 = 777.92

p-hat(Hagan) = 397.8/777.92 ~ 51.1%
p-hat(Tillis) = 380.12/777.92 ~ 48.9%

sp-hat = sqrt(.511*.489/777.92) ~ 1.79%

z(Hagan) = (51.1 - 50)%/1.79% ~ .61

This says Hagan's percentage is about 0.61 standard deviations above 50%. The percentage he will get in the actual election may be higher or lower than what we see here. We assume there's about a half a chance he will do better than the final opinion poll, and a half a chance he will do worse. What the public actually cares about is whether he wins or loses. What the confidence of victory method does is find the percentage that corresponds to the z-score. That number is the confidence level we have that the true percentages from the population polled will show that the leader in the poll will be the winner of the election. In this example, z=0.61 corresponds to .7291 on our positive z-score table. Because the confidence of victory method is sensitive to small changes, we should round to the nearest percent and use this sentence to describe the results.

If the election were held when the poll was taken, we are 73% confident that Hagan will hold on to the lead shown in the poll and win the election in North Carolina.

In 2008, the final polls in the 50 states and Washington D.C. had two states too close to call, Missouri and Indiana. Both elections were very close, called late in the evening, Missouri for McCain and Indiana for Obama. In the other 49 contests where confidence of victory claimed an advantage for one side or the other, 48 contests were won by the person leading in the most recent poll, which is to say the confidence of victory method was vindicated about 98% of the time. The only state where the confidence of victory method did not get the right result was North Carolina. McCain had a 60% confidence of victory in North Carolina, but Obama actually won the state.

In 2012, confidence of victory did much better. There was only one toss-up state, Florida, a state that did not give their results for four days. All the rest of the states were called correctly in the electoral college vote, as were all the U.S. Senate races.




Probabilities and payoffs

The probability of winning a game may be known or unknown, but that is different from the payoffs of the game, sometimes called the odds. There are two methods, which we will call the classic parimutuel  and the modern parimutuel.

Classic parimutuel: The statement has two numbers, sometimes written with a colon between them, such as 7:1, or sometimes a dash such as 7-1 or sometimes the word "to", like 7 to 1. In all these cases, the first number represents a ratio of the profit a player will get if the bet goes well and the second number is the amount at risk should the gamble fail. In any gamble, there is another person (or maybe a casino or a bookmaker) on the other side of the bet, and the profit and risk are reversed.

Example: 7:1 odds (first person's perspective)
If you get 7-1 odds and risk $10, the opponent must put up $70. The $80 is put someplace for safe keeping until the result is posted. The winner will get all $80. For the person getting 7:1 odds, there would be a $70 profit in case of a win, but the other side, who looks at the bet as a 1:7 proposition, there would only be a $10 profit in the case of a win.

Classic parimutuel is always given in numbers written in lowest terms, so 70:10 would be written as 7:1, or 25:10 would reduced to 5:2. Some like to have the low number be 1, so 5:2 will sometimes be written as 2.5:1.

Modern parimutuel: In modern parimutuel, only one number is given either a positive like +150 or a negative like -170. We need a second number, and in this system the second number is always 100.

Positive example: If the number is positive like +150, this number is the profit and the risk = 100.

Negative example: If the number is negative like -170, the number 170 (the absolute value) is the risk and the profit = 100.

Changing modern to classic: For +150, we would write the classic as 150:100 (profit:risk) and reduce the fraction to lowest terms, so here it would be 3:2.

For -170, the ratio would be 100:170, which would reduce to 10:17 in lowest terms.

Changing classic to modern: We will have two numbers, profit:risk and we are interested in which is smaller. What we will do is multiply both numbers by 100/small, and the number that is largest is the modern parimutuel, with a + sign if profit > risk and a negative sign if profit < risk.

Profit > risk: If the classic odds are 5:4, multiply both numbers by 100/4 = 25 to get 125:100. Because the big number is first, the modern version would be written +125.


Profit < risk: If the classic odds are 5:8, multiply both numbers by 100/5 = 20 to get 100:160. Because the big number is second, the modern version would be written -160.

The expected value of a game

The expected value of a two outcome game (win or lose) can be written as EV= p*(profit+risk)/risk. By dividing by risk, the game's expected value can be thought of as a percentage of money returned to you on average every time you play. Again, this is an average, so the average outcome doesn't have to be achieved. In many games, it never is. Expected value is really about the long run.

Flipping a fair coin, if you call "heads" every time, you should win about 50% of the time, and so EV = .5(1+1)/1 = 100%, so calling heads is a way to make this a fair game with a fair coin.

If you mix up your calls, it's also a fair game.

If you go with "rock" every time in rock/papers/scissors, your opponent will catch on soon enough and go with "paper" every time, and you will lose money in the long run.

Rock/scissors/paper is a fair game only if you can mix up your calls using some random method, or at least a method hard for your opponent to determine. The best method is 1/3 rock, 1/3 scissors and 1/3 paper, and the expected value is 100%, or that you will neither lose nor win, but break even.

Let's go back to roulette. We saw that playing a single number or playing either black or red produce the same expected value for the player.

Single number for player
p = 1/38
profit = 35
risk = 1
EV = 1/38*(35+1)/1 = 36/38 ~= 94.7%

Red (or Black) for player
p = 18/38
profit = 1
risk = 1
EV = 18/38*(1+1)/1 = 36/38 ~= 94.7%

For the casino, the probability of winning is 1 minus the probability for the player. The risk and profit numbers are switched from the players' values.

Single number for casino
p = 37/38
profit = 1
risk = 35
EV = 37/38*(35+1)/35 = (36*37)/(35*38) ~= 100.15%

Red (or Black) for player
p = 20/38
profit = 1
risk = 1
EV = 20/38*(1+1)/1 = 40/38 ~= 105.3%

When profit = risk, which is the same as saying the classic parimutuel odds are 1:1 or the modern parimutuel odds are +100 (which is the same as -100, though rarely written that way), the average of the two expected values will be 100%. You expect to lose about 5.3 cents every game and the casino expects to win about 5.3 cents.

When profit does not equal risk, we get different percentage advantages and disadvantages. Because the casino must risk 35 bets and the player only 1 bet when the player picks a single number at roulette, the expected value for the casino is still positive, but relatively small at .15 of a cent per game. In reality, the casino rarely spins the wheel with only one bettor playing, so the casino is not truly risking only its own money on a single spin, but can use the losses of some players to help offset any possible winner. Even if that weren't the case, a game with an expected value greater than 100% means a winner in the long run, and that is the business model the casino operate on, and very successfully as anyone can see.

Notes for February 25 and 27

The dependent probabilities in a 52 card deck

One of the simplest mathematical models of dependency is sampling without replacement, which is the way most card games or lotteries or the game of Bingo works. You have a set of outcomes which get effectively randomized and a trial is performed, meaning a card is taken from the deck or a ping pong ball is removed from the hopper or a bingo marker is removed from the spinner. Once removed, the number of possible outcomes has been reduced by one and probabilities for success and failure of certain outcomes change.

Looking for an ace: There are 52 cards is a standard deck and 4 of them are aces. If I draw a card from a randomized deck, the chances are 4/52 = 1/13 ~= 7.7% that the card will be an ace. What are the chances the second card is an ace?

That depends on the first card.

Probability that the second card is an ace, given the first card is an ace is 3/51 = 1/17 ~= 5.9%.

Probability that the second card is an ace, given the first card is not an ace is 4/51 ~= 7.8%.

Unlike the mathematical model of free throw shooting where we re-calculate the probabilities by adding the most recent make or miss into the percentage, which means a miss brings the odds down and a make brings the odds up, not getting an ace makes the odds a little better next time, and getting an ace makes the odds worse.


This is the formula for the dependent probability model of sampling without replacement is given at the left. The two numbers in parentheses are a binomial coefficient, the numbers you get when you use nCr on your calculator, which I pronounce "n choose r" in class. The pairs of numbers that look like a base and an exponent, except that the exponent is underlined, is the convention developed by Donald Knuth at Stanford for writing the numbers that you get on your calculator using the nPr function, which I pronounce "n fall r", referring to the name "the falling factorial". If we think about a deck of cards, the lowercase letters refer to the size of the hand n, where r is the number of successful trials (r for right) and w is the number of unsuccessful trials (w for wrong), and r+w=n. The uppercase letters refer to the size of the deck, where T is the size of the deck, G is the number of cards we consider success if we draw them and B is the number of cards we consider a failed trial if we draw them. The letter T stands for Total, G for Good and B for Bad. Again, we have an equation, G+B=T.

Example: If we want consider drawing a heart a success and anything else a failure, what is the probability of drawing three hearts and two non hearts in a five card hand from a well-shuffled 52 card deck.

Here are the six numbers we need.
n = 5
r = 3
w = 2
T = 52
G = 13
B = 39

On a TI-30XIIs, here are the keys you would press.

5[prb][right]3×13[prb]3×39[prb]2÷52[prb]5[enter]

The calculator will read as follows.

5 nCr 3*13 nPr 3*39 nPr 2/52 nPr 5
0.081542617

This means the probability of exactly three hearts and two cards of some other suit is about 8.15%.

Let's say instead the deck had 10,000 cards and 2,500 hearts. our numbers would change.
n = 5
r = 3
w = 2
T = 10000
G = 2500
B = 7500

On a TI-30XIIs, here are the keys you would press.

5[prb][right]3×2500[prb]3×7500[prb]2÷10000[prb]5[enter]

The calculator will read as follows.

5 nCr 3*2500 nPr 3*7500 nPr 2/10000 nPr 5
0.0878613102

The difference is small, but the second number is much closer to the odds of 3 out of 5 when the probability of success is .25 every time


5 nCr 3*.25^3*.75^2
0.087890625

The point of this is that as the size of the deck gets larger, dependent and independent probabilities get closer together.

When we use categorical data, the most important statistic we try to predict is the proportion of a value in the population, which we call p, which we will estimate using the proportion from the sample, known as p-hat.

Again we will create a confidence interval, and the formula for standard deviation is very different.

sp-hat = sqrt(p-hat * q-hat/n)

The confidence level multipliers for xx% are taken from the z-score table (Table A-2) instead of the t-score table, and this is because the standard deviation for the sample and the standard deviation for the population are expected to be relatively close to one another. The values for the CLMxx% are given on the first page of your yellow sheets in the lower left hand corner.

CLM90% = 1.645
CLM95% = 1.96
CLM99% = 2.575

Example: Consider data sets #1 and #2, and the proportion of males. Let's find the 95% confidence interval for the underlying population, which we will limit to students at Laney who take statistics.

Data set #1:
n = 38
f(males) = 18
p-hat(males) = 18/38 ~= .474
q-hat(males) = 1 - p-hat(males) ~= .526

.474 - 1.96*sqrt(.474*.526/38) &lt; p &lt; .474 + 1.96*sqrt(.474*.526/38)
.315 &lt; p &lt; .633

Given this sample of 38 students, we are 95% confident the percentage of male students taking statistics at Laney is between 31.5% and 63.3%.


Data set #2: n = 42
f(males) = 12
p-hat(males) = 12/42 ~= .286
q-hat(males) = 1 - p-hat(males) ~= .714

.286 - 1.96*sqrt(.286*.714/42) &lt; p &lt; .286 + 1.96*sqrt(.286*.714/42)
.149 &lt; p &lt; .423

Given this sample of 42 students, we are 95% confident the percentage of male students taking statistics at Laney is between 14.9% and 42.3%.

Data sets #1 and #2 combined: n = 80
f(males) = 30
p-hat(males) = 30/80 = .375
q-hat(males) = 1 - p-hat(males) = .625

.375 - 1.96*sqrt(.375*.625/80) &lt; p &lt; .375 + 1.96*sqrt(.375*.625/38)
.269 &lt; p &lt; .481


Given this sample of 40 students, we are 95% confident the percentage of male students taking statistics at Laney is between 26.9% and 48.1%.

Notice how much our intervals disagree with one another. This is because our best point estimates from the three sets are .474, .286 and .375. Also notice that the width of the 95% confidence interval tends to be smaller as n gets bigger. When the sample size is 38, the width of the confidence interval is .318. At n = 42, it is .274 wide. At n = 80, the width is .212. The most common way to make a confidence interval narrower is to increase the size of the sample.

There are two other ways to change the width. If you ask for a higher confidence level, the interval will get wider. If p and q are close to 50%, the confidence interval will be wider than if the are both far away from 50%.

Notes for February 18 and 20

A probability is a number between 0 and 1, inclusive, and is represent by p(event) if part of a sample or p-hat(event) if part of a population.

A probability space or event space is a list of what we call the simple events. The list follows two rules.

1. Every simple event is mutually exclusive from every other simple event, which means the cannot both happen simultaneously. For example, if I flip one coin, it cannot land "heads" and "tails" simultaneously.

2. When we add up the probabilities of all the simple events, the sum is 1. What this means is the simple events (sometimes called simple outcomes).

There are ways to create new probabilities from a set of simple events. The first methods we will discuss are AND, OR and NOT. Let's start with the most basic, NOT.

p(NOT x) is the probability that the event x will not happen. It is sometimes written as q(x) instead. It is always true that

p(x) + p(NOT x) = p(x) + q(x) = 1

Example #1: If we flip a coin, we have two possibilities, heads and tails. We usually assume that p(heads) = .5 and p(tails) = .5, but that doesn't have to be true. What is true in this cases with a categorical variable that has only two legitimate values is that p(NOT heads) = q(heads) = p(tails), and the sum is 1. This means if p(heads) = .51, then q(heads) = 1 - .51 = .49.

If we roll a six-sided die, we have six possible outcomes, a 1, a 2, a 3, a 4, a 5 or a 6. The probability for not rolling a 1 is  

p(NOT 1) = p(2) + p(3) + p(4) + p(5) + p(6)

Because these are simple events, we don't have to worry about overlap. And again, instead of calling it p(NOT 1), we can call it q(1).

If two events A and B are mutually exclusive, then

p(A OR B) = p(A) + p(B)

If they are not mutually exclusive, that means p(A AND B) does not equal zero, and the rule for finding OR changes to   

p(A OR B) = p(A) + p(B) - p(A AND B)

This is most easily explained with a contingency table. Let's say we have two variables for a game, one called Result (which is either a win or a loss) and Setting (which is either home or away). The easiest way to represent this situation is a contingency table, like the one below listing wins and losses for the Warriors so far this season, broken into home record and away record.

___H || A|| Total
W |16||15|| 31
L |10||12|| 22
  |26||27|| 53 = grand total

p(Home) = 26/53, which rounded to the nearest thousandth is .491
p(Away) = 27/53, which rounded to the nearest thousandth is .509. Notice that p(Home) = q(Away) and vice versa.
p(Wins) = 31/53, which rounded to the nearest thousandth is .585
p(Losses) = 22/53, which rounded to the nearest thousandth is .415. Notice that p(Wins) = q(Losses) and vice versa.

p(Wins AND Home) = 16/53, which rounded to the nearest thousandth is .302. What this represents is the number of home wins divided by the total number of games.

p(Wins OR Home) is the probability that a game picked at random is either a home game or a win. We will get this by adding all the wins to all the home games, but we have to subtract the home wins, because they were counted twice.

p(Wins OR Home) = 31/53 + 26/53 - 16/53 = 41/53 or .774 rounded to the nearest thousandth.


Besides AND, OR and NOT, we have the qualifier GIVEN. In a contingency table, this means we only look at the number is a single row or column.

p(Wins GIVEN Home) = 16/26 or .615 rounded to the nearest thousandth.

p(Home GIVEN Wins) = 16/31 or .516 rounded to the nearest thousandth.

Notice that p(Wins) = .585 but p(Wins GIVEN Home) = .615. When these numbers are different, we say that the two categories Wins and Home are dependent. If they were the same, we would say the categories are independent. A contingency table that was independent might look like this


___H || A|| Total
W |16|| 8|| 24
L |10|| 5|| 15
  |26||13|| 39 = grand total

Now, p(Wins) = p(Wins GIVEN Home) = p(Wins GIVEN Away). This hypothetical win-loss record is at the same proportion whether on the road or at home.

Binomial distribution of an independent variable


If I say someone is a 70% free throw shooter, is every free throw attempt independent of what happened before? Often, we set up such an experiment assuming independence just to make our work simpler, but the human factor is involved, so in reality it's very likely to be dependent. Some people get frustrated after a few misses and will do worse. Others will learn from the mistakes of a few misses and figure out what they are doing wrong and make improvements. A player might be having a bad day for some reason, or might instead have excellent concentration or just really good luck that day. But again, these kinds of experiments are often set up as though each free throw trial is independent of what came before.

Let's look at flipping coins. A list of all possible events is called the event space. Here are some examples of event spaces.

Event space for flipping one coin
Heads (H)
Tails (T)

ways to get one head = 1
ways to get no heads = 1

Event space for flipping two coins
HH
HT
TH
TT

ways to get two heads = 1
ways to get one head = 2
ways to get no heads = 1

Event space for flipping three coins
HHH
HHT
HTH
HTT
THH
THT
TTH
TTT

ways to get three heads = 1
ways to get two heads = 3
ways to get one head = 3
ways to get no heads = 1


The list of numbers of ways to get r successes in n trials is often written in the pattern of the picture shown here, and this pattern is called Pascal's Triangle, at least in most of the world. The Italians call it Tartaglia's Triangle and the Chinese call it Yanghui's Triangle. None of these people actually invented it or claimed to have invented it. It's been around since before the time of Christ, and it has been studied all around the world.

While it is very common to see it presented in the form here as an equilateral triangle, it can also be presented where the first numbers in each row are lined up straight as follows

1
1 1
1 2 1
1 3 3 1
1 4 6 4 1
... etc.

It is standard to start counting the top row as row 0, and the left most column as column 0. For example, the 6 we see in the middle of the last row I typed in is row 4, column 2. Instead of having a copy of Pascal's Triangle around, our calculators have these numbers available. On Texas Instruments calculators, the function is under the probability menu. On the TI-30XIIs, the way to get that 6 is to type

4 [prb][right arrow]2[enter]

The calculator will read

4 nCr 2
6

All scientific calculators should have this function available, but all of them are slightly different. The TI-89 writes it as nCr(4,2) and Casio calculators write it as 4 C 2. I will pronounce it "4 choose 2", and when I type on the blog, I will type C(4,2). When I write it on the board or on tests, I will put a 4 on top of a 2 and surround both numbers with a large parentheses. These numbers are called the binomial coefficients.


The formula for finding the probability for exactly r successes in n independent trials where the probability of success on any single trial is p is shown here. In some books, they don't use the letter q, instead replacing it with (1-p). Likewise, sometimes w is replaced with (n-r). I use the extra letters and include the relationships between them. The letters r and w stand for right and wrong. The letter p and q are standard in probability texts for the probability of a success or a failure.

Let's do an example. You are given a four question multiple choice test, each question having five possible answers. The test is given in a language you do not read, so all you can do is guess. Each question is independent from the others, meaning that if C is the right answer to the first question, it's also possibly the answer to the second. The probability p of a correct guess is 1 chance in 5, or .2, The probability of failure q is 1-.2 = .8, and of course p + q = 1.

Probability of no correct answers = C(4,0)*.2^0*.8^4 = .4096
Probability of exactly one correct answer = C(4,1)*.2^1*.8^3 = .4096
Probability of exactly two correct answers = C(4,2)*.2^2*.8^2 = .1536
Probability of exactly three correct answers = C(4,3)*.2^3*.8^1 = .0256
Probability of four correct answers = C(4,4)*.2^4*.8^0 = .0016

The expected value of correct answers is n*p, so in this case it's 4*.2 = .8, which isn't possible. You can't get a fraction of correct answers on a multiple choice test. The expected value in this case says that over the long run, a test like this should average .8 right answers out of four. As we can see, the most likely thing to happen is actually a tie for first, where getting either no answers right or one answer right both have a probability of about 41%. If you need to get three answers right to pass the test, the odds are less than 3% to get either three or four right, and the odds of getting everything right by chance is a very slim 16 chances in 10,000.

If you have a TI-83 or TI-84, there is a function under the distribution menu called binompdf(n,p,r). All you have to is enter the function, then the three values in the order given, separated by commas.

The function for three right in four trials with probability .2 at each trial is binompdf(4, .2, 3), which as we see above is .0256.


Practice problem.
The test is changed. There are now five multiple choice questions and four choices for each, but it is still given in a language you do not read.

Round the probabilities to four places after the decimal.

1. What is the expected value?

2. What is the probability of no correct answers?

3. What is the probability of exactly one correct answer?

4. What is the probability of exactly two correct answers?

5. What is the probability of exactly three correct answers?

6. What is the probability of exactly four correct answers?

7. What is the probability of five correct answers?

Answers in the comments.
  

Notes for February 11 and 13

Reverse look-up of z scores: Percentiles to z-scores, z-scores to raw scores

So far, we only use the percentile look-up table if we know a data set is normally distributed. In these cases, we are given the average mux and the standard deviation sigmax and find a z-score z(x) for a raw score x using this formula.

z(x) = (x - mux)/sigmax

We can then use this z-score to look up a percentile on the orange look-up table.  

Example: For men, the average height is 70.5 inches (5' 10.5") and the standard deviation is 2.8 inches. What is the z-score for a man 6'0" tall (72 inches)?

z(72) = (72 -70.5)/2.8 = 1.5/2.8 = 0.535714..., which rounded to the nearest hundredth is 0.54. Using our lookup table in row 0.5 and column 0.04, we get .7054, which means a man 6'0" is as tall or taller than 70.54% of the male population. Subtracting from 100%, we can also say that 29.46% of the male population is 6'0" or taller.

What if instead we wanted to find the cut-off height for the 75th percentile of men. What we have to find is when the table has two consecutive numbers that look like this

... .74xx .75xx ...

What that would mean is that between those two z-score will be the cutoff for the 75th percentile. Looking on the positive side of the chart we find

0.6 ... .7486 .7517 ...

In the columns 0.06 and 0.07. This means the 75th percentile lies in between 0.66 and 0.67. Since .7486 is .0014 below .7500 and .7517 is .0017 above, it's fair to say the 75th percentile is about half way in between, and 0.665 is a good approximation.

Now that we have a z-score, we use this formula to find the raw score x.

x = mux + z(x) × sigmax

In this case we get

x = 70.5" + 0.665 × 2.8 = 70.5" + 1.862" = 72.362" = 6" 0.4"

This says the 75th percentile of men's heights is at just below six feet and one half inch tall.

What about the percentage of men between 5'11" and 6'2". Here were find the percentages for both hieghts, then subtract the smaller percentage from the larger.

6'2" = 74", and z(74) = (74-70.5)/2.8 = 3.5/2.8 = 1.25

z = 1.25 corresponds to 0.8944

5'11" = 71" and z(71) = (71-70.5)/2.8 = 0.5/2.8 = 0.17857... = 0.18

z = 0.18 corresponds to 0.5714

0.8944 - .5714 = 0.323, which says about 32.3% of men fall between exactly 5'11" and exactly 6'2" tall.

The Central Limit Theorem 

 If we have a normally distributed population and take a random sample, z(x-bar) is different from z(x) by changing the standard deviation. The simplest way to compute z(x-bar) is to take the z-score the normal way then multiply by the square root of n, the size of the sample. For example, if 10 men averaged to a height of 6'0", they would have a different z-score and correspond to a different proportion of the population.

 z(72) = (72 -70.5)/2.8 × sqrt(10) = 1.694..., which rounds to 1.69. Looking up in row 1.6 and column 0.09, we get .9545.  This says a sample of 10 men averaging 6'0" tall is taller than 95.45% of all samples of ten men, and only 4.55% of all samples of ten men are shorter than that.

This rule is called the Central Limit Theorem. We will be using it to try to get estimates of the average of a population using the average of a sample that is assumed to be representative.

Notes for 4 February 2014

Two different versions of standard deviation of a numerical data set, sx and sigmax

The two formulas here are standard deviations for a numerical data set. When we have a sample, we use sx. When the data set is a population, we use sigmax. We get two different values because the denominator in the first case is n-1 and N in the latter case. The reason for this is degrees of freedom.

The idea of degrees of freedom is to count how much information you need to get an answer. For example, if a football game ends in regulation we know that

final score = score in 1st + score in 2nd + score in 3rd + score in 4th

There are five pieces of information, but if you have any four of these numbers, you can find the fifth. Here we would say the degrees of freedom are 4, which is 5-1. For example, if we know the Seahawks scored 43 points total in the Super Bowl, and the scored 8 points in the first quarter, 14 points in the second quarter and 14 points in the third, we can figure out how much they scored in the 4th quarter without being told.

score in 4th = final score - score in 1st - score in 2nd - score in 3rd = 43 - 8 - 14 - 14 = 7

This is a situation where the degrees of freedom are n-1, just like with the standard deviation of the sample. The idea is that if we know the average and somehow we get all the scores except for one, we can find the last score by multiplying the average by n then subtracting all the scores we know to find the one score we don't know.

A new way to measure data: z-scores

 
We can compare two sets of data against one another using the averages (x-bar or mux) and the standard deviation (sx or sigmax) with a formula known as the z-score. We subtract the average from the raw score x. If x is above average, we will get a positive number, if x is below average, we will get a negative number. (if x is exactly at teh average, we'll get zero.) We then divide by the standard deviation to get the z-score. This tells us how many standard deviations we are away from average, either high or low.

Here is an example using the American and National League final standings last year. The data set we will check out is the number of wins. I will treat both leagues as samples of all of Major League Baseball and the reason there is a difference in the average wins is because of inter-league play.



American League:
x-bar = 81.3
sx = 13.7

National League:
x-bar = 80.7
sx = 11.1

The AL had slightly more wins, but their standard deviation is higher because the data is more spread out, largely because the Houston Astros were so terrible. In any data set, we can use z-scores to find outliers.

z is greater than 3: The score is very unusually high
z is greater than 2: The score is unusually high
z is less than -3: The score is very unusually low
z is less than -2: The score is unusually low

The Astros had 51 wins, so their z-score is (51-81.3)/13.7 = -2.21. They are the only team in either league that can be considered an outlier, either high or low. 

We can also compare teams in the different leagues. For example, both the As and the Braves had 96 wins, but because they are in different leagues they won't have the same z-score.

z-score for As: (96-81.3)/13.7 = 1.07
z-score for Braves: (96-80.7)/11.1 = 1.38

What this would say is that it was more impressive for the Braves to win 96 that for the As. A big part of this is that the Athletics played 19 games against the Astros, winning 15 and losing 4. The Braves never played the Astros, and that alone meant they got the same number of wins against a tougher set of opponents, which accounts for their higher z-score.

Z-scores in normally distributed sets

Not every set can be assumed to be normally distributed. Usually, we will be told a set is normally distributed and be given both mux and sigmax. If we have such a set, we can take a raw score and turn it into a percentile using the look-up table we got on the orange hand-out on Tuesday. Let's take an example.

It is assumed that IQ scores are normally distributed, where the average is 100 and the standard deviation is 15. This would say an IQ of 110 has a z-score of (110-100)/15 = 0.67. Because of our assumption, we can use the look-up table to find the percentile for an IQ of 110.

1. Use the Positive z-score side
2. Look in the row next to the 0.6 label in bold
3. Look in the column labeled 0.07.

The value at that position on the table is .7486. What this means is that 74.86% of the population has an IQ of 110 or less. If we subtract 74.86% from 100%, we get 25.14%. Rounding to the nearest percent, this means the 110 IQ is at about the cutoff for the 75th percentile.

We will discuss this in greater detail on Tuesday, February 11.

Notes for 30 January 2014

Standard deviation formulas for samples and populations

The new topic on Thursday was standard deviation, a measure of how far spread out a data set is based on the average. (Note that the five number summary is also about the spread of the data, but it is based on the median.) For the first time, not only are there different symbols for standard deviation, sx for the sample and sigmax for a population, but the formulas to derive the values are different as well. (For average, the formulas for x-bar and mux are essentially the same.)

There are different methods for finding the standard deviation (you can see the alternate formulas at this page on the blog), but the one present here is the simplest computationally. I will not force you to compute these by hand, but if you don't have a calculator that has statistic functions, this is the easiest way to do the job.

Let's take the hockey scores data from earlier this week. The length of the list is 22, which we can think of as n, size of a sample or N, the size of a population.

7, 6, 5, 5, 5, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 2, 2, 1, 1, 1, 0, 0

These are the x values. The sum is 69.  We also need the x² values and their sum.

49, 36, 25, 25, 25, 16, 16, 16, 16, 9, 9, 9, 9, 9, 9, 4, 4, 1, 1, 1, 0, 0

The sum of the x² is 289. This means the numerator of both fractions is

289 - 69²/22 = 72.59090909...

The square root of 72.59090909/22 is 1.81647..., which is the value for sigmax.

The square root of 72.59090909/21 is 1.859222..., which is the value for sx.

The reason for the difference in formulas is a math concept called degrees of freedom, which we will discuss on Tuesday.

Notes for 28 January 2014

Frequency tables

On last Thursday, we took a first look at stem and leaf plots, a method for writing a data set using less space and less ink (or if we are on the Internet, less bandwidth). For some data sets that have a lot of duplication, we can use a frequency table, where the numerical values x are matched up with their corresponding frequencies f(x).  Here is an example using hockey scores.

x || f(x)
7 || 1
6 || 1
5 || 3
4 || 4
3 || 6
2 || 2
1 || 3
0 || 2 

This set isn't that big, so let me write it out as a list in order.

7, 6, 5, 5, 5, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 2, 2, 1, 1, 1, 0, 0

If we want to find the standard statistics for mean, median and mode, and the length of the list n, here are the methods. 

Length of list n. we take the sum of the f(x) column.


x || f(x)
7 || 1
6 || 1
5 || 3
4 || 4
3 || 6
2 || 2
1 || 3
0 || 2 
  ||22

The list has 22 values. We can use this to find mean and median.

Mode. Mode is easy using a frequency table, we see the biggest f(x) is 6 and it corresponds to x = 3. That means the mode is 3, the number that shows up most often.

Mean. We need to know the sum of all 22 values. For this we need to multiply x*f(x). The idea is that there is one 7, so that adds 7 to the total, but there are three 5s, so that adds 15 to the total.

x || f(x) || x*f(x)
7 || 1        7
6 || 1        6
5 || 3       15
4 || 4       16
3 || 6       18
2 || 2        4
1 || 3        3
0 || 2        0
  ||22   ||  69

The average or mean is 69/22 = 3.1363636... Since the data is all whole numbers, we round the average to the nearest tenth, which we would write as x-bar = 3.1

Median. To find the median, we need to find the median position, which is always (n + 1)/2. In this case, that is (22+1)/2 = 23/2 = 11.5. The median will be the average of the 11th and 12th numbers on the list. Here's how we find those numbers using the frequency table.

x || f(x) || positions
7 || 1        1
6 || 1        2
5 || 3      3 through 5
4 || 4      6 through 9
3 || 6     10 through 15
2 || 2     16 through 17
1 || 3     18 through 20
0 || 2     21 through 22

I went to the end of the list, but I could have stopped as soon as we knew that the value 3 was in positions 10 through 15. That means the 11th number is a 3 and so is the 12th number. The average of 3 and 3 is

(3+3)/2 = 3.

In this case, the median is 3.

Relative frequencies

We are now dealing with proportions, and the formulas are as follows:
Population: p = F/N
Sample: p-hat = f/n
We often want to compare one proportion to another, either two from the same sample or comparable proportions from different samples, or the sample proportion to the population proportion. Because of this, it is better to write the numbers as decimals or in scales based on powers of 10.
Scales based on powers of 10: The most famous scale base on powers of ten in percentage, which really means "per 100". It is much more common to see "53% of the people agree with the president's plan" than ".53 of the people..." or "53 out of every 100 people...". Technically, all those phrases are saying the same thing, but percentage is the most popular.
One of the places where decimals are used for proportions is in the sports pages. A batting average in baseball (hits/at bats) is given as a percent to three decimal place, and likewise winning proportions (win/total games) are written as .xxx. If a batter has 27 hits in 92 at-bats, the batting average 27/92 = .293478261... is shortened to .293 and pronounced "two ninety three". Likewise, a team who has won 17 games and lost 5 will have a winning proportion of 17/22 = .77272727... = .773, and often stated as "team has a winning percentage of seven seventy three." Technically, this is a mistake, because "percentage" means out or 100. The correct word from the dictionary, which no one ever uses, is "permillage", which means out of 1,000. The team in question would have a winning percentage of 77, and a winning permillage of 773.
In both of the cases from the sports pages, the greater number of place after the decimal is used to break ties. For example, a team with 14 wins and 4 losses is at .778, which is better than 17 wins and 5 losses, while 20 wins and 6 losses is at .769, so is slightly worse.
To get a number based on a power of 10 scale, you take the proportion and multiply by the power of ten, so it is either p*scale or p-hat*scale, depending on population or sample. Besides greater precision for breaking ties, sometimes we need greater precision because the proportions are so small.
When I ask a class what is the legal limit for blood alcohol while driving, invariably someone will say "point oh eight" and most people will agree. But .08 is wrong. .08 = 8%, and the correct answer is .08% = .0008. I don't blame the students. The number is badly represented and it is an easy mistake to make. Let's take a look at the number on other scales of 10.
.08 out of 100 is the same as
.8 out of 1,000 0r
8 out of 10,000 or
80 out of 100,000
80 parts out of 100,000 is a tiny proportion. To give an idea, ounce of pure alcohol mixed into ten gallons of blood would give you 78 parts out of 100,000, and most people have between a half gallon and a gallon and a half of blood in their body, between 4 and 12 pints. The amount of alcohol in a person's blood stream that is over the legal limit is about the same amount of alcohol as found in a capful of mouthwash used after brushing your teeth.
We will be dealing with much smaller proportions later in the class, where there are things that can be hazardous to your health at ranges measure in parts per billion, but for now, we will look at the per 100,000 scale for another type of statistic, measurements of mortality rates.
Here are the number of homicides in some American cities in 2013.


Chicago: 415
New York City: 333
Detroit: 333
Washington, DC: 103
Oakland: 92

This makes it look like Chicago is most dangerous and Oakland is safest, but this doesn't take into account the populations of the cities. Here is the list with populations rounded to the nearest ten thousand.

Chicago: 415 murders, pop. 2,720,000
New York City: 333 murders, pop. 19,650,000
Detroit: 333 murders, pop. 700,000
Washington, DC: 103 murders, pop. 650,000
Oakland: 92 murders, pop. 390,000

Because the number of murders is much smaller than the populations, if we just divide we will get a very small decimal number. For example in Chicago.

415/2720000 = 0.0001525735...

With death statistics, the scale of measurement is usually number per 100,000. To get that, we multiply the answer by the scale 100,000.

0.0001525735... * 100,000 = 15.25735...

Rounding to the nearest tenth, we would say the murder rate per 100,000 people in Chicago last year was 15.3.  Here are all the numbers, rounded to the nearest tenth.

Chicago: 415 murders, pop. 2,720,000 15.3 per 100,000
New York City: 333 murders, pop. 19,650,000 1.7 per 100,000
Detroit: 333 murders, pop. 700,000  47.6 per 100,000
Washington, DC: 103 murders, pop. 650,000 15.8 per 100,000
Oakland: 92 murders, pop. 390,000 23.6 per 100,000

Looking at the number on this scale, it is clear that Detroit has a much higher murder rate than the other cities listed Oakland has the second highest murder rate. Chicago and DC are very similar and New York City's rate is very low.