Notes for February 18 and 20

A probability is a number between 0 and 1, inclusive, and is represent by p(event) if part of a sample or p-hat(event) if part of a population.

A probability space or event space is a list of what we call the simple events. The list follows two rules.

1. Every simple event is mutually exclusive from every other simple event, which means the cannot both happen simultaneously. For example, if I flip one coin, it cannot land "heads" and "tails" simultaneously.

2. When we add up the probabilities of all the simple events, the sum is 1. What this means is the simple events (sometimes called simple outcomes).

There are ways to create new probabilities from a set of simple events. The first methods we will discuss are AND, OR and NOT. Let's start with the most basic, NOT.

p(NOT x) is the probability that the event x will not happen. It is sometimes written as q(x) instead. It is always true that

p(x) + p(NOT x) = p(x) + q(x) = 1

Example #1: If we flip a coin, we have two possibilities, heads and tails. We usually assume that p(heads) = .5 and p(tails) = .5, but that doesn't have to be true. What is true in this cases with a categorical variable that has only two legitimate values is that p(NOT heads) = q(heads) = p(tails), and the sum is 1. This means if p(heads) = .51, then q(heads) = 1 - .51 = .49.

If we roll a six-sided die, we have six possible outcomes, a 1, a 2, a 3, a 4, a 5 or a 6. The probability for not rolling a 1 is  

p(NOT 1) = p(2) + p(3) + p(4) + p(5) + p(6)

Because these are simple events, we don't have to worry about overlap. And again, instead of calling it p(NOT 1), we can call it q(1).

If two events A and B are mutually exclusive, then

p(A OR B) = p(A) + p(B)

If they are not mutually exclusive, that means p(A AND B) does not equal zero, and the rule for finding OR changes to   

p(A OR B) = p(A) + p(B) - p(A AND B)

This is most easily explained with a contingency table. Let's say we have two variables for a game, one called Result (which is either a win or a loss) and Setting (which is either home or away). The easiest way to represent this situation is a contingency table, like the one below listing wins and losses for the Warriors so far this season, broken into home record and away record.

___H || A|| Total
W |16||15|| 31
L |10||12|| 22
  |26||27|| 53 = grand total

p(Home) = 26/53, which rounded to the nearest thousandth is .491
p(Away) = 27/53, which rounded to the nearest thousandth is .509. Notice that p(Home) = q(Away) and vice versa.
p(Wins) = 31/53, which rounded to the nearest thousandth is .585
p(Losses) = 22/53, which rounded to the nearest thousandth is .415. Notice that p(Wins) = q(Losses) and vice versa.

p(Wins AND Home) = 16/53, which rounded to the nearest thousandth is .302. What this represents is the number of home wins divided by the total number of games.

p(Wins OR Home) is the probability that a game picked at random is either a home game or a win. We will get this by adding all the wins to all the home games, but we have to subtract the home wins, because they were counted twice.

p(Wins OR Home) = 31/53 + 26/53 - 16/53 = 41/53 or .774 rounded to the nearest thousandth.


Besides AND, OR and NOT, we have the qualifier GIVEN. In a contingency table, this means we only look at the number is a single row or column.

p(Wins GIVEN Home) = 16/26 or .615 rounded to the nearest thousandth.

p(Home GIVEN Wins) = 16/31 or .516 rounded to the nearest thousandth.

Notice that p(Wins) = .585 but p(Wins GIVEN Home) = .615. When these numbers are different, we say that the two categories Wins and Home are dependent. If they were the same, we would say the categories are independent. A contingency table that was independent might look like this


___H || A|| Total
W |16|| 8|| 24
L |10|| 5|| 15
  |26||13|| 39 = grand total

Now, p(Wins) = p(Wins GIVEN Home) = p(Wins GIVEN Away). This hypothetical win-loss record is at the same proportion whether on the road or at home.

Binomial distribution of an independent variable


If I say someone is a 70% free throw shooter, is every free throw attempt independent of what happened before? Often, we set up such an experiment assuming independence just to make our work simpler, but the human factor is involved, so in reality it's very likely to be dependent. Some people get frustrated after a few misses and will do worse. Others will learn from the mistakes of a few misses and figure out what they are doing wrong and make improvements. A player might be having a bad day for some reason, or might instead have excellent concentration or just really good luck that day. But again, these kinds of experiments are often set up as though each free throw trial is independent of what came before.

Let's look at flipping coins. A list of all possible events is called the event space. Here are some examples of event spaces.

Event space for flipping one coin
Heads (H)
Tails (T)

ways to get one head = 1
ways to get no heads = 1

Event space for flipping two coins
HH
HT
TH
TT

ways to get two heads = 1
ways to get one head = 2
ways to get no heads = 1

Event space for flipping three coins
HHH
HHT
HTH
HTT
THH
THT
TTH
TTT

ways to get three heads = 1
ways to get two heads = 3
ways to get one head = 3
ways to get no heads = 1


The list of numbers of ways to get r successes in n trials is often written in the pattern of the picture shown here, and this pattern is called Pascal's Triangle, at least in most of the world. The Italians call it Tartaglia's Triangle and the Chinese call it Yanghui's Triangle. None of these people actually invented it or claimed to have invented it. It's been around since before the time of Christ, and it has been studied all around the world.

While it is very common to see it presented in the form here as an equilateral triangle, it can also be presented where the first numbers in each row are lined up straight as follows

1
1 1
1 2 1
1 3 3 1
1 4 6 4 1
... etc.

It is standard to start counting the top row as row 0, and the left most column as column 0. For example, the 6 we see in the middle of the last row I typed in is row 4, column 2. Instead of having a copy of Pascal's Triangle around, our calculators have these numbers available. On Texas Instruments calculators, the function is under the probability menu. On the TI-30XIIs, the way to get that 6 is to type

4 [prb][right arrow]2[enter]

The calculator will read

4 nCr 2
6

All scientific calculators should have this function available, but all of them are slightly different. The TI-89 writes it as nCr(4,2) and Casio calculators write it as 4 C 2. I will pronounce it "4 choose 2", and when I type on the blog, I will type C(4,2). When I write it on the board or on tests, I will put a 4 on top of a 2 and surround both numbers with a large parentheses. These numbers are called the binomial coefficients.


The formula for finding the probability for exactly r successes in n independent trials where the probability of success on any single trial is p is shown here. In some books, they don't use the letter q, instead replacing it with (1-p). Likewise, sometimes w is replaced with (n-r). I use the extra letters and include the relationships between them. The letters r and w stand for right and wrong. The letter p and q are standard in probability texts for the probability of a success or a failure.

Let's do an example. You are given a four question multiple choice test, each question having five possible answers. The test is given in a language you do not read, so all you can do is guess. Each question is independent from the others, meaning that if C is the right answer to the first question, it's also possibly the answer to the second. The probability p of a correct guess is 1 chance in 5, or .2, The probability of failure q is 1-.2 = .8, and of course p + q = 1.

Probability of no correct answers = C(4,0)*.2^0*.8^4 = .4096
Probability of exactly one correct answer = C(4,1)*.2^1*.8^3 = .4096
Probability of exactly two correct answers = C(4,2)*.2^2*.8^2 = .1536
Probability of exactly three correct answers = C(4,3)*.2^3*.8^1 = .0256
Probability of four correct answers = C(4,4)*.2^4*.8^0 = .0016

The expected value of correct answers is n*p, so in this case it's 4*.2 = .8, which isn't possible. You can't get a fraction of correct answers on a multiple choice test. The expected value in this case says that over the long run, a test like this should average .8 right answers out of four. As we can see, the most likely thing to happen is actually a tie for first, where getting either no answers right or one answer right both have a probability of about 41%. If you need to get three answers right to pass the test, the odds are less than 3% to get either three or four right, and the odds of getting everything right by chance is a very slim 16 chances in 10,000.

If you have a TI-83 or TI-84, there is a function under the distribution menu called binompdf(n,p,r). All you have to is enter the function, then the three values in the order given, separated by commas.

The function for three right in four trials with probability .2 at each trial is binompdf(4, .2, 3), which as we see above is .0256.


Practice problem.
The test is changed. There are now five multiple choice questions and four choices for each, but it is still given in a language you do not read.

Round the probabilities to four places after the decimal.

1. What is the expected value?

2. What is the probability of no correct answers?

3. What is the probability of exactly one correct answer?

4. What is the probability of exactly two correct answers?

5. What is the probability of exactly three correct answers?

6. What is the probability of exactly four correct answers?

7. What is the probability of five correct answers?

Answers in the comments.
  

Class notes for 5/4, part 3: Bayesian probability and double testing

If a trait is very rare, only a very accurate test gives us useful information. For example, if a trait shows up in only 1 in 10,000 people but the test for the trait has an error rate of 1 in 1,000, we should expect about 10 false positives for every true positive. Here is the completed table for that situation.

________don't___have____row total

test + ____9,999__999______10,998__

test - _9,989,001___1_____9,989,002__

col.___9,999,000_1,000______10,000,000 grand total

In a situation such as this, testing positive twice could give us useful information, as testing positive once has an error rate of about 90.9%. We have to assume the errors are random and not deterministic. For example, if a test for a chemical compound in opium also catches a similar compound found in poppy seed bagels, testing twice won't get rid of the errors. Assuming just random errors here is what we do.

Step 1: The top row of the first contingency table is the column total/grand total row of the second contingency table. What this does is takes the numbers from the people who tested positive the first time and makes them the totals for those who will be tested twice.

________don't___have____row total

test + __________________________

test - __________________________

col._______9,999__999_____10,998 grand total

Step 2: Multiply error rate by have column total to find the number who have that test negative. Round to the nearest whole number. (We didn't have to round before, but now we do.)
999*1/1000 = .999 ~= 1, this means test positive and have is 998.

________don't___have____row total

test + ___________998____________

test - _____________1____________

col._______9,999__999_____10,998 grand total

Step 3: Multiply error rate by don't have column total to find the errors. 9,999*1/1000 = 9.999 ~= 10. That means the test negative in that column is 9,999 - 10 = 9,989.

________don't___have____row total

test + _______10__998____________

test - _____9,989____1____________

col._______9,999__999_____10,998 grand total

Step 4: row totals

________don't___have____row total

test + _______10__998_____1,008___

test - _____9,989____1_____9,990___

col._______9,999__999_____10,998 grand total

Step 5: Find the error rate for testing positive twice. 10/1,008 = .0099... or about 1%.

Of the ten million people tested, we would send letters to 1,008 telling them they tested positive twice. Of those people, ten don't have the trait and are getting false information, but 998 are getting the right information. In the first test, there was someone with the trait who tested negative, and the same is true in the second test, so there are two people with the trait who did not get two positive test results. While this isn't a perfect situation, it's much better than the over 90% error rate we got for positive tests the first time through.

Class notes for 5/4, part 2: Bayesian probability

Earlier in the term, we created contingency tables from reading data sets and filling in the positions of the table, then finding the row totals, column totals and the grand total. We then learned about conditional probability, where we find that p(left, given female) might not equal p(left, given male) or p(left). If these probabilities are not equal, we call them dependent, because it depends on if we are looking a the whole population or some specific sub-population. If they are all equal, the probabilities are independent.

In Bayesian probability, we will be building a contingency table "backwards". Instead of filling in each value in the table then finding row totals, column totals and grand total, we will start with a trait in the population and a test for that trait. We will make a 2x2 contingency table, where the columns deal with having the trait on not and the rows refer to testing positive of testing negative. If the test has an error rate, as is often the case, some people are going to get incorrect information. What we will see is that the overall error rate can sometimes be quite different for the error rate for those who test positive and the error rate for those who test negative.

Let's say there is a genetic trait in the population that shows up in 25% of subjects, which we will write as 1 in 4. The test for the trait has a 2% error rate, so it is 1 in 50.

Step 1: The grand total is the product of the denominators of the fractions.

In our case, 4*50 = 200


________don't___have____row total

test + __________________________

test - __________________________

col._____________________200 grand total

Step 2: Multiply the grand total by the trait proportion to find the column totals.

Since 25% of the population has the trait, 25% of 200 = 50 subjects have the trait in our idealized sample. By subtraction, 150 don't have the trait.


________don't___have____row total

test + __________________________

test - __________________________

col._____150_____50______200 grand total

Step 3: Fill in the "have the trait" column by multiplying the error rate by the column total to fill in the mistaken position, and fill in the rest by subtracting.

In our case, the error rate is 1 in 50. This means for the people who have the trait, 1 person will test negative, while the other 49 will correctly test positive.

________don't___have____row total

test + ___________49______________

test - ____________1______________

col._____150_____50______200 grand total

Step 4: Fill in the "don't have the trait" column using the same method.

3 of the 150 will get the wrong information, which in their case will be a positive test. The other 147 will get the right information, a negative test result.

________don't___have____row total

test + ____3______49______________

test - ___147______1_______________

col._____150_____50______200 grand total

Step 5: Fill in the row totals.

________don't___have____row total

test + ____3______49______52______

test - ___147______1______148______

col._____150_____50______200 grand total

The error numbers are marked in bold and blue for the next step.

Step 6: Find the error rates given test positive and test negative.
p(error) = (3+1)/200 = 1/50 = .02, which was the advertised error rate.
p(error, given test positive) = 3/52 ~= .058, much higher than .02
p(error, given test negative) = 1/148 ~= .0068, much lower than .02

Unless the trait shows up in 50% of the population, we expect to get differences between the error rates for test positive and test negative. Whichever is the smaller part of the population should see a higher error rate. Just how significant the differences are in the error rates between the two test groups depends on the size of the error rate to the trait rate. A 99% accurate test sounds good, but if the trait is very rare, we might well get more false positives than true positives.

Class Notes for 4/27 and 4/29

Dependence and Independence
The first line of Leo Tolstoy's Anna Karenina is "All happy families are alike; each unhappy family is unhappy in its own way." In statistics, all independent trials are alike, in that the probability of a particular outcome of one trial does not effect the outcome of later trials, nor was it effected by earlier trials. With dependent probability, the outcome of one trial is effected by the outcome of previous trials, but how that effects things is not always the same.

For example, if we talk about a 70% free throw shooter taking two shots, does missing the first shot effect the probability of missing the second shot? Let's look at this simple problem three different ways.

Predicting the future mathematically by carefully studying the past: Let's say I called this person a 70% free throw shooter because so far in the season, she has made 7 of 10 shots from the line. If she misses, she has now made 7 of 11 shots from the line, and she is now a 63.6% shooter. Should we factor that in to be more precise? Some mathematical models would say yes.

Using this method, missing a shot would make her percentage worse and making one would make it better, no matter how many shots she had taken. But if she had made 70 of 100 so far in the season, one miss would make her 70 of 101, which would lower her percentage to 69.3%, a much smaller effect than it has if she had made 7 of 10. If instead we were looking at her entire career instead a single season, perhaps she has made 700 of 1000, and missing one would mean she was 700 of 1001, which changes her percentage to 69.9%. When discussing free throw percentage, announcers on TV usually round to the nearest percent, so the first example of 7 of 10 to 7 of 11 would be a drop from 70% to 64%, the second example would be a drop from 70% to 69%, and in the third example, 69.9% would round to 70% and the change would be too small to notice.

Factors effecting success and failure: The free throw shooter goes to the line for two shots and misses the first. Is there a physical reason? We might treat free throw shooting as we would picking a random number from 1 through 100, where we count any number from 1 through 70 as a success and 71 through 100 as failures, but shooting free throws includes the human factor. Maybe she missed because she is nervous or distracted. Maybe it's late in the game and she is tired or injured, changing her technique. If any of these are the case, it might make more sense for us to downgrade the probability of making the next shot, though exactly how much it should be downgraded is no longer some simple formula like turning a fraction into a percentage.

Compensating for failure: Again, let's add in the human factor in this problem. She misses the first free throw, and her coach notices her technique looks inconsistent. "Elbow up!" the coach shouts from the sidelines, and the shooter hears the coach and readjusts her technique to match the way she shoots in practice. Should this change bring her back to being a 70% shooter, or even upgrade her chance of success? That is uncertain, but the failure on the first shot and the diagnosis of at least one reason for the failure could effect the odds, and that effect means the second shot should not be considered independent of the first.

The dependent probabilities in a 52 card deck

One of the simplest mathematical models of dependency is sampling without replacement, which is the way most card games or lotteries or the game of Bingo works. You have a set of outcomes which get effectively randomized and a trial is performed, meaning a card is taken from the deck or a ping pong ball is removed from the hopper or a bingo marker is removed from the spinner. Once removed, the number of possible outcomes has been reduced by one and probabilities for success and failure of certain outcomes change.

Looking for an ace: There are 52 cards is a standard deck and 4 of them are aces. If I draw a card from a randomized deck, the chances are 4/52 = 1/13 ~= 7.7% that the card will be an ace. What are the chances the second card is an ace?

That depends on the first card.

Probability that the second card is an ace, given the first card is an ace is 3/51 = 1/17 ~= 5.9%.

Probability that the second card is an ace, given the first card is not an ace is 4/51 ~= 7.8%.

Unlike the mathematical model of free throw shooting where we re-calculate the probabilities by adding the most recent make or miss into the precentage, which means a miss brings the odds down and a make brings the odds up, not getting an ace makes the odds a little better next time, and getting an ace makes the odds worse.


This is the formula for the dependent probability model of sampling without replacement is given at the left. The two numbers in parentheses are a binomial coefficient, the numbers you get when you use nCr on your calculator, which I pronounce "n choose r" in class. The pairs of numbers that look like a base and an exponent, except that the exponent is underlined, is the convention developed by Donald Knuth at Stanford for writing the numbers that you get on your calculator using the nPr function, which I pronounce "n fall r", referring to the name "the falling factorial". If we think about a deck of cards, the lowercase letters refer to the size of the hand n, where r is the number of successful trials (r for right) and w is the number of unsuccessful trials (w for wrong), and r+w=n. The uppercase letters refer to the size of the deck, where T is the size of the deck, G is the number of cards we consider success if we draw them and B is the number of cards we consider a failed trial if we draw them. The letter T stands for Total, G for Good and B for Bad. Again, we have an equation, G+B=T.

Example: If we want consider drawing a heart a success and anything else a failure, what is the probability of drawing three hearts and two non hearts in a five card hand from a well-shuffled 52 card deck.

Here are the six numbers we need.
n = 5
r = 3
w = 2
T = 52
G = 13
B = 39

On a TI-30XIIs, here are the keys you would press.

5[prb][right]3×13[prb]3×39[prb]2÷52[prb]5[enter]

The calculator will read as follows.

5 nCr 3*13 nPr 3*39 nPr 2/52 nPr 5
0.081542617

This means the probability of exactly three hearts and two cards of some other suit is about 8.15%.

The Expected Value (EV) of a two outcome game

Let us assume we have a game that has only two outcomes, winning and losing. Let us further assume that two players have decided to wager on this game, both putting money into a pooled amount and the winner taking all at the end.


If we look at the game from the point of view of one of the players, we need to know the probability of winning p, how much that player put in, which we call Risk and how much the opponent put in, called Profit. The expected value EV equals the probability of victory p times the sum of Profit and Risk divided by Risk.

Different books use different formulas for this. Some do not divide by Risk. By dividing, the number we get is a percentage of return, and a game of flipping coins for $1 a game is equivalent to a game of flipping coins for $100 a game. Some subtract 1 from this formula. This just changes the most important number in identifying results from 1 to 0.

If EV = 1, we consider this a "fair game". For every $1 risked on this game, the expected value is that you will have that dollar returned to you, breaking even. Notice if we are flipping coins, that event never happens on any single play. Either the player makes a dollar profit or a dollar loss, but expected value is about the long run.

If EV > 1, the game is advantageous to the player. If EV < 1, the game is disadvantageous to the player.

In the game of roulette, there are 38 slots where the ball can land, and for simplicity's sake we will assume each has an equal chance of showing up, so p = 1/38. For every $1 you risk, you can make a profit of $35 if you correctly guess in the exact slot where the ball will land. To find the expected value using the TI-30xIIs, you should type in this.

1÷38×(35+1)÷1[enter]

The calculator will read as follows.

1/38*(35+1)/1
0.947368421

What this number means is that for every dollar risked on the spin of a roulette wheel, you should expect about 94.7 cents returned to you in change. In other words, about 5.3 cents is lost from every dollar you bet on every spin of the wheel.

Another way to play the game is to bet red or black. Of the 38 compartments, 18 are red and 18 are black and 2 are green. The probability of victory on betting one of the two major colors is 18/38 = 9/19 ~= 0.473684211. The profit and risk are now both $1. Here's what to type on the TI-30xIIs.


16÷38×(1+1)÷1[enter]

The calculator will read as follows.

16/38*(1+1)/1
0.947368421

The game has changed, both in probability and amount of profit compared to risk, but from the player's point of view, the expected value is precisely the same and still in favor of the casino.

No matter what the levels of profit and risk are, we can find a probability p that will make the expected value equal to 1, and that is p = Risk/(Profit+Risk). If the probability is increased with the profit and risk remaining unchanged, the game becomes advantageous. If is is decreased, the game becomes disadvantageous.

Modern and Classic Parimutuel odds

Profit and risk are listed either in classic form like 3-1 or 2-7 (or sometimes with colons 3:1 or 2:7), where profit is the first number and risk is the second.

In online betting sites, the numbers are given as numbers with absolute value greater than 100, with either a + or - in front of them. +250 means 250 is the profit and 100 is the risk, while -250 means 10o is the profit and 250 is the risk. The fourth page of the yellow sheet explains this in greater detail and shows how to switch back and forth between the two systems.

Practice problems

1. With a well-shuffled 52 card deck, find the probability of getting exactly r hearts in a five card hand when
a) r = 0
b) r = 1
c) r = 2
d) r = 3 (already solved above)
e) r = 4
f) r = 5

2. Find the break-even p when Profit and Risk are as given. Round to three places after the decimal point.

a) Modern parimutuel = +150
b) Modern parimutuel = -110
c) Classic parimutuel = 5:3
d) Classic parimutuel = 5:11

Answers in the comments.