Showing posts with label relative frequency. Show all posts
Showing posts with label relative frequency. Show all posts
Wednesday, January 29, 2014
Notes for 28 January 2014
Frequency tables
On last Thursday, we took a first look at stem and leaf plots, a method for writing a data set using less space and less ink (or if we are on the Internet, less bandwidth). For some data sets that have a lot of duplication, we can use a frequency table, where the numerical values x are matched up with their corresponding frequencies f(x). Here is an example using hockey scores.
x || f(x)
7 || 1
6 || 1
5 || 3
4 || 4
3 || 6
2 || 2
1 || 3
0 || 2
This set isn't that big, so let me write it out as a list in order.
7, 6, 5, 5, 5, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 2, 2, 1, 1, 1, 0, 0
If we want to find the standard statistics for mean, median and mode, and the length of the list n, here are the methods.
Length of list n. we take the sum of the f(x) column.
x || f(x)
7 || 1
6 || 1
5 || 3
4 || 4
3 || 6
2 || 2
1 || 3
0 || 2
||22
The list has 22 values. We can use this to find mean and median.
Mode. Mode is easy using a frequency table, we see the biggest f(x) is 6 and it corresponds to x = 3. That means the mode is 3, the number that shows up most often.
Mean. We need to know the sum of all 22 values. For this we need to multiply x*f(x). The idea is that there is one 7, so that adds 7 to the total, but there are three 5s, so that adds 15 to the total.
x || f(x) || x*f(x)
7 || 1 7
6 || 1 6
5 || 3 15
4 || 4 16
3 || 6 18
2 || 2 4
1 || 3 3
0 || 2 0
||22 || 69
The average or mean is 69/22 = 3.1363636... Since the data is all whole numbers, we round the average to the nearest tenth, which we would write as x-bar = 3.1
Median. To find the median, we need to find the median position, which is always (n + 1)/2. In this case, that is (22+1)/2 = 23/2 = 11.5. The median will be the average of the 11th and 12th numbers on the list. Here's how we find those numbers using the frequency table.
x || f(x) || positions
7 || 1 1
6 || 1 2
5 || 3 3 through 5
4 || 4 6 through 9
3 || 6 10 through 15
2 || 2 16 through 17
1 || 3 18 through 20
0 || 2 21 through 22
I went to the end of the list, but I could have stopped as soon as we knew that the value 3 was in positions 10 through 15. That means the 11th number is a 3 and so is the 12th number. The average of 3 and 3 is
(3+3)/2 = 3.
In this case, the median is 3.
Relative frequencies
We are now dealing with proportions, and the formulas are as follows:
Population: p = F/N
Sample: p-hat = f/n
We often want to compare one proportion to another, either two from the same sample or comparable proportions from different samples, or the sample proportion to the population proportion. Because of this, it is better to write the numbers as decimals or in scales based on powers of 10.
Scales based on powers of 10: The most famous scale base on powers of ten in percentage, which really means "per 100". It is much more common to see "53% of the people agree with the president's plan" than ".53 of the people..." or "53 out of every 100 people...". Technically, all those phrases are saying the same thing, but percentage is the most popular.
One of the places where decimals are used for proportions is in the sports pages. A batting average in baseball (hits/at bats) is given as a percent to three decimal place, and likewise winning proportions (win/total games) are written as .xxx. If a batter has 27 hits in 92 at-bats, the batting average 27/92 = .293478261... is shortened to .293 and pronounced "two ninety three". Likewise, a team who has won 17 games and lost 5 will have a winning proportion of 17/22 = .77272727... = .773, and often stated as "team has a winning percentage of seven seventy three." Technically, this is a mistake, because "percentage" means out or 100. The correct word from the dictionary, which no one ever uses, is "permillage", which means out of 1,000. The team in question would have a winning percentage of 77, and a winning permillage of 773.
In both of the cases from the sports pages, the greater number of place after the decimal is used to break ties. For example, a team with 14 wins and 4 losses is at .778, which is better than 17 wins and 5 losses, while 20 wins and 6 losses is at .769, so is slightly worse.
To get a number based on a power of 10 scale, you take the proportion and multiply by the power of ten, so it is either p*scale or p-hat*scale, depending on population or sample. Besides greater precision for breaking ties, sometimes we need greater precision because the proportions are so small.
When I ask a class what is the legal limit for blood alcohol while driving, invariably someone will say "point oh eight" and most people will agree. But .08 is wrong. .08 = 8%, and the correct answer is .08% = .0008. I don't blame the students. The number is badly represented and it is an easy mistake to make. Let's take a look at the number on other scales of 10.
.08 out of 100 is the same as
.8 out of 1,000 0r
8 out of 10,000 or
80 out of 100,000
80 parts out of 100,000 is a tiny proportion. To give an idea, ounce of pure alcohol mixed into ten gallons of blood would give you 78 parts out of 100,000, and most people have between a half gallon and a gallon and a half of blood in their body, between 4 and 12 pints. The amount of alcohol in a person's blood stream that is over the legal limit is about the same amount of alcohol as found in a capful of mouthwash used after brushing your teeth.
We will be dealing with much smaller proportions later in the class, where there are things that can be hazardous to your health at ranges measure in parts per billion, but for now, we will look at the per 100,000 scale for another type of statistic, measurements of mortality rates.
Here are the number of homicides in some American cities in 2013.
Chicago: 415
New York City: 333
Detroit: 333
Washington, DC: 103
Oakland: 92
This makes it look like Chicago is most dangerous and Oakland is safest, but this doesn't take into account the populations of the cities. Here is the list with populations rounded to the nearest ten thousand.
Chicago: 415 murders, pop. 2,720,000
New York City: 333 murders, pop. 19,650,000
Detroit: 333 murders, pop. 700,000
Washington, DC: 103 murders, pop. 650,000
Oakland: 92 murders, pop. 390,000
Because the number of murders is much smaller than the populations, if we just divide we will get a very small decimal number. For example in Chicago.
415/2720000 = 0.0001525735...
With death statistics, the scale of measurement is usually number per 100,000. To get that, we multiply the answer by the scale 100,000.
0.0001525735... * 100,000 = 15.25735...
Rounding to the nearest tenth, we would say the murder rate per 100,000 people in Chicago last year was 15.3. Here are all the numbers, rounded to the nearest tenth.
Chicago: 415 murders, pop. 2,720,000 15.3 per 100,000
New York City: 333 murders, pop. 19,650,000 1.7 per 100,000
Detroit: 333 murders, pop. 700,000 47.6 per 100,000
Washington, DC: 103 murders, pop. 650,000 15.8 per 100,000
Oakland: 92 murders, pop. 390,000 23.6 per 100,000
Looking at the number on this scale, it is clear that Detroit has a much higher murder rate than the other cities listed Oakland has the second highest murder rate. Chicago and DC are very similar and New York City's rate is very low.
Labels:
frequency tables,
relative frequency,
scales of 10
Friday, January 24, 2014
Notes for 23 January 2014
Five number summary, IQR and outlier thresholds
Consider the number of wins for each team in the National League at the end of the 2014 season. In order, the list looks like this. n = 15 and the five numbers are in bold.
97, 96, 94, 92, 90, 86, 81, 76, 76, 74, 74, 74, 73, 66, 62
The five number summary is as follows.
High: 97
Q3 : 92
Q2 : 76
Q1 : 74
Low: 62
Now we check to see if any of the numbers are outliers.
IQR = 92 - 74 = 18
Q3 + 1.5*IQR = 92 + 27 = 119 (no data higher than this, so no high outliers)
Q1 - 1.5*IQR = 74 - 27 = 47 (no data lower than this, so no low outliers)
Here is the data for the American League.
97, 96, 93, 92, 91, 86, 85, 85, 78, 74, 71, 66, 63, 51
The five number summary is as follows.
High: 97
Q3 : 92
Q2 : 85
Q1 : 71
Low: 51
Now we check to see if any of the numbers are outliers.
IQR = 92 - 71 = 21
Q3 + 1.5*IQR = 92 + 31.5 = 123.5 (no data higher than this, so no high outliers)
Q1 - 1.5*IQR = 71 - 31.5 = 39.5 (no data lower than this, so no low outliers)
The number that looks out of place is the 51, the number of wins for the Houston Astros, by far the worst team in the major leagues. But even though they won eleven less games than the next worst team, the data is so spread out that their very bad year doesn't count as a low outlier.
Stem and leaf format
Here are the numbers for both leagues in a stem and leaf format. Because these are all two digit numbers, the stem is the tens places and the leaves are the one places.
National League
9 | 02467
8 | 16
7 | 344467
6 | 26
American League
9 | 122367
8 | 556
7 | 148
6 | 36
5 | 1
The National league has one clump of good teams over 90 and a clump of slightly less than average teams between 73 and 77 wins. The American League had six teams with more than 90 wins and the rest of the league is split fairly evenly in the 80s, 70s and 60s, with just Houston with 59 wins or less.
Frequencies and relative frequencies
The frequency of a value is how many times it shows up on the list and is denoted by either an F is the set is a population or f if the set is a sample. We can also combine values as follows, looking at the data from the American League.
f(92) = 2
f(over 90) = 6
f(70 to 79) = 3
Frequencies are always whole numbers, either zero or positive integers.
Relative frequencies are numbers between 0 and 1 and are sometimes called proportions or probablilites. Sometimes the word percentages is used, but that should only be used if the number is represented with a percent sign. In a population, we use the lowercase letter p and in a sample, the symbol is called p-hat. Let's take the relative frequencies for the f statistics above, writing them as fractions, decimals and percents.
p-hat(92) = 2/15 = .13333... or approximately 13.3%
p-hat(over 90) = 6/15 = .4 = 40%
p-hat(70 to 79) = 3/15 = .2 = 20%
Practice for five number summary.
Practice for frequency and relative frequency
Here are the National League wins again. Find the following frequencies and relative frequencies, writing the relative frequencies as fractions, decimals and percentages. Round the decimals to the nearest thousandth and the percentages to the nearest tenth of a percent. For example, 2/15 would be .133 to the nearest thousandth and 13.3% to the nearest tenth of a percent.
97, 96, 94, 92, 90, 86, 81, 76, 76, 74, 74, 74, 73, 66, 62
f(74) = _________
p-hat(74) = _________
f(between 70 and 79) = _________
p-hat(between 70 and 79) = _________
f(over 90) = _________
p-hat(over 90) = _________
Answers to the frequency and relative frequency question in the comments.
Consider the number of wins for each team in the National League at the end of the 2014 season. In order, the list looks like this. n = 15 and the five numbers are in bold.
97, 96, 94, 92, 90, 86, 81, 76, 76, 74, 74, 74, 73, 66, 62
The five number summary is as follows.
High: 97
Q3 : 92
Q2 : 76
Q1 : 74
Low: 62
Now we check to see if any of the numbers are outliers.
IQR = 92 - 74 = 18
Q3 + 1.5*IQR = 92 + 27 = 119 (no data higher than this, so no high outliers)
Q1 - 1.5*IQR = 74 - 27 = 47 (no data lower than this, so no low outliers)
Here is the data for the American League.
97, 96, 93, 92, 91, 86, 85, 85, 78, 74, 71, 66, 63, 51
The five number summary is as follows.
High: 97
Q3 : 92
Q2 : 85
Q1 : 71
Low: 51
Now we check to see if any of the numbers are outliers.
IQR = 92 - 71 = 21
Q3 + 1.5*IQR = 92 + 31.5 = 123.5 (no data higher than this, so no high outliers)
Q1 - 1.5*IQR = 71 - 31.5 = 39.5 (no data lower than this, so no low outliers)
The number that looks out of place is the 51, the number of wins for the Houston Astros, by far the worst team in the major leagues. But even though they won eleven less games than the next worst team, the data is so spread out that their very bad year doesn't count as a low outlier.
Stem and leaf format
Here are the numbers for both leagues in a stem and leaf format. Because these are all two digit numbers, the stem is the tens places and the leaves are the one places.
National League
9 | 02467
8 | 16
7 | 344467
6 | 26
American League
9 | 122367
8 | 556
7 | 148
6 | 36
5 | 1
The National league has one clump of good teams over 90 and a clump of slightly less than average teams between 73 and 77 wins. The American League had six teams with more than 90 wins and the rest of the league is split fairly evenly in the 80s, 70s and 60s, with just Houston with 59 wins or less.
Frequencies and relative frequencies
The frequency of a value is how many times it shows up on the list and is denoted by either an F is the set is a population or f if the set is a sample. We can also combine values as follows, looking at the data from the American League.
f(92) = 2
f(over 90) = 6
f(70 to 79) = 3
Frequencies are always whole numbers, either zero or positive integers.
Relative frequencies are numbers between 0 and 1 and are sometimes called proportions or probablilites. Sometimes the word percentages is used, but that should only be used if the number is represented with a percent sign. In a population, we use the lowercase letter p and in a sample, the symbol is called p-hat. Let's take the relative frequencies for the f statistics above, writing them as fractions, decimals and percents.
p-hat(92) = 2/15 = .13333... or approximately 13.3%
p-hat(over 90) = 6/15 = .4 = 40%
p-hat(70 to 79) = 3/15 = .2 = 20%
Practice for five number summary.
Practice for frequency and relative frequency
Here are the National League wins again. Find the following frequencies and relative frequencies, writing the relative frequencies as fractions, decimals and percentages. Round the decimals to the nearest thousandth and the percentages to the nearest tenth of a percent. For example, 2/15 would be .133 to the nearest thousandth and 13.3% to the nearest tenth of a percent.
97, 96, 94, 92, 90, 86, 81, 76, 76, 74, 74, 74, 73, 66, 62
f(74) = _________
p-hat(74) = _________
f(between 70 and 79) = _________
p-hat(between 70 and 79) = _________
f(over 90) = _________
p-hat(over 90) = _________
Answers to the frequency and relative frequency question in the comments.
Monday, June 29, 2009
Representing the age at death demographics

The graphical representation of three different groups and their relative frequencies in multiple demographic categories makes the most sense in a bar chart with multiple colors. Red represents the whole population, green represents the Caucasian sub-population and blue represents the African American sub-population.
Click on the picture to see a larger version.
Wednesday, February 4, 2009
Class notes for 2/4
More on the five number summary and outlying data. Consider the number of wins for each team in the NFC at the end of the 2008 season. In order, the list looks like this.
12, 12, 11, 10, 9, 9, 9, 9, 9, 8, 8, 7, 6, 4, 2, 0
The five number summary is as follows.
High: 12
Q3 : 9.5
Q2 : 9
Q1 : 6.5
Low: 0
Now we check to see if any of the numbers are outliers.
IQR = 9.5 - 6.5 = 3
Q3 + 1.5*IQR = 9.5 + 4.5 = 14 (no data higher than this, so no high outliers)
Q1 - 1.5*IQR = 6.5 - 4.5 = 2 (the value 0 is a low outlier, the value 2 is just inside the threshold)
The Detroit Lions' total of no wins in 16 games was the low outlier in the NFC.
Here is the data for the AFC.
13, 12, 12, 11, 11, 11, 9, 8, 8, 8, 7, 5, 5, 4, 4, 2
The five number summary is as follows.
High: 13
Q3 : 11
Q2 : 8
Q1 : 5
Low: 2
Now we check to see if any of the numbers are outliers.
IQR = 11 - 5 = 6
Q3 + 1.5*IQR = 11 + 6 = 17 (no data higher than this, so no high outliers)
Q1 - 1.5*IQR = 5 - 6 = -1 (no data lower than this, so no low outliers)
Here are the two conferences' box and whisker plots shown above and below a scale that goes from 0 to 13.


Parameters and statistics for categorical data. We have already defined some numbers that are associated with data sets, which we call parameters if the data set is a population and statistics if the data set is a sample. Statistics has the idea of reserved symbols, like N and n for size of the data set, population and sample respectively, and mux and x-bar for the mean. With categorical data, the number of units that share a value for a given categorical variable is called the frequency of the value, denoted by F(value) in a population of f(value) in a sample. Two data sets with the same variable can be compared, but if the size of the sets is significantly different, it is fairer to compare the relative frequency, which if the frequency divided by the size of the data set, denoted by p in a population and p-hat in a sample.
For example, let's look at Data Set #2 and the variable of Major. Here are the frequencies of each value that has at least one subject.
f(AH) = 6
f(BF) = 13
f(CESM) = 2
f(HE) = 4
f(SS) = 4
f(Other) = 11
f(Und) = 2
To get the relative frequencies, we divide the frequencies by n, which in this case is 42. This gives us the p-hat values, which we will usually express as a percent. In the following equations, the symbol ~= means "approximately equal to".
p-hat(AH) = 6/42 ~= 14%
p-hat(BF) = 13/42 ~= 31%
p-hat(CESM) = 2/42 ~= 5%
p-hat(HE) = 4/42 ~= 10%
p-hat(SS) = 4/42 ~= 10%
p-hat(Other) = 11/42 ~= 26%
p-hat(Und) = 2/42 ~= 5%
The sum of all of relative frequency should give us 100%, but sometimes we get rounding error. In this case, the sum is 101%. When this happens, we round to the nearest tenth of a percent, hoping that the total will be either 99.9%, 100% or 100.1%. This is called the +/- .1% rule for rounding relative frequency.
p-hat(AH) = 6/42 ~= 14.3%
p-hat(BF) = 13/42 ~= 31.0%
p-hat(CESM) = 2/42 ~= 4.8%
p-hat(HE) = 4/42 ~= 9.5%
p-hat(SS) = 4/42 ~= 9.5%
p-hat(Other) = 11/42 ~= 26.2%
p-hat(Und) = 2/42 ~= 4.8%
The sum of the more closely rounded numbers is now 100.1%, which is close enough. With some sets of fractions that add up to 1, which in general are called probability distributions, it doesn't matter how far you go when rounding, you will never get exactly 100%, which is why we decide on this "close enough" method. For instance 1/3 + 1/3 + 1/3 = 1, but if we round 1/3 and then add them up, we get 99% or 99.9% or 99.99%, etc. depending on how far we round the numbers. Because these kinds of cases can show up, we decide on a fixed number we consider close enough, known in math as an epsilon value, and round all the numbers to the same number of digits after the decimal until the total is close enough.
Example of a worst case scenario: If we have 16 values for a variable and all the values show up once, every number in our probability distribution is 1/16 = .0625 = 6.25%
Rounded to nearest percent: 1/16 ~= 6%. 16*6% = 96%, way too low
Rounded to nearest tenth of a percent: 1/16 ~= 6.3%. 16*6.3% = 100.8%, too high
Rounded to nearest hundredth of a percent: 1/16 = 6.25%. Now the total is exactly 100%, since there is no rounding at all.
12, 12, 11, 10, 9, 9, 9, 9, 9, 8, 8, 7, 6, 4, 2, 0
The five number summary is as follows.
High: 12
Q3 : 9.5
Q2 : 9
Q1 : 6.5
Low: 0
Now we check to see if any of the numbers are outliers.
IQR = 9.5 - 6.5 = 3
Q3 + 1.5*IQR = 9.5 + 4.5 = 14 (no data higher than this, so no high outliers)
Q1 - 1.5*IQR = 6.5 - 4.5 = 2 (the value 0 is a low outlier, the value 2 is just inside the threshold)
The Detroit Lions' total of no wins in 16 games was the low outlier in the NFC.
Here is the data for the AFC.
13, 12, 12, 11, 11, 11, 9, 8, 8, 8, 7, 5, 5, 4, 4, 2
The five number summary is as follows.
High: 13
Q3 : 11
Q2 : 8
Q1 : 5
Low: 2
Now we check to see if any of the numbers are outliers.
IQR = 11 - 5 = 6
Q3 + 1.5*IQR = 11 + 6 = 17 (no data higher than this, so no high outliers)
Q1 - 1.5*IQR = 5 - 6 = -1 (no data lower than this, so no low outliers)
Here are the two conferences' box and whisker plots shown above and below a scale that goes from 0 to 13.


Parameters and statistics for categorical data. We have already defined some numbers that are associated with data sets, which we call parameters if the data set is a population and statistics if the data set is a sample. Statistics has the idea of reserved symbols, like N and n for size of the data set, population and sample respectively, and mux and x-bar for the mean. With categorical data, the number of units that share a value for a given categorical variable is called the frequency of the value, denoted by F(value) in a population of f(value) in a sample. Two data sets with the same variable can be compared, but if the size of the sets is significantly different, it is fairer to compare the relative frequency, which if the frequency divided by the size of the data set, denoted by p in a population and p-hat in a sample.
For example, let's look at Data Set #2 and the variable of Major. Here are the frequencies of each value that has at least one subject.
f(AH) = 6
f(BF) = 13
f(CESM) = 2
f(HE) = 4
f(SS) = 4
f(Other) = 11
f(Und) = 2
To get the relative frequencies, we divide the frequencies by n, which in this case is 42. This gives us the p-hat values, which we will usually express as a percent. In the following equations, the symbol ~= means "approximately equal to".
p-hat(AH) = 6/42 ~= 14%
p-hat(BF) = 13/42 ~= 31%
p-hat(CESM) = 2/42 ~= 5%
p-hat(HE) = 4/42 ~= 10%
p-hat(SS) = 4/42 ~= 10%
p-hat(Other) = 11/42 ~= 26%
p-hat(Und) = 2/42 ~= 5%
The sum of all of relative frequency should give us 100%, but sometimes we get rounding error. In this case, the sum is 101%. When this happens, we round to the nearest tenth of a percent, hoping that the total will be either 99.9%, 100% or 100.1%. This is called the +/- .1% rule for rounding relative frequency.
p-hat(AH) = 6/42 ~= 14.3%
p-hat(BF) = 13/42 ~= 31.0%
p-hat(CESM) = 2/42 ~= 4.8%
p-hat(HE) = 4/42 ~= 9.5%
p-hat(SS) = 4/42 ~= 9.5%
p-hat(Other) = 11/42 ~= 26.2%
p-hat(Und) = 2/42 ~= 4.8%
The sum of the more closely rounded numbers is now 100.1%, which is close enough. With some sets of fractions that add up to 1, which in general are called probability distributions, it doesn't matter how far you go when rounding, you will never get exactly 100%, which is why we decide on this "close enough" method. For instance 1/3 + 1/3 + 1/3 = 1, but if we round 1/3 and then add them up, we get 99% or 99.9% or 99.99%, etc. depending on how far we round the numbers. Because these kinds of cases can show up, we decide on a fixed number we consider close enough, known in math as an epsilon value, and round all the numbers to the same number of digits after the decimal until the total is close enough.
Example of a worst case scenario: If we have 16 values for a variable and all the values show up once, every number in our probability distribution is 1/16 = .0625 = 6.25%
Rounded to nearest percent: 1/16 ~= 6%. 16*6% = 96%, way too low
Rounded to nearest tenth of a percent: 1/16 ~= 6.3%. 16*6.3% = 100.8%, too high
Rounded to nearest hundredth of a percent: 1/16 = 6.25%. Now the total is exactly 100%, since there is no rounding at all.
Subscribe to:
Posts (Atom)

