Notes for 23 January 2014

Five number summary, IQR and outlier thresholds

Consider the number of wins for each team in the National League at the end of the 2014 season. In order, the list looks like this. n = 15 and the five numbers are in bold.

97, 96, 94, 92, 90, 86, 81, 76, 76, 74, 74, 74, 73, 66, 62

The five number summary is as follows.

High: 97
Q3 : 92
Q2 : 76
Q1 : 74
Low: 62

Now we check to see if any of the numbers are outliers.

IQR = 92 - 74 = 18
Q3 + 1.5*IQR = 92 + 27 = 119 (no data higher than this, so no high outliers)
Q1 - 1.5*IQR = 74 - 27 = 47 (no data lower than this, so no low outliers)

Here is the data for the American League.

 97, 96, 93, 92, 91, 86, 85, 85, 78, 74, 71, 66, 63, 51

The five number summary is as follows.

High: 97
Q3 : 92
Q2 : 85
Q1 : 71
Low: 51

Now we check to see if any of the numbers are outliers.

IQR = 92 - 71 = 21
Q3 + 1.5*IQR = 92 + 31.5 = 123.5 (no data higher than this, so no high outliers)
Q1 - 1.5*IQR = 71 - 31.5 = 39.5 (no data lower than this, so no low outliers)


The number that looks out of place is the 51, the number of wins for the Houston Astros, by far the worst team in the major leagues. But even though they won eleven less games than the next worst team, the data is so spread out that their very bad year doesn't count as a low outlier.

Stem and leaf format

Here are the numbers for both leagues in a stem and leaf format. Because these are all two digit numbers, the stem is the tens places and the leaves are the one places.

National League
9 | 02467
8 | 16
7 | 344467
6 | 26

American League
9 | 122367
8 | 556
7 | 148
6 | 36
5 | 1

The National league has one clump of good teams over 90 and a clump of slightly less than average teams between 73 and 77 wins. The American League had six teams with more than 90 wins and the rest of the league is split fairly evenly in the 80s, 70s and 60s, with just Houston with 59 wins or less.

Frequencies and relative frequencies

 The frequency of a value is how many times it shows up on the list and is denoted by either an F is the set is a population or f if the set is a sample. We can also combine values as follows, looking at the data from the American League.

f(92) = 2
f(over 90) = 6
f(70 to 79) = 3

Frequencies are always whole numbers, either zero or positive integers.

Relative frequencies are numbers between 0 and 1 and are sometimes called proportions or probablilites. Sometimes the word percentages is used, but that should only be used if the number is represented with a percent sign.  In a population, we use the lowercase letter p and in a sample, the symbol is called p-hat. Let's take the relative frequencies for the f statistics above, writing them as fractions, decimals and percents.

p-hat(92) = 2/15 = .13333... or approximately 13.3%
p-hat(over 90) = 6/15 = .4 = 40%
p-hat(70 to 79) = 3/15 = .2 = 20%


Practice for five number summary.

Practice for frequency and relative frequency
 Here are the National League wins again. Find the following frequencies and relative frequencies, writing the relative frequencies as fractions, decimals and percentages. Round the decimals to the nearest thousandth and the percentages to the nearest tenth of a percent. For example, 2/15 would be .133 to the nearest thousandth and 13.3% to the nearest tenth of a percent.


97, 96, 94, 92, 90, 86, 81, 76, 76, 74, 74, 74, 73, 66, 62

f(74) = _________

p-hat(74) = _________

f(between 70 and 79) = _________
p-hat(between 70 and 79) = _________

f(over 90) = _________
p-hat(over 90) = _________

Answers to the frequency and relative frequency question in the comments.

frequency table to dot plot and to stem and leaf plot

Here is our frequency table for heights in inches.

_x____f(x)
60_____3
61_____1
62_____2
63_____2
64_____1
65_____2
66_____8
68_____2
69_____3
70_____2
71_____3
72_____2
74_____1
76_____1
77_____1
78_____1

Here is a dot plot using this information.
* * * * * * *
* * * * * * *
* * * * * * *
* * * * * * *
* * * * * * *
* * * * * * * * * * * *
* * * * * * * * * * * * *
* * * * * * * * * * * * * * * * * * *
_____________________________________
6 6 6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8

Here is a stem and leaf, with just two stems, 6 and 7.
6 | 000122334556666666688999
7 | 00111224678

We can make a stem and leaf where each of the stems is split into high and low, 60 to 64, 65 to 69, 70 to 74 and 75 to 79.
6 | 000122334
6 | 556666666688999
7 | 00111224
7 | 678

The idea is that the stem and leaf gives the shape of the data. It's okay to have the stem values go from low to high or high to low, as long as the values are consistent.

7 | 00111224678
6 | 000122334556666666688999

Or if we split the decades, we get this.

7 | 678
7 | 00111224
6 | 556666666688999
6 | 000122334

Class notes for 1/28

More on frequency tables: When we have a lot of duplicate values in a data set, a frequency table lets us write that data more compactly, by listing each value x, followed by the positive number of times it shows up, f(x). Let's look at the data list from the 1/26 notes, which were the scores in one of the two classes this semester on the first quiz.

x || f(x)

20 || 2
19 || 1
18 || 2
17 || 5
16 || 7
15 || 2
14 || 5
13 || 5
12 || 3
11 || 5
_9 || 2
_8 || 1
_7 || 2
_4 || 3


We already discussed how to find the mode and median, but what about the mean? We need to find n, and we need to find the sum of all the x values. To get those numbers, n = sum(f(x)) and sum(x) = sum(x*f(x)). Here is the list again with another column of numbers added, the value multiplied by its frequency.

x || f(x) || x*f(x)

20 || 2 || 40
19 || 1 || 19
18 || 2 || 36
17 || 5 || 85
16 || 7 ||112
15 || 2 || 30
14 || 5 || 70
13 || 5 || 65
12 || 3 || 36
11 || 5 || 55
_9 || 2 || 18
_8 || 1 || _8
_7 || 2 || 14
_4 || 3 || 12

The sum of the second column is 45, the sum of the third column is 600, so x-bar=600/45=13.333..., or rounded to the nearest tenth, 13.3.


The ideas of John Tukey. John Tukey was a statistician and computer scientist who did a lot of his best work back in the 1960s. Many of his ideas in statistics are about easier and shorter ways to present data sets. We will study three of them, the stem and leaf plot, the five number summary and its graphical representation, the box and whiskers plot.

Stem and leaf plots. If we don't have a lot of duplication, a frequency table is not going to make presenting data any shorter. What Tukey did was split numbers into two parts, the stem and the leaf. The standard way to do this is to make the last digit the leaf and the rest of the number the stem. For example, the number 87 would have stem = 8 and leaf = 7. Tukey's idea was to put the stem numbers at the left and list the leaves from low to high, using a mono-spaced font, which back in the day of typewriters was the only choice available for someone typing. (Courier is a mono-spaced font. The letter i is the same width as the letter w or any other symbol. Most fonts used today aren't mono-spaced anymore, but in this situation, we want to use Courier.)

Here is a list of numbers. It is the number of wins by teams in the American League at the end of the 2008 regular season.

97, 95, 89, 86, 68, 89, 88, 81, 75, 74, 100, 79, 75, 61

The highest number is 100, so we make the stem 10 and the leaf 0. All the numbers on the list are put in stem and leaf form, where the stem is underlined and followed by a |, then the leaves are listed from low to high from left to right.

____
10 | 0
_9 | 57
_8 | 16899
_7 | 4559
_6 | 18

Sometimes data is too spread out to use the last digit as the leaf, and instead the last two digits are the leaf, and we put a space between the two digit leaves for readability. Here is an example. This data set is the number of points scored by the teams in the AFC in the 2008 regular season.

Data set: 347, 448, 388, 440, 234, 298, 394, 367, 223, 244, 364, 350, 342, 356, 309, 317

44 | 08
43 |
42 |
41 |
40 |
39 | 4
38 | 8
37 |
36 | 47
35 | 06
34 | 27
33 |
32 |
31 | 7
30 | 9
29 | 8
28 |
27 |
26 |
25 |
24 | 4
23 | 4
22 | 3

If each stem is a span of ten values, this data is too spread out, but if the stems go from 200 to 299, 300 to 399 and 400-499, the stem and leaf method works very well. When the leaves are two digits, we put a space between them for easier reading.

___
4 | 40 48
3 | 09 17 42 47 50 56 64 67 88 94
2 | 23 34 44 98

Now the data is in a more compact list, and the big idea of stem and leaf, which is not only to give the values but to give the shape of the data, becomes clear. There are only a few values in the 400-499 range, most are between 300 and 399, with some low values in the 200s.

It's possible to have the opposite problem where a stem has too many leaves. Here is the stem and leaf version of the data set we represented with a frequency table earlier.

___
2 | 00
1 | 11111222333334444455666666677777889
0 | 44477899

Obviously, the vast majority of the data is between 10 and 19. It makes sense to make more stems by splitting each category in half, going from 0 to 4, 5 t0 9, 10 to 14, 15 to 19 and 20 to 24. Since there is no data above 25, we don't have to create an empty stem from 25 to 29.


___
2 | 00
1 | 55666666677777889
1 | 111112223333344444
0 | 77899
0 | 444

This shows us that the most common values were between 10 and 14, with 15 to 19 almost as common, with the frequency trailing off as values get above 19 or under 10.


Five number summary. Both frequency tables and stem and leaf plots tell us about all the data. The five number summary, yet another idea of John Tukey, gives us an idea of how the data is distributed, but leaves a lot of information out. The five numbers are the highest value, the lowest value, and three intermediate numbers, Q1, Q2 and Q3. Q stands for quartile, so these numbers are respectively the cut-off points for the bottom 25% of the data, the bottom 50% of the data and the bottom 75% of the data. We already know Q2 by another name, the median. Once we remove the median from a set of data, we then have two new subsets, the bottom half and the top half. Q1 is the median of the bottom half, while Q3 is the median of the top. Let's do an example with a set of data we have already dealt with, the points scored by AFC teams. First, here is the list of 16 numbers put in order from high to low. The middle two values are marked in bold type.

448, 440, 394, 388, 367, 364, 356, 350, 347, 342, 317, 309, 298, 244, 234 223

The median, or Q2 is (350+347)/2 = 348.5.

Top half: 448, 440, 394, 388, 367, 364, 356, 350

The median of the top half, or Q3 is (388+367)/2 = 377.5.

Bottom half: 347, 342, 317, 309, 298, 244, 234 223

The median of the bottom half, or Q1 is (309+298)/2 = 303.5.

In order, the five number summary is:

High: 448
Q3: 377.5
Q2: 348.5
Q1: 303.5
Low: 223

Box and whisker plots: The box and whisker plot is a graphical representation of the five number summary. Each of the five points is plotted against a number line for scale. The area between and is the box, and it represents the middle 50% of the data. The whiskers are lines from the edge of the box to the high and low values. is shown by a dotted line somewher inside the box, though not exactly in the middle, necessarily.

Here is a representation of five number summary above as a box and whisker plot. It could either be on a vertical scale, as pictured here, or it could be represented on a horizontal scale. When financial data from a set of time periods is summarized, the data is often represented with box and whiskers plots, though some websites call them candlesticks instead. The vertical scale represents the value in currency (usually dollars), while the horizontal scale represents the change in time, where the time periods can be any standardized amount, whether day to day or week to week or month to month.