Transcript Anatomy of the histogram

Those who don’t know
statistics are condemned to
reinvent it…
David Freedman
All you ever wanted to know
about the histogram and
more ...
Distribution of No of Graphics
on web pages (N=1873)
400
Mean = 17.93
Median = 16.00
300
Std. Dev = 17.92
N = 1873
200
100
Graphic Count
0
0.0
10.0
5.0
20.0
15.0
30.0
25.0
40.0
35.0
50.0
45.0
60.0
55.0
70.0
65.0
80.0
75.0
90.0
85.0
95.0
Horizontal Scale
Distribution of Redundant Link
% on web pages (N =1861)
1000
Mean = 22.1
Median = 14
800
600
Std. Dev = 37.33
N = 1861.00
400
200
0
Plotting a histogram:
endpoint convention,
plot frequencies,
make equal intervals etc.
Frequency Table
FrequencyInterval MidPoint
110
430
860
280
180
40
20
10
0-2
2-4
4-6
6-8
8-10
10-12
12-14
14-16
1
3
5
7
9
11
13
15
convention: include the left endpoint in the class interval
Frequency/Probability
Frequency Probability Interval
MidPoint
110
0.06
0-2
1
430
0.22
2-4
3
860
0.45
4-6
5
280
0.15
6-8
7
180
0.09
8-10
9
40
0.02
10-12
11
20
0.01
12-14
13
10
0.01
14-16
15
No of fonts used on a web-page
1000/ .5
Frequency
/probability
800/ .4
600/ .3
400/ .2
200/ .1
0/ 0
1
3
5
7
9
11
13
15
Frequency
110
430
860
280
180
40
20
10
Probability
.06
.22
.45
.15
.09
.02
.01
.01
Cleaning up a histogram:
getting rid of outliers
Distribution of word count
(N=1903)
1600
1400
Mean = 393.2
Median = 223
1200
Std. Dev = 725.24
1000
Minimum = 0
800
Maximum = 20,357
600
400
200
0
Distribution of word count
(N=1897) top six removed
800
Mean = 368.0
Median = 223
600
Std. Dev = 474.04
Minimum = 0
400
Maximum = 4132
200
0
Distribution of word count
(N=1873)
500
Mean = 333.4
Median = 220
400
Std. Dev = 360.30
300
Minimum = 0
Maximum = 4132
200
100
0
WORDCNT2
What can histograms tell
you
Distribution of link count on
good & bad web-pages
3 0 0
2 0 0
1 0 0
0
0 .0
4 0 .0
8 0 .0
Good Sites
1 2 0 .0
1 6 0 .0
2 0 0 .0
2 4 0 .0
2 8 0 .0
Bad Sites
Making inferences from histograms:
Incidence of riots and temperature
30
40
50
60
70
80
temperature
90
100
110
Mean and Median
Mean is arithmetic average, median is 50% point
Mean is point where graph balances
Mean shifts around,
Median does not shift much, is more stable
Computing Median:
for odd numbered N
find middle number
For even numbered N
interpolate between middle 2,
e.g. if it is 7 and 9, then 8 is the median
The instability of means and standard
deviations
11.00
9.00
10.00
4.00
5.00
7.00
8.00
11.00
20.00
Mean
Median
SD
9.44
9.00
4.67
Add two numbers: watch the mean, median, & SD
Mean
Median
SD
11.00
9.00
12.00
4.00
5.00
7.00
8.00
11.00
20.00
11.00
9.00
12.00
4.00
5.00
7.00
8.00
11.00
40.00
9.00
7.00
9.67
9.00
4.74
11.60
9.00
10.31
Add one outlier...
Mean
Median
SD
11.00
9.00
12.00
4.00
5.00
7.00
8.00
11.00
20.00
11.00
9.00
12.00
4.00
5.00
7.00
8.00
11.00
40.00
9.00
7.00
11.00
9.00
12.00
4.00
55.00
7.00
8.00
11.00
40.00
9.67
9.00
4.74
11.60
9.00
10.31
17.44
11.00
17.61
Standard Deviation: a measure of
spread
Same mean, different spread
SD
SD
The Standard Deviation
Mean
SD
Minimum
Maximum
12.00
12.00
10.00
9.00
7.00
6.00
5.00
18.00
15.00
11.00
7.00
7.00
2.00
1.00
8.71
2.81
5.00
12.00
8.71
6.34
1.00
18.00
The SD says how far away numbers
on a list are from their average.
Most entries on the list will be
somewhere around one SD away
from the average. Very few will be
more than two or three SD’s away.
Understanding the standard
deviation
Lets start with a list: 1, 2, 2, 3
50%
25%
0%
Histogram is symmetric about 2,
2 is mean,
and 50% to left of 2, 50% to right
50%
25%
0%
50%
25%
0%
List: 1, 2, 2, 3
Average = 2
SD = .8
List: 1, 2, 2, 5
Average =2.5
SD = 1.73
50%
25%
0%
List: 1, 2, 2, 7
Average =3
SD = 2.71
Computing the standard deviation
List: 20, 10, 15, 15
Average = 15
Find deviations from average=
5, -5, 0, 0
Square the deviations:
(5)2 (-5)2 (0)2 (0)2 = 50
divide it by N-1 = 50/3 = 16.67
Square root it= 16.67 = 4.08
Properties of the standard
deviation
• The standard deviation is in the
same units as the mean
• The standard deviation is inversely
related to sample size (therefore
as a measure of spread it is
biased)
• In normally distributed data 68%
of the sample lies within 1 SD
Properties of the Normal
Probability Curve
• The graph is symmetric about the mean
(the part to the right is a mirror image of
the part to the left)
• The total area under the curve equals
100%
• Curve is always above horizontal axis
• Appears to stop after a certain point (the
curve gets really low)
1 SD= 68%
2 SD = 95%
3 SD= 99.7%
• The graph is symmetric about the mean =
• The total area under the curve equals
100%
• Mean to 1 SD = +- 68%
• Mean to 2 SD = +- 95%
• Mean to 3 SD = +- 99.7%
• You can disregard rest of curve
Distribution of judges ratings for
the Webby Awards
500
400
Mean = 6.3
Median = 6.3
300
Std. Dev = 1.98
200
N = 1867.00
100
Skewness = -.43
Kurtosis = -.201
0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0
10.0
It is a remarkable fact that many histograms in
real life tend to follow the Normal Curve.
For such histograms, the mean and SD are
good summary statistics.
The average pins down the center, while the SD
gives the spread.
For histogram which do not follow the normal
Curve, the mean and SD are not good
summary statistics.
What when the histogram is not normal ...
Distribution of word count on web pages
500
400
300
Std. Dev = 384.83
Mean = 348.3
200
100
0
+- 3 SD = (384 * 3) = 1152
Mean - 1152 = about 30% sample had negative number of
links
When SD is influenced by outliers
Use inter quartile range
75th percentile - 25th percentile
Note.
A percentile is a score below which a certain % of sample is
Measures of Normality
• Visual examination
• Skewness: measure of symmetry
Positively Skewed
Negatively Skewed
Symmetric
Kurtosis: Does it cluster in the
middle?
Kurtosis is based on a distributions tail.
Distributions with a large tail: leptokurtic
Distributions with a small tail: platykurtic
Distributions with a normal tail: mesokurtic
Large tail
Small tail
Normal Tail
Positively Skewed and
Leptokurtic: Word Count
1600
1400
1200
1000
800
Mean = 393.2
Median = 223
Std. Dev = 725.24
Skewness = 13.62
Kurtosis = 321.84
600
N = 1903.00
400
200
0
Distribution of word count
(N=1897) top six removed
800
Kurtosis = 16.40
Skewness = 3.49
600
400
Mean = 368.0
Median = 223
Std. Dev = 474.04
N = 1897.00
200
0
Degree of Freedom
• The number of independent pieces of
information remaining after estimating one or
more parameters
• Example: List= 1, 2, 3, 4
Average= 2.5
• For average to remain the same three of the
numbers can be anything you want, fourth is
fixed
• New List = 1, 5, 2.5, __ Average = 2.5