Transcript EECS 252 Graduate Computer Architecture Lec

ECE562/468 Advanced Computer Architecture
Prof. Honggang Wang
Ch1. Fundamentals of Computer Design
2. Performance
ECE Department
University of Massachusetts
Dartmouth
285 Old Westport Rd.
North Dartmouth, MA 02747-2300
Slides based on the PowerPoint Presentations created by David Patterson as
part of the Instructor Resources for the textbook by Hennessy & Patterson
Updated by Honggang Wang.
Administrative Issues (02/02/2016)
• Project team set-up due Thursday, Feb. 11
– Each group has 3-4 group members
– Two types of groups: ECE 468 and ECE 562
• If you registered ECE 468, you only can join ECE 468 group
• If you registered ECE 562, you only can join ECE 562 group
– Your group leader must send me an email about your
group information by 02/11
• If you missed the first week class, go to the
course website for syllabus and 1st lecture.
• My office hours:
– T./TH. 11 am-12:45 pm, Fri. 1:00-1:30 pm
www.faculty.umassd.edu/honggang.wang/teaching.html
2
Review of Lecture #1
In the first lecture, we covered the
• Technology Trends
1.Define and quantify power
2.Define and quantify relative cost
3.Define and quantify dependability
3
Outline
• Careful, quantitative comparisons: Performance
1. Compute Mean
2. Benchmark
4
Definition: Performance
• Performance is in events per sec:
– bigger is better
• If we are primarily concerned with response time:
• the time between the start and the completion of an
event (execution time).
1
performance( x) 
execution _ time( x)
Performance( X ) Execution _ time(Y )
n

Performance(Y ) Execution _ time( X )
• " X is n times faster than Y" means:
5
Performance: What to measure
•
To increase predictability, collections of benchmark applications, called
benchmark suites, are popular
•
•
•
•
SPECCPU: popular desktop benchmark suite
–
–
–
–
•
reliable measure of the performance is the execution time of real program
the users would simply compare the execution time of their workloads
a popular measure of performance of processors with a variety of applications
CPU only, split between integer and floating point programs
SPECint2000 has 12 integer, SPECfp2000 has 14 integer pgms
SPECCPU2006 to be announced Spring 2006
SPECSFS (NFS file server) and SPECWeb (WebServer) added as server
benchmarks
Transaction Processing Council measures server performance and costperformance for databases (www.tpc.org)
–
–
–
–
TPC-C Complex query for Online Transaction Processing
TPC-H models ad hoc decision support
TPC-W a transactional web benchmark
TPC-App application server and web services benchmark
6
How Summarize Suite Performance (1/5)
• Arithmetic average of execution time of all pgms?
– But they vary by 4X in speed, so some would be more important
than others in arithmetic average
• Could add a weights per program, but how pick up
weight?
– Different companies want different weights for their products
• SPECRatio: Normalize execution times to reference
computer, yielding a ratio proportional to performance
Time on refrence computer
Performanc e 
Time on computer being rated
7
How Summarize Suite Performance (2/5)
• If program SPECRatio on Computer A is
1.25 times bigger than Computer B, then
ExecutionTimereference
SPECRatio A
ExecutionTime A
1.25 

SPECRatioB ExecutionTimereference
ExecutionTimeB
ExecutionTimeB PerformanceA


ExecutionTime A PerformanceB
• Note that when comparing 2 computers as a ratio,
execution times on the reference computer drop
out, so choice of reference computer is irrelevant
8
How Summarize Suite Performance (3/5)
• Since ratios, proper mean is geometric mean
(SPECRatio unitless, so arithmetic mean meaningless)
GeometricMean  n
n
 SPECRatio
i
i 1
1. The SPECRatio are multiplied and then the nth root
(where n is the count of SPECRatio in the set) of the
resulting product is taken
2. indicates the central tendency or typical value of a
set of numbers
9
How Summarize Suite Performance (3/5)
• Suppose an orange tree yields 100 •
oranges one year and then 180, 210
and 300 the following years, so the •
growth is 80%, 16.7% and 42.9% for
each year respectively.
• Using the arithmetic mean
calculates a (linear) average growth
of 46.5% (80% + 16.7% + 42.9%
•
divided by 3).
• However, if we start with 100
oranges and let it grow 46.5% each
year, the result is 314 oranges, not
300, so the linear average overstates the year-on-year growth.
Instead, we can use the geometric
mean.
Growing with 80% corresponds to
multiplying with 1.80, so we take the
geometric mean of 1.80, 1.167 and
1.429, i.e. , thus the "average" growth
per year is 44.3%.
If we start with 100 oranges and let
the number grow with 44.3% each
year, the result is 300 oranges.
10
How Summarize Suite Performance (4/5)
• Does a single mean well summarize performance of programs in
benchmark suite?
• Can decide if mean is a good predictor by characterizing
variability of distribution using standard deviation
• Like geometric mean, geometric standard deviation is
multiplicative rather than arithmetic
• Can simply take the logarithm of SPECRatios, compute the
standard mean and standard deviation, and then take the
exponent to convert back:
1 n

GeometricMean  exp    ln SPECRatioi 
 n i 1

GeometricStDev  exp StDevln SPECRatioi 
11
How Summarize Suite Performance (5/5)
• Standard deviation is more informative if know
distribution has a standard form
– bell-shaped normal distribution, whose data are
symmetric around mean
– lognormal distribution, where logarithms of data--not
data itself--are normally distributed (symmetric) on a
logarithmic scale
• For a lognormal distribution, we expect that
68% of samples fall in range mean / gstdev2 , mean  gstdev2
95% of samples fall in range mean / gstdev , mean  gstdev 
• Note: Excel provides functions EXP(), LN(), and
STDEV() that make calculating geometric mean and
multiplicative standard deviation easy
12
Example Standard Deviation (1/2)
• GM and multiplicative StDev of SPECfp2000 for Itanium 2
14000
10000
GM = 2712
GSTEV = 1.98
8000
6000
5362
4000
2712
2000
1372
apsi
sixtrack
lucas
ammp
facerec
equake
art
galgel
mesa
applu
mgrid
swim
0
fma3d
Outside 1 StDev
wupwise
SPECfpRatio
12000
13
Example Standard Deviation (2/2)
• GM and multiplicative StDev of SPECfp2000 for AMD Athlon
14000
10000
GM = 2086
GSTEV = 1.40
8000
6000
4000
2911
2086
1494
2000
apsi
sixtrack
lucas
ammp
facerec
equake
art
galgel
mesa
applu
mgrid
swim
0
fma3d
Outside 1 StDev
wupwise
SPECfpRatio
12000
14
Comments on Itanium 2 and Athlon
• Standard deviation of 1.98 for Itanium 2 is much
higher-- vs. 1.40--so results will differ more
widely from the mean, and therefore are likely
less predictable
• Falling within one standard deviation:
– 10 of 14 benchmarks (71%) for Itanium 2
– 11 of 14 benchmarks (78%) for Athlon
• Thus, the results are quite compatible with a
lognormal distribution (expect 68%)
15
Fallacies and Pitfalls (1/2)
• Fallacies - commonly held misconceptions
– When discussing a fallacy, we try to give a counter example.
• Pitfalls - easily made mistakes.
– Often generalizations of principles true in limited context
– Show Fallacies and Pitfalls to help you avoid these errors
• Fallacy: Benchmarks remain valid indefinitely
– Once a benchmark becomes popular, tremendous pressure to improve
performance by targeted optimizations or by aggressive interpretation
of the rules for running the benchmark: “benchmarksmanship.”
– 70 benchmarks from the 5 SPEC releases. 70% were dropped from the
next release since no longer useful
• Pitfall: A single point of failure
– Rule of thumb for fault tolerant systems: make sure that
every component was redundant so that no single
component failure could bring down the whole system
(e.g, power supply)
16
Fallacies and Pitfalls (2/2)
• Fallacy: Rated MTTF of disks is 1,200,000 hours or
 140 years, so disks practically never fail
– manufactures will put thousands of disks in a room, run them for a
few months and count the number that fail
– But disk lifetime is 5 years  replace a disk every 5 years;
on average, 28 replacements wouldn't fail
– A better unit: % that fail (1.2M MTTF = 833 FIT)
– Fail over lifetime: if had 1000 disks for 5 years
= 1000*(5*365*24)*833 /109 = 36,485,000 / 106 = 37
= 3.7% (37/1000) fail over 5 yr hr MTTF)
• But this is under pristine conditions lifetime (1.2M
– little vibration, narrow temperature range  no power failures
• Real world: 3% to 6% of SCSI drives fail per year
– 3400 - 6800 FIT or 150,000 - 300,000 hour MTTF [Gray & van Ingen 05]
17
Summary
• Quantify and summarize performance
– Ratios, Geometric Mean, Multiplicative
Standard Deviation
18
Next Topics
Fundamentals of Computer Design
3. Principles
Things To Do
• Find your partners for the class project
– Feb. 11, Thursday (email me the team information)
• Check out the class website about
– lecture notes
– reading assignments
19