Transcript 18.337 Introduction Super

18.337 Introduction
News you can use
• Hardware
– Multicore chips (2009: mostly 2 cores and 4
cores, but doubling) (cores=processors)
– Servers (often 2 or 4 multicores sharing
memory)
– Clusters (often several, to tens, and many
more servers not sharing memory)
Performance
• Single processor speeds for now no longer growing.
• Moore’s law still allows for more real estate per core
(transistors double/nearly every two years)
– http://www.intel.com/technology/mooreslaw/index.htm
• People want performance but hard to get
• Slowdowns seen before speedups
• Flops (floating point ops / second)
– Gigaflops (109), Teraflops (1012), Petaflops(1015)
• Compare matmul with matadd. What’s the difference?
Some historical machines
Earth Simulator was #1 now #30
Some interesting hardware
Nvidia
Cell Processor
Sicortex – “Teraflops from Milliwatts”
http://www.sicortex.com/products/sc648
http://www.gizmag.com/mit-cycling-human-powered-computation/8503/
Programming
• MPI: The Message Passing Interface
– Low level “lowest common denominator” language that the world
has stuck with for nearly 20 years
– Can get performance, but can be a hindrance as well
• Some say that there are those that will pay for a 2x
speedup, just make it easy
• Reality is that many want at least 10x and more for a
qualitative difference in results
• People forget that serial performance can depend on
many bottlenecks including time to memory
• Performance (and large problems) are the reason for
parallel computing, but difficult to get the “ease of use”
vs “performance” trade-off right.
Places to Look
• Best current news:
– http://www.hpcwire.com/
• Huge Conference:
– http://sc09.supercomputing.org/
• MIT Home grown software, now
Interactive Supercomputing (Star-P for
MATLAB®, Python, and R)
– http://www.interactivesupercomputing.com
Architecture Diagrams from Sam
Williams @ Berkeley
Bottom Up Performance Engineering: Understanding Hardware’s
implications on performance up to software
Top Down: measuring software and tweaking sometimes aware and
sometimes unaware of hardware
http://www.cs.berkeley.edu/~samw/research/talks/sc07.pdf
Want to delve into hard numerical
algorithms
• Examples:
– FFTs and Sparse Linear Algebra
• At the MIT level:
– Potential “not quite right” question: How do
you parallelize these operations?
– Rather what issues arise and why is getting
performance hard?
• Why is nxn matmul easy? Almost cliché?
• Comfort level in this class to delve in?
Old Homework
(emphasized for effect)
• Download a parallel program from
somewhere.
– Make it work
• Download another parallel program
– Now, …, make them work together!
SIMD
•
SIMD (Single Instruction, Multiple Data) refers to parallel hardware that can
execute the same instruction on multiple data. (Think the addition of two
vectors. One add instruction applies to every element of the vector.)
– Term was coined with one element per processor in mind, but with today’s deep
memories and hefty processors, large chunks of the vectors would be added on
one processor.
– Term was coined with a broadcasting of an instruction in mind, hence the single
instruction, but today’s machines are usually more flexible.
– Term was coined with A+B and elementwise AxB in mind and so nobody really
knows for sure if matmul or fft is SIMD or not, but these operations can certainly
be built from SIMD operations.
•
Today, it is not unusual to refer to a SIMD operation (sometimes but not
always historically synonymous with Data Parallel Operations though this
feels wrong to me) when the software appears to run “lock-step” with every
processor executing the same instruction.
– Usage: “I hear that machine is particularly fast when the program primarily
consists of SIMD operations.”
– Graphics processors such as NVIDEA seem to run fastest on SIMD type
operations, but current research (and old research too) pushes the limits of
SIMD.
SIMD summary
• SIMD (Single Instruction, Multiple Data)
refers to parallel hardware that can
execute the same instruction on multiple
data.
• One may also refer to a SIMD operation
(sometimes but not always historically
synonymous with a Data Parallel
Operation) when the software appears to
run “lock-step” with every processor
executing the same instructions.