Transcript MIMD Distributed Memory Architectures

MIMD Distributed Memory
Architectures
message-passing
multicomputers
MIMD-DM organization
• Each node includes
– full processor (control and ALU)
– memory
– connection to interconnect network
• Typically commodity processors, memory
• Value in interconnect
– high speed, high bandwidth
MIMD-DM
Node
CPU
Comm
Node
M
e
m
CPU
Comm
Network
Node
CPU
Comm
M
e
m
M
e
m
MIMD-DM Issues
• Connection Network
– fast
– high bandwidth
– scalable
• Communications
– explicit message passing
– parallel languages
• Occam 2, variations of C, Pascal
– libraries for sequential languages
• PVM, MPI, Java with CSP
Message Passing
• Point-to-Point
• Requires explicit commands in program
– Send, Receive
• Must be synchronized among different processors
– Sends and Receives must match
– Avoid Deadlock -- all processors waiting, none able to
communicate
• Multi-processor communications
– e.g. broadcast, reduce
Deadlock
Send
Send
Send
Send
Message Passing Systems
• PVM Parallel Virtual Machine
– developed at national lab
– intended for use with local area networks
– adapted for most MIMD parallel computers
• IBM SP2, Cray T3E, SGI Origin
– Provides library of function calls for C or FORTRAN
• Send, Receive, broadcast, reduce
• message packing/unpacking
• synchronization
Message Passing Systems
• MPI Message Passing Interface
– developed by consortium of vendors, users, labs
– intended to replace proprietary systems, PVM
• thus providing portability
– takes best ideas from several systems
– adapted for most MIMD parallel computers
• IBM SP2, Cray T3E, SGI Origin
– Provides library of function calls for C or FORTRAN
• Send, Receive, broadcast, reduce
• message packing/unpacking
• synchronization
Message Passing Systems
• Occam 2
–
–
–
–
full parallel language
co-developed with processor, Inmos Transputer
provides parallelism within and among processors
uses CSP model
• Communicating Sequential Processes
• Developed by Anthony Hoare
• explicit point-to-point channels for communications
– no longer imp[ortant
• transputer fell behind in development race
Message Passing Systems
• Java with CSP
– intended for concurrent and parallel computing in
Java
– Based on CSP / Occam 2 model
– Provides processes, channels, in Java
• for a single processor
• between processors
– Processor-to-processor channels developed at Colgate
Interconnection Network
• Speed and Bandwidth are critical
• Low cost networks
– local area network (ethernet, token ring)
– can be set up with packages
• PVM
• MPI
• High Speed Networks
– The heart of a MIMD-DM Parallel Machine