Transcript PowerPoint - OptIPuter

The OptIPuter Project –
Removing Bandwidth as an Obstacle
In Data Intensive Sciences
Opening Remarks
OptIPuter Team Meeting
University of California, San Diego
February 6, 2003
Dr. Larry Smarr
Director, California Institute for Telecommunications and
Information Technologies
Harry E. Gruber Professor,
Dept. of Computer Science and Engineering
Jacobs School of Engineering, UCSD
The Move to Data-Intensive Science & Engineeringe-Science Community Resources
ALMA
LHC
Sloan Digital Sky Survey
ATLAS
Why Optical Networks Are Emerging
as the 21st Century Driver for the Grid
Scientific American,
January 2001
Parallel Lambdas Will Drive This Decade
The Way Parallel Processors Drove the 1990s
A LambdaGrid Will Be
the Backbone for an e-Science Network
Apps Middleware
Clusters
Dynamically
Allocated
Lightpaths
Switch Fabrics
Physical
Monitoring
C
O
N
T
R
O
L
P
L
A
N
E
Source: Joe Mambretti, NU
The Biomedical Informatics Research Network
a Multi-Scale Brain Imaging Federated Repository
BIRN Test-beds:
Multiscale Mouse Models of Disease, Human Brain Morphometrics, and
FIRST BIRN (10 site project for fMRI’s of Schizophrenics)
NIH Plans to Expand
to Other Organs
and Many Laboratories
GEON’s Data Grid Team
Has Strong Overlap with BIRN and OptIPuter
• Learning From The BIRN Project
– The GEON Grid:
– Heterogeneous Networks, Compute Nodes, Storage
– Deploy Grid And Cluster Software Across GEON
– Peer-to-Peer Information Fabric for Sharing:
– Data, Tools, And Compute Resources
NSF ITR Grant
$11.25M
2002-2007
Two Science “Testbeds”
Broad Range Of Geoscience Data Sets
Source: Chaitan Baru, SDSC, Cal-(IT)2
NSF’s EarthScope
Rollout Over 14 Years Starting
With Existing Broadband Stations
Data Intensive Scientific Applications
Require Experimental Optical Networks
• Large Data Challenges in Neuro and Earth Sciences
– Each Data Object is 3D and Gigabytes
– Data are Generated and Stored in Distributed Archives
– Research is Carried Out on Federated Repository
• Requirements
–
–
–
–
Computing Requirements  PC Clusters
Communications  Dedicated Lambdas Over Fiber
Data  Large Peer-to-Peer Lambda Attached Storage
Visualization  Collaborative Volume Algorithms
• Response
– OptIPuter Research Project
OptIPuter Inspiration--Node of
a 2009 PetaFLOPS Supercomputer
DRAM – 16 GB
DRAM
- 4MB
GB- -HIGHLY
HIGHLYINTERLEAVED
INTERLEAVED
64/256
5 Terabits/s
MULTI-LAMBDA
Optical Network
CROSS BAR
2nd LEVEL CACHE
Coherence
8 MB
640 GB/s
2nd LEVEL CACHE
8 MB
24 Bytes wide
240 GB/s
VLIW/RISC CORE
40 GFLOPS
10 GHz
...
24 Bytes wide
240 GB/s
VLIW/RISC CORE
40 GFLOPS
10 GHz
Updated From Steve Wallach, Supercomputing 2000 Keynote
Global Architecture of a 2009 COTS
PetaFLOPS System
10 meters=
50 nanosec Delay
3
2
4
5 ...
16
1
17
64
ALL-OPTICAL
SWITCH
63
...
18
...
32
49
48
Systems Become
GRID Enabled
128 Die/Box
4 CPU/Die
47
I/O
LAN/WAN
... 33 Multi-Die
Multi-Processor
46
Source: Steve Wallach, Supercomputing 2000 Keynote
From SuperComputers to SuperNetworks-Changing the Grid Design Point
• The TeraGrid is Optimized for Computing
–
–
–
–
1024 IA-64 Nodes Linux Cluster
Assume 1 GigE per Node = 1 Terabit/s I/O
Grid Optical Connection 4x10Gig Lambdas = 40 Gigabit/s
Optical Connections are Only 4% Bisection Bandwidth
• The OptIPuter is Optimized for Bandwidth
–
–
–
–
32 IA-64 Node Linux Cluster
Assume 1 GigE per Processor = 32 gigabit/s I/O
Grid Optical Connection 4x10GigE = 40 Gigabit/s
Optical Connections are Over 100% Bisection Bandwidth
Convergence of Networking Fabrics
• Today's Computer Room
– Router For External Communications (WAN)
– Ethernet Switch For Internal Networking (LAN)
– Fibre Channel For Internal Networked Storage (SAN)
• Tomorrow's Grid Room
– A Unified Architecture Of LAN/WAN/SAN Switching
– More Cost Effective
– One Network Element vs. Many
– One Sphere of Scalability
– ALL Resources are GRID Enabled
– Layer 3 Switching and Addressing Throughout
Source: Steve Wallach, Chiaro Networks
The OptIPuter Experimental
The UCSD OptIPuter Deployment
UCSD Campus Optical Network
To CENIC
Phase I, Fall 02
Phase II, 2003
Production Router
SDSC
SDSC
SDSC
SDSC
Annex
Annex
JSOE
Engineering
CRCA
Arts
SOM
Medicine
Chemistry
Phys.
Sci Keck
Collocation point
Preuss
High
School
6th
Undergrad
College
College
Node M
Collocation
Chiaro Router
SIO
Earth
Sciences
½ Mile
Source: Phil Papadopoulos, SDSC; Greg Hidley, Cal-(IT)2
Metro Optically Linked Visualization Walls
with Industrial Partners Set Stage for Federal Grant
• Driven by SensorNets Data
–
–
–
–
Real Time Seismic
Environmental Monitoring
Distributed Collaboration
Emergency Response
• Linked UCSD and SDSU
– Dedication March 4, 2002
Linking Control Rooms
UCSD
SDSU
44 Miles of Cox Fiber
Cox, Panoram,
SAIC, SGI, IBM,
TeraBurst Networks
SD Telecom Council
National Light Rail- Serving Very High-End
Experimental and Research Applications
• Extension of CalREN-XD Dark Fiber Network
– Serves Network Researchers in California Research
Institutions
– Four UC Institutes, USC/ISI, Stanford and CalTech
– 10Gb Wavelengths (OC-192c or 10G LANPHY)
– Dark Fiber
– Point-Point, Point-MultiPoint 1G Ethernet Possible
• NLR is a Dark Fiber National Footprint
– 4 - 10GB Wavelengths Initially
– Capable of 40 10Gb Wavelengths at Build-Out
– Partnership model
John Silvester, Dave Reese, Tom West-CENIC
National Light Rail Footprint
Layer 1 Topology
SEA
POR
SAC
NYC
CHI
OGD
DEN
SVL
CLE
FRE
PIT
KAN
NAS
STR
LAX
RAL
PHO
WAL
SDG
BOS
ATL
STH
DAL
JAC
15808 Terminal, Regen or OADM site
(OpAmp sites not shown)
Fiber route
John Silvester, Dave Reese, Tom West-CENIC
WDC
Calient Lambda Switches Now Installed
at StarLight and NetherLight
Data plane
8 GigE Data plane
64x64
MEMS
Optical Switch
8 GigE
128x128
MEMS
Optical Switch
16 GigE
8 GigE
16 GigE
“Groomer”
at StarLight
16-processor
cluster
8-processor
cluster
2 GigE
16 GigE
1 92
C
O
ps)
b
G
(10
“Groomer”
at
NetherLight
16-processor
cluster
2 GigE
8 GigE
16 GigE
Switch/Router
Switch/Router
Control plane
Control plane
NETHERLIGHT
GigE = Gigabit Ethernet (Gbps connection type)
Source: Maxine Brown
Amplified Collaboration Environments
Collaborative
Passive Stereo
Display
Collaborative Tiled Display
Accessgrid
Multisite
Video Conferencing
Collaborative
Touch Screen
Whiteboard
Wireless
Laptops &
Tablet PCs To Steer The Displays
Source: Jason Leigh
The OptIPuter 2003
Experimental Network
Wide Array of Vendors
OptIPuter Software Research
• Near-term Goals:
– Build Software To Support Applications With Traditional Models
– High Speed IP Protocol Variations (RBUDP, SABUL, …)
– Switch Control Software For DWDM Management And Dynamic Setup
– Distributed Configuration Management For OptIPuter Systems
• Long-Term Goals:
– System Model Which Supports:
– Grid
– Single System
– Multi-System Views
– Architectures Which Can:
– Harness High Speed DWDM
– Exploit Flexible Dispersion Of Data And Computation
– New Communication Abstractions & Data Services
– Make Lambda-Based Communication Easily Usable
– Use DWDM to Enable Uniform Performance View Of Storage
Source: Andrew Chien, UCSD
Photonic Data Services & OptIPuter
6. Data Intensive Applications (UCI)
5a. Storage (UCSD)
5b. Data Services –
SOAP, DWTP, (UIC/LAC)
4. Transport – TCP, UDP, SABUL,… (USC,UIC)
3. IP
2. Photonic Path Serv. – ODIN, THOR,... (NW)
1. Physical
Source: Robert Grossman, UIC/LAC
OptIPuter is Exploring Quanta
as a High Performance Middleware
• Quanta Is A High Performance Networking Toolkit / API
• Quanta Uses Reliable Blast UDP:
– Assumes An Over-Provisioned Or Dedicated Network
– Excellent For Photonic Networks
– Don’t Try This On Commodity Internet!
– It Is Fast!
– It Is Very Predictable
– We Give You A Prediction Equation To Predict Performance
– It Is Most Suited For Transferring Very Large Payloads
• RBUDP, SABUL, and Tsunami Are All Similar Protocols
That Use UDP For Bulk Data Transfer
Source: Jason Leigh, UIC
XCP Is A New Congestion Control Scheme
Which is Good for Gigabit Flows
• Better Than TCP
– Almost Never Drops Packets
– Converges To Available Bandwidth Very Quickly, ~1Round Trip Time
– Fair Over Large Variations In Flow Bandwidth and RTT
• Supports existing TCP semantics
– Replaces Only Congestion Control, Reliability Unchanged
– No Change To Application/Network API
• Status
– To Date: Simulations and SIGCOMM Paper (MIT).
– See Dina Katabi, Mark Handley, and Charles Rohrs, "Internet Congestion
Control for Future High Bandwidth-Delay Product Environments." ACM
SIGCOMM 2002, August 2002. http://ana.lcs.mit.edu/dina/XCP/
– Current:
– Developing Protocol, Implementation
– Extending Simulations (ISI)
Source: Aaron Falk, Joe Bannister, ISI USC
Multi-Lambda Security Research
• Security Frequently Defined Through Three Measures:
– Integrity, Confidentiality, And Reliability (”Uptime”)
• Can These Measures Can Be Enhanced By Routing
Transmissions Over Multiple Lambdas Of Light?
• Can Confidentiality Be Improved By Dividing The
Transmission Over Multiple Lambdas And Using
“Cheap” Encryption?
• Can Integrity Be Ensured Or Reliability Be Improved
Through Sending Redundant Transmissions And
Comparing?
Research on Developing
an Integrated Control Plane
Megabit
Stream
Logical
Label
Switching
Gigabit
Stream
Optical
Lambda
Switching
Bursty
Traffic
Tera/Peta
Stream
Optical
Lambda
Burst
Inverse
Switching Multiplexing
Integrated Control Plane
Multiple User
Data Planes
OptIPuter Transforms Individual Laboratory
Visualization, Computation, & Analysis Facilities
Fast polygon and
volume rendering
with stereographics
+
GeoWall
= 3D APPLICATIONS:
Earth Science
Underground
Earth Science
Anatomy
Neuroscience
GeoFusion GeoMatrix Toolkit
Rob Mellors and Eric Frost, SDSU
SDSC Volume Explorer
Visible Human Project
NLM, Brooks AFB,
SDSC Volume Explorer
Dave Nadeau, SDSC, BIRN
SDSC Volume Explorer
The Preuss School UCSD OptIPuter Facility