Transcript PowerPoint - OptIPuter

OptIPuter Physical Testbed at UCSD,
Extensions Beyond the Campus Border
Philip Papadopoulos and Cast of Real Workers:
Greg Hidley
Aaron Chin
Sean O’Connell
Max Okumoto
Praveen Kumar
Mason Katz
David Hutches
Physical Campus Connections
• O(300) nodes (Storage, Compute, Visualization)
• O(30) switches
• How nodes are connected to the network
– Optical Core
– Site switches (2 x 10GigE) – 48 port GigE
– Rack Switches
• 24 or 48 port GigE copper
UCSD Packet Test Bed – Year 2/3
Quartzite Extensions
• Funded as NSF Major Research Instrumentation Award –
Companion to OptIPuter
• Observation: Packet-only switch structure not rich enough for
OptIPuter research
• 300+ OptIPuter Nodes
– Viz (3 tiled displays), Storage (48 node, 300 spindles), Compute several
hundred (x86, Opteron, Itanium never materialized)
• Current UCSD OptIPuter network is channel-bonded GigE. Starting
some 10 GigE Deployment (network not “fat” enough for optIPuter
research)
– Very odd behavior with 10-year old etherchannel technology
• Quartzite more closely matches the network capability to the nodes.
– At the end of three years: 0.5 Terabits into the Quartzite Switching Core
• Quartzite fundamental capability: Build hybrid networks
– Packet-switched, Circuit-switched, Wavelength-switched. Hybrid
combinations, and reconfigurable
Building Blocks of Quartzite
•
CWDM is inexpensive ($2500/connected endpoint)
– Colored GBIC
• Use 8 GigE frequencies centered at 1550nm (+/- j*20nn (j=1,2,3,4))
• + LR 10 GigE at 1320nm. Colored 10GigE XenPAK or SFP may be available by end of
proposal
– Passive optical multiplexers (Coarse). 9 channel, 1550nm centered GBICs +
1320nm Passband for 10GigE
– Comment: Since proposal (January 2004), uncooled DWDM lasers are more
readily available.
•
•
•
OOO optical switch (commercial) Glimmerglass. Use as optical patch panel
Custom-built wavelength-selective switch from Lucent
Standard packet switch-router. Want to extend the Chiaro but $$ limit this
expansion
– 32-port (10GigE) capable 6509 switch ordered with 8 10-GigE ports
•
Existing single mode fiber plant (already tested and terminated)
– 4 pairs/site on the UCSD campus
Quartzite Year 1 Deployment Plans
• Deploy OOO
Switch
– Optical “patch
panel”
• Add some L2 10
GigE connections
• Initial Experience
with CWDM
Year 2 Deployment Plans
• Add Several CWDM
Channels
• Deploy Wavelength
Switch
• Begin Significant 10
GigE Buildout
– Packet Switch
– 10 GigE into some
endpoint nodes
Final Planned Configuration
• >50 endpoints
connected at 10 GigE
• >= 32 Packet
switched
• >= 32 switched
wavelengths
• >= 300 Connected
endpoint
Glimmerglass Switch – Now Deployed
A Whole Host of Real Issues
• Network:
– Where are nodes connected?
– How are switches interconnected
•
•
•
•
Channel bond? 10-GigE?
Are there wavelengths to be allocated? How?
Can I build a VLAN that traverses specific physical links
Can have a library of different network configs for the testbed
• Node Software:
– Can I have root on nodes? How do I specify a particular OS Image on a
node?
– What nodes can I have? For how long? How do I determine that I am
(or not) in competition for resources on the testbed
• Grid Software:
– What Grid software is available as base configuration? What should be
• Performance
– Why does my long-distance channel give me asymmetric performance
Network Inventory Management
•
•
Schema designed to support management of
both physical and virtual topologies. Physical
topology is the network with connections as it
is. Virtual topology shows the actual network
flow paths.
Physical Topology Schema components
–
•
Switch model
( Mfr, Model name)
– Switch unique information
( MAC, IP, Hostname etc )
– Host model
( Processor, Memory etc )
– Hosts unique information
( MAC, IP, Hostname etc)
– Connections
( Physical connection among switch or host
entities )
– Trunk based connections
( Ports participating in an physical
trunk )
Connections are the basic primitives which allow
us to build, visualize and navigate the topology. Any
information corresponding to the network can be had
by an join of connections and other tables in the
schema.
PHYSICAL TOPOLOGY GRAPH
Optiputer Network Inventory Management
LOGICAL TOPOLOGY (Single VLAN) GRAPH
•
Logical topology adds an VLAN table to the
physical topology tables.
– VLAN composed of trunks.
– Each Trunk can be a single/multiple port to port
connection between same set of switches
– Schema supports retaining VLAN id when
modifying trunks and vice-versa.
•
•
•
Schema contains network information needed to program the network to construct virtual topologies.
VLAN table in conjunction with Connections and Trunks can be used to create the needed setup. User
can specify inputs in terms of IP address or Host names, which will be mapped to the tables above
through host and switch tables.
Database implemented in MySQL with Python wrappers to insert data and generate output using
GraphViz.
Work in progress to automate data collection.
OS and Software Integrations
• Rocks configurations on endpoints allow us to build
libraries of OS configs.
• Rolls allow programmatic extensions to a complete
cluster installation.
– E.g., Grid, Scheduler, Kernel Rolls allow overwrite/extension
– Visualization Roll captures tiled-display wall software
Extensions to Wider Area
Southern California CalREN-XD Build Out
Expanding the OptIPuter LambdaGrid
StarLight
Chicago
UIC EVL
NLR
PNWGP
Seattle
U Amsterdam
NU
NetherLight
Amsterdam
NLR
1 GE Lambda
10 GE Lambda
NASA
Ames
NASA Goddard
NASA
JPL
ISI
UCI
2
UCSD
NLR
Level 3
2
2
CENIC
Los Angeles
GigaPOP
NLR
SDSU
CalREN-XD
8
CICESE
CENIC/Abilene
Shared Network
8
CENIC
San Diego
GigaPOP
via CUDI
Southern California CalREN-HPR