Transcript Abiliene and FRGP Update

Introduction to NETS
Marla Meehl
NETS Manager
SCD Network Engineering and
Technology Section (NETS)
December 8, 1998
NCAR Scientific Computing Division
Supercomputing • Communications • Data
1
Basic Contextual Information
NCAR Scientific Computing Division
Supercomputing • Communications • Data
2
Role of NETS in UCAR
• NETS is responsible for almost all of UCAR networking
– Historical evolution for SCD to manage all UCAR networking
– Important for NETS to remain in SCD (periodic discussion of moving
NETS to UCAR administrative domain)
» http://www.scd.ucar.edu/nets/Introducing/organizationlocation.html)
• NETS has additional SCD networking responsibilities
– Discussed later
• NETS advised by NCAB
–
–
–
–
NCAB: Network Coordination and Advisory Board
Reports to SCD Director
Technical representatives from all parts of UCAR
Successful paradigm proposed by ITC to be replicated for other UCARwide functions to be managed in an NCAR Division
NCAR Scientific Computing Division
Supercomputing • Communications • Data
3
NETS Responsibilities
• Types of networking supported for UCAR & SCD
– All LANs
– All MANs
– All WANs
• Levels of networking supported for UCAR & SCD
–
–
–
–
Layer1: All physical cabling plant for UCAR/SCD
Layer2: All logical networking - VLANs/ELANs, etc. for UCAR/SCD
Layer3: All routing (99.9% IP) for UCAR/SCD
Layer4 & above support: a little for UCAR; a lot for SCD
» More details later
NCAR Scientific Computing Division
Supercomputing • Communications • Data
4
What NETS Doesn’t Do
• “NETS responsibility ends at the wallplate”
– “wallplate” means “telecommunications outlet” and is the
point at which building infrastructure network leaf-node
cabling terminates
– Other Divisions are responsible “past the wallplate”
» This mainly means they do the host-networking part
» NETS does consult on configuration, performance, etc.
» “Private” networking beyond the wall plates isn’t forbidden
– For SCD, NETS is involved with all aspects of networking:
» Supercomputer networking
» Host-based networking: routing, configuration, etc.
» Special networking research projects
• National Laboratory for Advanced Network Research (NLANR)
Engineering
• Hosting NLANR/CAIDA Web Cache Research Project
NCAR Scientific Computing Division
Supercomputing • Communications • Data
5
What NETS Doesn’t Do (cont.)
• NETS doesn’t do DNS, email, security policy, etc.
– NETS does implement security perimeters based on CSAC
recommendations
• NETS doesn’t do MSS networking: HiPPI, FC, etc.
– These use non-IP channel-extension protocols
• NETS doesn’t do telephones and PBXs
– NETS does install the telephone cabling
– And we do inter-site tie-lines
• NETS doesn’t do first-level NOC/operations
– Handled by Computer Room Operators
– They determine which Network Engineer to call
– We will visit network monitor station later
NCAR Scientific Computing Division
Supercomputing • Communications • Data
6
How Networking is Paid For
• UCAR networking funding mechanisms
– Space tax: all UCAR programs (including SCD) pay for
networking via an annual “tax” based upon square footage
occupied by the program
– Space tax pays for “standard service” as defined by NCAB
» Includes all LAN, MAN, and WAN networking necessary for,
and benefiting, UCAR as a whole
» Includes all UCAR cabling and core networking to the
“wallplate”
» Includes 10-Mbps service to the office
» Includes telephone wiring and inter-site telephone tie-lines
– NETS charges back for anything beyond standard service
» Host-connects greater than 10-Mbps
» “Rush” jobs (less than 1-week advance notice)
» “Special” networking (e.g., satellite hookups)
• SCD networking funding mechanism
– Line item in SCD budget
NCAR Scientific Computing Division
Supercomputing • Communications • Data
7
Magnitude of NETS Work
• NETS supports ~1,136 UCAR employees
– Located in 9 buildings at 4 different sites
•
•
•
•
NETS supports ~3,000 network-attached devices
NETS supports ~114 IP subnetworks
46 dialup lines (via 2 all-digital PRI T1 links)
~100 pieces of network-equipment
– routers, switches, monitorable repeaters, etc.
• Building cabling
– 920 Standard “wallplates” installed
– 1,360 “wallplates” to install by end of FY2000
• NETS consults with 63 UCAR member universities
– Involves 700 users of just SCD facilities, with 345 projects
involving 90 university facilities
NCAR Scientific Computing Division
Supercomputing • Communications • Data
8
Networking “Fun”Facts
• Total number of Ethernet switch ports available:
1950
• Total number of Ethernet switch ports used: 1750
• Total number of feet of backbone cable: 27,000
feet
• Total number of feet of wallplate cable:
–
–
–
–
–
Fiber: 17,000 feet
CAT5: 240,000 feet
10BaseT: 230,000 feet
Telephone: 300,000 feet
Total: 787,000 feet
NCAR Scientific Computing Division
Supercomputing • Communications • Data
9
Resources Available to NETS
• NETS budget (FY1999)
– $2,341,100 UCAR funding to NETS
– $261,769 SCD funding to NETS
• Total NETS staff: 15 people
– Type of Staff
» 8 Network Engineers
• Perform design, operation, tuning, trouble-shooting, etc.
» 4 Network Technicians
• Mainly Layer1 (cabling) construction
» 3 Administrative/Support Staff
• Source of staff funding
– 12 UCAR-funded staff
– 2 SCD-funded staff
– 1 staff funded by outside funding (NSF NLANR Program)
NCAR Scientific Computing Division
Supercomputing • Communications • Data
10
Overview of UCAR
LANs, MANs, and WANs
NCAR Scientific Computing Division
Supercomputing • Communications • Data
11
12
LANs
NCAR Scientific Computing Division
Supercomputing • Communications • Data
13
LAN Cabling
• Standard “wallplate” to each workspace
– Connects nearest telecommunications closet to:
» 4 Cat5 cables
» 2 Fiber cable pairs (62.5 micron Multimode)
» 2 Cat3 cables (mainly for telephone)
– Only 40% of space meets this standard (920 wallplates)
– 1,360 new wallplates must to be installed by end of FY2000
» Required to support Fast Ethernet (100BaseX)
» $2,000,000 project (approved by UCAR management)
• Closets connect to root closet with fiber bundles
– ML root closet is in SCD machine room (ML 29)
– FL root closet is in SCD machine room (FL2 3095)
• Network equipment goes in closets (~35 closets)
NCAR Scientific Computing Division
Supercomputing • Communications • Data
14
15
16
LAN Design & Equipment
• Backbone UCAR LAN network is largely ATM
– OC-3 (155-Mbps) so far; some OC-12 testing
– Use ATM ELANs in the core: one per VLAN
– 3 Cisco ATM switches (model 1010)
• Rest of network is mainly switched Ethernet
– VLAN-based (one VLAN per IP-subnet)
– 10BaseX and 100BaseX technology
– 23 Cisco 5500 Ethernet packet switches
• Routing
– 4 Cisco 7507 routers
– 1 Cisco 4700 router
– 1 Cisco 2500 router
• UCAR is essentially an all-Cisco shop
NCAR Scientific Computing Division
Supercomputing • Communications • Data
17
18
Important LAN Projects
• FY1999
–
–
–
–
FUN Recabling Project (FL4 Uniform Network)
ATD, MIS, COMET Computer Room Recabling
FL1 South Atrium Recabling
Y2K engineering
• FY2000
– Year 2000 Recabling Project
– 100BaseX standard service implementation/expansion
– Y2K troubleshooting
NCAR Scientific Computing Division
Supercomputing • Communications • Data
19
MANs
NCAR Scientific Computing Division
Supercomputing • Communications • Data
20
Basic MAN Networking
• Inter-site connectivity
– ML-FL OC-3 (155-Mbps) ATM link
» Also carries two virtual T1 voice tie-lines
– 10 Mbps link to Jeffco site
– T1 (1.5 Mbps) link to Marshall site
– UCAR-owned fiber between all FL campus buildings
• Home dial-up to NCAR
– 2 PRI T1 lines (46 56Kbps/ISDN lines)
– Cisco 5300 Remote Access Server
• OC3 ATM atmospheric laser link to NOAA,
Boulder (owned and operated by NOAA)
NCAR Scientific Computing Division
Supercomputing • Communications • Data
21
22
The BRAN MAN
NCAR Scientific Computing Division
Supercomputing • Communications • Data
23
BRAN
• Boulder Research and Administration Network
– “Fiber for a healthy community”
• Consortium to build private fiber loop in Boulder
–
–
–
–
–
City of Boulder
University of Colorado-Boulder
National Institute of Standards and Technology (NIST) - Boulder
National Oceanic and Atmospheric Administration (NOAA) - Boulder
NCAR/UCAR
• Connects partners’ facilities + US West & ICG POPs
– Includes ML-FL link of ~20 fiber pairs
• Construction estimated at $350,000/partner
• Essentially provides unlimited free bandwidth
• Bypasses US West
– Provides competition between US West & ICG
NCAR Scientific Computing Division
Supercomputing • Communications • Data
24
25
WANs
NCAR Scientific Computing Division
Supercomputing • Communications • Data
26
UCAR WAN Connections
• Commodity Internet Connection
– DS-3 (45-Mbps) Cable and Wireless service
– Cost-shared with local gigapop partners (more later)
– Steady 50% utilization; 85% peaks (5 min averages)
• OC-3 (155-Mbps) connection to NSF’s vBNS
• Planned OC-3 connection to UCAID’s Abilene
Internet2 network
• All UCAR WAN connections part of the Front
Range GigaPop (FRGP) operated by NETS (details
later)
NCAR Scientific Computing Division
Supercomputing • Communications • Data
27
NSF’s vBNS:
very-high-speed
Backbone Network Service
NCAR Scientific Computing Division
Supercomputing • Communications • Data
28
vBNS: History
• vBNS goals
– jumpstart use of high-performance networking for advanced
research while advancing research itself with high-performance
networking
– supplement Commodity Internet which has been inadequate for
universities since NSFnet was decommissioned
• vBNS started about 3 years ago with the 5 NSF
supercomputing centers
• vBNS started adding universities about 2 years ago
• NSF funding for vBNS ends March 2000
NCAR Scientific Computing Division
Supercomputing • Communications • Data
29
vBNS: The Network
•
•
•
•
Operated by MCI/Worldcom
ATM based network using mainly IP
OC-12 (622-Mbps) backbone
OC-12 (622-Mbps), OC-3 (155-Mbps) & DS-3 (45Mbps) to institutions
• 73 institutions currently connected
• 131 institutions approved for connection to vBNS
NCAR Scientific Computing Division
Supercomputing • Communications • Data
30
vBNS and NCAR
• NCAR was an original vBNS node
• 43 of 63 UCAR member-universities are
approved for vBNS (at last check on 8/1998)
• 28 UCAR members currently connected
• Major benefit for UCAR and its members
– greatly superior to the Commodity Internet
– example: more UNIDATA data possible
– example: terabyte data transfers possible
NCAR Scientific Computing Division
Supercomputing • Communications • Data
31
32
UCAID’s Abilene Internet2 Network
NCAR Scientific Computing Division
Supercomputing • Communications • Data
33
Abilene: History
• First called the Internet2 Project
• Then non-profit UCAID (University Corporation for
Advanced Internet Development) was founded
– UCAID is patterned after the UCAR model
– UCAID currently has 130 members (mostly universities)
• Abilene is the name of UCAID’s first network
• Note: Internet2 used to refer to:
– the Internet organization, which is now called UCAID
– the actual network, which is now named Abilene
– the concept for a future network, soon to be reality in the form of
Abilene
NCAR Scientific Computing Division
Supercomputing • Communications • Data
34
Abilene: Goals
• Goals: jumpstart use of high-performance
networking for advanced research while
advancing research itself with high-performance
networking (same as vBNS)
• To be operated and managed by the members
themselves (similar to the UCAR model)
• Provide an alternative when NSF support of the
vBNS terminates on March 2000
NCAR Scientific Computing Division
Supercomputing • Communications • Data
35
Abilene: The Basic Network
• Uses Qwest OC48 (2.4Gbps) fiber optic backbone
– grow to OC192 (9.6Gbps) fiber optic backbone
– Qwest to donate .5 billion worth of fiber leases over 5 years
• Hardware provided by Cisco Systems and Nortel
(Northern Telecom)
• Internet Protocol (IP) over SONET
– no ATM layer
• Uses 10 core router nodes at Qwest POPs
– Denver is one of these
NCAR Scientific Computing Division
Supercomputing • Communications • Data
36
Abilene: Status
• Abilene soon to be designated by NSF as an NSFapproved High-Performance Network (HPN)
– puts Abilene on an equal basis with vBNS
• Abilene reached peering agreement with vBNS so
NSF HPC (High Performance Connection) schools
have equal access to each other regardless of
vBNS or Abilene connection
• UCAID expects Abilene to come online 1/1999
– UCAID expects 50 universities online on 1/1999
– UCAID expects 13 gigapops online on 1/1999
• Abilene beta network now includes a half-dozen
universities
– plus exchanging routes with vBNS
NCAR Scientific Computing Division
Supercomputing • Communications • Data
37
Abilene and NCAR
• 48 of 63 UCAR member-universities are UCAID
members (at last check on 8/1998)
• NSF funding of vBNS terminates March 2000
• Same benefit for UCAR and its members as vBNS
– greatly superior to the Commodity Internet
– example: more UNIDATA data possible
– example: terabyte data transfers possible
NCAR Scientific Computing Division
Supercomputing • Communications • Data
38
39
40
The GigaPop Concept
NCAR Scientific Computing Division
Supercomputing • Communications • Data
41
What Is A GigaPop?
• Multiple sites agree to aggregate to a central
location and share high-speed access from
there, instead of each maintaining direct links
to multiple networks
• Share costs through sharing infrastructure
• Share Commodity Internet expenses
• Essentially statistical multiplexing of
expensive high-speed resources
– at any given time much more bandwidth is available to
each institution than each could afford without sharing
• Share engineering and management expertise
• More clout with vendors
NCAR Scientific Computing Division
Supercomputing • Communications • Data
42
43
Front Range GigaPop (FRGP)
NCAR Scientific Computing Division
Supercomputing • Communications • Data
44
FRGP: Current NCAR Services
•
•
•
•
•
•
vBNS access
Shared Commodity Internet access
Intra-Gigapop access
Web cache hosting
24 x 365 NOC (Network Operation Center)
Engineering and management
NCAR Scientific Computing Division
Supercomputing • Communications • Data
45
46
FRGP+Abilene: What Should NCAR Do?
• Why should NCAR connect to Abilene?
– fate of vBNS is unknown after March 2000
– 48 of 63 UCAR members are also Internet2 members
• Why should NCAR join a joint FRGP/Abilene effort?
– combined FRGP/Abilene effort saves NCAR money
– provides excellent intra-gigapop connectivity
– provides greater depth and redundancy of commodity internet
access
NCAR Scientific Computing Division
Supercomputing • Communications • Data
47
FRGP: Why NCAR as GP Operator?
• NCAR already has considerable gigapop operational
experience
• NCAR is already serving the FRGP members
– Abilene connection is an incremental addition to existing gigapop
– doesn’t require a completely new effort from scratch
• NCAR already has a 24 x 365 NOC
• NCAR has an existing networking staff to team with
the new FRGP engineer
• NCAR is university-neutral
NCAR Scientific Computing Division
Supercomputing • Communications • Data
48
FRGP: Membership Types
• “Full” members
– both Commodity Internet + Abilene and/or vBNS access
• Commodity-only members
– just Commodity Internet access
NCAR Scientific Computing Division
Supercomputing • Communications • Data
49
FRGP: Full Members
•
•
•
•
•
University of Colorado - Boulder
Colorado State University
University of Colorado - Denver
NCAR/UCAR
University of Wyoming
NCAR Scientific Computing Division
Supercomputing • Communications • Data
50
FRGP: Commodity-only Members
• Colorado School of Mines
• Denver University
• University of Northern Colorado
NCAR Scientific Computing Division
Supercomputing • Communications • Data
51
FRGP: Possible Future Members
•
•
•
•
U of C System
NOAA/Boulder
NIST/Boulder
NASA/Boulder
NCAR Scientific Computing Division
Supercomputing • Communications • Data
52
FRGP: But!!!
•
•
•
•
This is far from a done deal at this time!
Members still have funding issues
No agreements have yet been decided
Latest developments
– Qwest asked to bid on FRGP, but bid was unacceptably
expensive
NCAR Scientific Computing Division
Supercomputing • Communications • Data
53
54
FRGP: Why Add a Denver
Gigapoint?
• Much cheaper for most members to backhaul to
Denver instead of to existing NCAR gigapoint
– U of Wyoming, Colorado State, UofC Denver
• UofC Denver has computer room space that’s two
blocks from Denver’s telco hotel.
• But also don’t want to re-engineer NCAR gigapoint
– wanted to preserve vBNS backhaul to NCAR
– wanted to preserve MCI Commodity Internet backhaul to NCAR
– wanted to minimize changes to the existing gigapoint
• Incremental addition of Denver gigapoint is most
cost-effective engineering option
NCAR Scientific Computing Division
Supercomputing • Communications • Data
55
FRGP: Abilene Implications for NCAR
•
•
•
•
New annual expenses for NCAR
New one-time share of startup costs
NCAR employs & manages new FRGP engineer
NCAR manages additional network equipment
– including new off-site equipment in Denver
• Increased engineering responsibilities for NCAR
• Increased administrative/accounting responsibilities
for NCAR
NCAR Scientific Computing Division
Supercomputing • Communications • Data
56
Useful URLs
•
•
•
•
•
http://www.scd.ucar.edu
http://www.scd.ucar.edu/nets
http://www.ucar.edu/ucargen/groups/ncab/
http://www.vbns.net
http://www.ucaid.net
NCAR Scientific Computing Division
Supercomputing • Communications • Data
57