Transcript Brookhaven Science Associates US Department of - Indico

Network Services
BNL Update
John Bigrow
October 16, 2008
Brookhaven Science Associates
U.S. Department of Energy
1
Network Services

BNL LHCOPN Overview
• LHCOPN Failover and Policy Based Routing (PBR) Problems
– The Observed LHCOPN Failure Modes
– Solution Options
• BNL / Tier-2 Implementations
• Performance / Monitoring
Brookhaven Science Associates
U.S. Department of Energy
2
Network Services

Why PBR is ineffective at BNL
• Many services and systems support multiple projects such as RHIC
and Atlas together (Cost and Resource constraints).
• Many of these systems are multi-homed crossing multiple subnets
within the BNL campus.
• Security constraints require firewall protections at the BNL
perimeter.
Brookhaven Science Associates
U.S. Department of Energy
3
Network Services

Why PBR is ineffective at BNL cont.
• PBR is a single box solution.
• Limits access to other BNL computing resources such as RHIC
compute clusters, QCD, and Blue Gene if desired.
Brookhaven Science Associates
U.S. Department of Energy
4
Network Services

Some Key BNL Network Characteristics
• All 130.199.0.0/16 is Firewall, ACL, and Proxy protected at the BNL site perimeter boundary.
– The entire BNL campus address space is inside the perimeter firewall.
• 192.12.15.0/24 External to Perimeter Firewall (Direct WAN).
– Not further custom sub-netted into different access requirement zones. Treated as a single entity
for both Internet and LHCOPN.
• Internal IDS and Host monitoring capabilities.
– Layer-2 (MAC) block or Layer-3 (Null Route) for Policy non-compliance, virus infestation or
malicious scanning activity. This applies to both inbound and outbound IP traffic.
– Firewalls prevent spoofed source address from exiting BNL.
• Many additional Internal (130.199.0.0/16) Firewalled Enclaves.
– Many BNL internal campus firewall protected enclaves to provide additional security and restrict
and monitor access.
Brookhaven Science Associates
U.S. Department of Energy
5
Network Services
LHCOPN Basic Diagram
Internet
ES net Routers
BNL Routers
Dual Attach Links
Single Attach Links
Layer-2 MPLS Circuits
CERN Routers
Third Party Routes
Local MPLS
BGP Peerings
Layer-2 MPLS Circuits
GEANT MPLS
GEANT MPLS
LHC net MPLS
ES net MPLS
PBR Traffic
Map
PIC Routers
BGP Peerings
192.12.15.0/24
Perimeter Firewall
130.199.0.0/16
BNL Atlas Dcache
Servers
BNL SRM
Brookhaven Science Associates
U.S. Department of Energy
6
PIC LAN
Network Services
BNL LHCOPN Flows
BNL Routing Hierarchy
Internet
ES net Internet
Router
Cisco 7500
SERIES
PBR
130.199.48.0/23
130.199.185.0/24
E2E Circuits CERN
SARA
Triumf
Prague
Cisco 7500
SERIES
192.12.15.0/24
BNL Internet
Router
BNL Perimeter
Firewall
130.199.0.0/16
0
A
1
2
3
4
5
RSP
RSP
8
9
10
11
12
B
BNL Atlas
Networks
0
A
1
2
3
4
5
RSP
RSP
8
9
10
11
12
B
Brookhaven Science Associates
U.S. Department of Energy
BNL Campus
Network
7
Network Services

PBR Limitations, Problems and Issues
• MPLS doesn’t detect or propagate link-layer failure indicators.
• Third – Party route propagation.
• Limited criteria for defining “Policy” and “Actions”.
–
–
–
–
Only routers support PBR, Firewalls do not
Essentially an access list to define allowed traffic
Limited to single router implementation
Infancy technology
Brookhaven Science Associates
U.S. Department of Energy
8
Network Services

PBR Limitations, Problems and Issues cont.
• Initial tests didn’t detect failures since the Dcache
workers are on a separate network that does not use
PBR but relies on default routing.
– First observed problem was with the Dcache SRM service from Triumf (on
internal BNL LAN).
– Second observed problem was with Dcache SRM service with PIC
disconnected from the LHCOPN (on internal BNL LAN).
– Third observed problem was with Panda service, not defined as part of the
LHCOPN (on internal BNL LAN).
Brookhaven Science Associates
U.S. Department of Energy
9
Network Services

Perimeter Architecture
BNL Perimeter & ATLAS LANs
M. O’Connor ESnet
130.199.0.0/16
130.199.188.0/24
130.199.48.0/23
RHIC/ATLAS
192.12.15.0/24
BNL Firewall
ATLAS
BNL Perim. Sw.
HSRP
Amon/Mutt
Gen. IP
130.199.3.24
130.199.3.224
192.12.15.124
130.199.3.124
192.12.15.224
AMON
IBGP
MUTT
IBGP
Source
Inadmissible
EBGP
TE-7/3.3514 USLHCNET Second 10G
HSRP
Mutt/Amon
ATLAS
192.12.15.24
PBR
ATLAS
Admissible
EBGP
TE-3/1.2603
IP primary
TE-2/4.2605
IP Second
TE-7/3.2608
TE-7/3.3513
TE-7/3.3514
TE-7/3.1220
TE-7/3.2612
Triumph 1G
USLHCNET Prime 10G
USLHCNET Second 10G
CESNET/Prague 1G
SARA 1G
BNL-MR1
Primary IP Not Policed
CESNET/Prague 1G
SARA 1G
USLHCNET 3514 2nd 3G
LIMAN #2
LIMAN #1
Secondary IP Not Policed
Triumph 1G
USLHCNET 3513 1st 3G
USLHCNET 3513
Internet2 IP
AOFA-CR2
AOFA-SDN1
Prague (CESNET)
SARA (GEANT)
All circuits 10G Ethernet
ESnet SDN
IP
Brookhaven Science Associates
U.S. Department of Energy
10
Network Services
Current Site Architecture
BNL / CVLP
PDN
BNL / Esnet
PDN
S 3513, 3514, 3524
P 2603 2604
Te 13/3-4
10G
PIX Firewall SERIES
POWER
10G
PIX Firewall SERIES
POWER
FWSM
BNL.GOV
BNL.ORG
NSLSUSERS.ORG
ACT
NETWORK
ESnet
CV/LP
Atrica
FE
GE 7/2
12/1 12/2
Stateful
Link
130.199.3.24
HSRP
M=130.199.3.124
A=130.199.3.224
Mutt
Amon
ACT
NETWORK
P 3513, 3514, 3524Keyspan
S 2603 2604 Adva 30g λ
Te 13/3-4
Cern and Atlas
Routing
Interfaces
192.12.15.24
HSRP
M=192.12.15.124
A=192.12.15.224
SW9
IBGP peering
AS 43
Core
Mutt handling
CV/LP
Amon All
others
Shu
Tefnut
Anubis
Isis
Nephthys
Osiris
Setup Duplicate WAN
interfaces on Mutt and
Amon. Keep in shutdown
till needed.
Brookhaven Science Associates
U.S. Department of Energy
11
SW7
Network Services

Options
1) Open default routing for 192.12.15.0/24 and
130.199.0.0/16 (Our Preference No PBR)
– Minimal security and performance impact.
- If the end hosts are reachable they will still be reachable.
- Doesn’t alter host-based ACL ‘s or user-based access
requirements.
- Simply another connection path to the destination networks in
lieu of the Internet
- Just another inbound path into BNL
– BNL IDS will still block all inbound or outbound malicious activity.
– Mirrors our intended Tier-2 architecture.
Brookhaven Science Associates
U.S. Department of Energy
12
Network Services

Options
1) Open default routing for 192.12.15.0/24 and
130.199.0.0/16 (Our Preference No PBR) cont.
– Most scalable solution.
- Maximizes availability for access to and from BNL
- Allows a service to be accessible anywhere within BNL
- Future experiments and collaborations with 130.199.0.0/16
– Supports redundant network hardware infrastructure
– Covers all observed PBR failure modes
Brookhaven Science Associates
U.S. Department of Energy
13
Network Services

Options cont.
2) Service migration to 192.12.15.0/24, eliminate 130.199.0.0/16
on LHCOPN
– Operates with default routing no PBR, not needed completely defined
network.
– Outside Firewall for both LHCOPN and Internet access.
– Has transitioned all known failure modes.
– Totally removes 130.199/16 from the LHCOPN. PBR via Internet for
130.199.0.0/16 for LHCOPN prefixes.
– Limited IP address space on 192.12.15.0/24.
- This network might need to be shared with RHIC / CCJ near term.
– Not fully hardware redundant (single point of failure for LHC circuits).
Brookhaven Science Associates
U.S. Department of Energy
14
Network Services

Options cont.
3) Object (SLA) or Virtual Route Tracking with PBR
–
–
–
–
Might be processor bound (route tracking)
Handles the direct CERN / Triumf Peerings
Does not handle Third Party routes
PBR does not scale beyond an individual router
- Forces a single point of failure network design
Brookhaven Science Associates
U.S. Department of Energy
15
Network Services

BNL / Tier-2 Connectivity Options
• NO PBR for security, but firewall conduits may be
necessary for access into 130.199.0.0/16
– Path / Traffic engineering via Starlight or 32AOA with ES net
- Requesting a BNL peering presence at both locations
– Path to BNL via Internet (default and backup)
– Generic Internet routing for access to and from BNL
Brookhaven Science Associates
U.S. Department of Energy
16
Network Services

Performance / Monitoring
• Internet 2 toolkit with perfSONAR, NDT, NPAD etc.
• Cacti for psudo-realtime and historical information
Brookhaven Science Associates
U.S. Department of Energy
17
Network Services
Brookhaven Science Associates
U.S. Department of Energy
18
Network Services
Brookhaven Science Associates
U.S. Department of Energy
19
Network Services
Brookhaven Science Associates
U.S. Department of Energy
20
Network Services
Brookhaven Science Associates
U.S. Department of Energy
21
Network Services

Thanks
• A few acknowledgements and words of thanks are in
order.
– ES net for providing BNL bandwidth and connectivity.
– Internet2 for the perfSONAR software and support.
Brookhaven Science Associates
U.S. Department of Energy
22
Network Services
Questions/Comments
???
Brookhaven Science Associates
U.S. Department of Energy
23
Network Services
BNL Points of Contact

Scott Bradley, Manager of Network Services
• 631.344.5745, [email protected]

John Bigrow, Senior Network Architect
• 631.344.2648, [email protected]
Brookhaven Science Associates
U.S. Department of Energy
24