Transcript R estoring Service Post Attack, September 11th

I C S
Integrated Communication Services
The Network During the Crisis
- A New York City Perspective
Doug Carlson, Director of Network Services
Tim Lance, President and Board Chair, NYSERNet
N
ew York University
N
ew York University




Largest private university in US
Located in Greenwich Village area of New
York City
 Concentrated North of Houston (NoHo) up
through 14th Street
School of Medicine and NYU Hospital near
21st Street
Residence Halls extend down to blocks away
from Ground Zero
legend
OC-3
OC-12
DS-3
NYSERNet 2000 PoP
Gateway Network
Albany
Syracuse
Rochester
CA*Net
Buffalo
Abilene
New York
Abilene
vBNS+
Topology of the NYSERNet Network.
R
esearch Network









Connected Institutions (R&E Network)
Amer. Museum of Natural History
Columbia University
Cornell University
Hauptman-Woodward Institute *
Marist College *
New York University
Pace University *
Rensselaer Polytechnic Institute









Rochester Institute of Technology
Rockefeller University
SUNY Albany
SUNY Binghamton
SUNY Buffalo
SUNY Stony Brook
Syracuse University
University of Rochester
Weill Medical College
* Approved for Funding.
Manhattan Project Participants – Geographic Map
efore September 11th
B
Photograph by Doug Carlson – 7/2001
mpact of September
th
11
I
Photograph by Doug Carlson – 9/23/2001
Relationship between 7 World
Trade Center and 140 West
Street prior to September
11th.
Image of Manhattan from
IKONOS Satellite taken
Summer 2000.
60 Hudson Street
Images courtesy www.spaceimages.com.
140 West Street
World Trade Center
Image of Manhattan from
IKONOS Satellite taken
shortly after the attack on
the World Trade Center.
60 Hudson Street
Images courtesy www.spaceimages.com.
140 West Street
World Trade Center
World Trade Center complex
showing relationship between
7 World Trade Center and 140
West Street.
Photographs Copyright 2001 Verizon Communications. All rights reserved.
Exterior view of 140 West
Street showing debris and
damage from collapse of 7
World Trade Center.
Photographs Copyright 2001 Verizon Communications. All rights reserved.
An interior view of 140 West
Street showing damage from
collapse of 7 World Trade
Center.
N
etwork Status - NYSERNet








As a result of the attack…
CO at 140 West Street severely damaged.
NYSERNet OC-48 SONET ring operational but no longer
a ring.
NYSERNet R&E Network operational.
NYSERNet POP at 60 Hudson Street operational.
Abilene connections operational.
vBNS+ connection down.
R&E service for SUNY Stony Brook and AMNH is down.
N
etwork Status - Commodity








As a result of the attack…
CO at 140 West Street severely damaged.
AppliedTheory equipment largely undamaged.
AppliedTheory Network operational.
AppliedTheory POP at 60 Hudson Street operational.
Customers connecting to 60 Hudson operational.
Customers connecting at 140 West are down.
Customers connecting via Garden City, Deer Park,
and White Plains are down.
R
estoring Service
Restoring Service to Mt.
Sinai & School of Medicine
Prepared by Doug Carlson,
New York University
Photo of 140 West Street. Copyright 2001
Verizon Communications. All rights reserved.
R
estoring Service

Post Attack



Phone systems immediately get overloaded
Lines by all payphones in NYU area
Manhattan phone and cell systems overwhelmed
Access to Lower Manhattan impossible
Pedestrian access limited below 14th Steet (Union Square)
Pedestrians required to have reason to go beyond
checkpoint and be able to show ID
Some students and others left ID behind
R
estoring Service

Post Attack



Initially, no traffic below 14th Street unless emergency or
government vehicle
NYU worked with Mayor’s office to get some critical
deliveries through to NYU locations
Most businesses and restaurants closed below 14th Street
NYU Administration establishes a Crisis Command Center
Initially, did not have sufficient phone or data service
R
estoring Service




Post Attack
Campus phone system becomes overloaded for a period of time
(no dial-tone)
Internet connections provide only reliable information links to
family, friends, co-workers:

Email

Web

Instant Messaging
VoIP (Cisco) phones used to communicate with staff unable to
come into Manhattan and unable to get through on traditional
telephone/cell systems
R
estoring Service


Post Attack
Information Technology Services (ITS) takes point in keeping NYU
community informed via Web, mass-mailings via email, voice mail
announcements and Help Desk services.

Mt. Sinai Hospital traffic routed through NYU.

Phone banks set up to allow students to call home for free.

Temporary shelter set up for students and staff who were
evacuated.
R
estoring Service


Post Attack
Evacuated students and staff are relocated to hotels around the
city:

Students given some money for essentials.



Arrangements are made to get new sets of books for
students.
Loans of laptops and desktop computers offered.
Preparations made to increase dial-in capacity via ISP.
(Eventually not needed due to students returning to their
residence halls.)
R
estoring Service
Columbia, IP
Telephony & The
New York Academy of
Medicine.
Prepared by Alan
Crosswell, Columbia
University
Photo of 140 West Street and WTC 7
courtesy of the Westchester Emergency
Communications Association,
www.weca.org
R
estoring Service




Early Morning, September 11th
Dealing with start of semester file server performance issues
after a failed Summer file server upgrade project.
Trying to get a handle on filtering Code Red with our Catalyst
6509’s.
Turning off many user network ports for compromised hosts day in
and day out.
R
estoring Service




Post Attack, September 11th
Trying to find out where all missing staff are.
Discover phone trunks are overloaded.
Send email to friends asking them to call our families and let
them know we are OK.
R
estoring Service

Post Attack, September 11th

Administration establishes response team. They request that we:



Set up large lecture halls with CNN; Students are crowding
around TV monitors in lounge areas.
Establish net2phone-like functionality so students can phone
home.
Send a mass e-mail to the Columbia community from the
President.
R
estoring Service




Getting CNN Out There, September 11th
Start planning to drag an IPTV encoder over to a cable box in a
dorm.
Staff drag some coax from a lounge TV monitor to a large lecture
hall.
Read on wg-multicast list that Northwestern has started
multicasting CNN, but find our multicast connectivity is broken.
R
estoring Service






Getting Phones Up, September 11th
One staff member tests net2phone….
Send email to Videnet & others + CS SIP group requesting use of
PSTN gateways to route around local congestion. ~11am.
End up configuring H.323 Polycoms via UNC Chapel Hill, and
Cisco IP phones via Penn State, and
SIP phones via 4 sites (Yale, Dynamicsoft, Nortel, Clarent).
R
estoring Service





Getting Phones Up, September 11th
Net2phone works but it’s not easy & requires a credit card….
Polycoms work but also not so easy….
CS puts 4 SIP phones in CS conference room. 2:20pm
We put 2 Cisco IP phones via Penn State in student center lobby.
Just dial 8 and get a PSU dial tone and call home! 4:00pm.
R
estoring Service




Getting Phones Up, September 11th
By the time we had the IP phones in place trunk congestion had
declined.
We had many many other Videnet sites offer their H.323 – H.320
PSTN gateways.
Thanks to all of you!
R
estoring Service





September 11th Onward
One of our dialup modem pools is out (and still out). A major
expensive carrier.
Our free T1’s from a small carrier are up on our main pool.
Weekly Email volume increased by 40% over prior week.
Daily volume on Thursday 9/13 was 800,000 messages: double last
year’s.
R
estoring Service




September 11th Onward
Filtering SIRCAM, nimda, Code Red, WTC viruses.
Turning off lots of ports of compromised hosts and attempting to
deal with helping people reformat and reinstall their systems.
Established outbound P2P traffic shaping.
R
estoring Service



September 18th
After a week, NYAM’s T1
service via 140 West St is still
down and low priority for
restoration.
Set up an 802.11b link across
Central Park and turn it up
Wednesday 9/19.
R
estoring Service
Restoring Service to
Rockefeller, Weill, & HSS.
Prepared by Armand
Gazes, The Rockefeller
University
Photo of new cable being placed on West
Street. Copyright 2001 Verizon
Communications. All rights reserved.
key
Commodity Path
Weill Medical
Center of CU
Hospital for
Special Surgery
The Rockefeller
University
R&E Path
Rockefeller
Router
AppliedTheory
Router
Rockefeller
NYSERNet
Switch
AppliedTheory
Router
AppliedTheory
NYSERNet
Internet
Internet2
Network Connectivity Prior to September 11th.
key
Commodity Path
Weill Medical
Center of CU
Hospital for
Special Surgery
The Rockefeller
University
R&E Path
Rockefeller
Router
AppliedTheory
Router
Rockefeller
NYSERNet
Switch
AppliedTheory
Router
AppliedTheory
NYSERNet
Internet
Internet2
Network Connectivity Subsequent to September 11th.
key
Commodity Path
Weill Medical
Center of CU
Hospital for
Special Surgery
R&E Path
The Rockefeller
University
Rockefeller
Router
AppliedTheory
Router
Rockefeller
NYSERNet
Switch
AppliedTheory
Router
AppliedTheory
NYSERNet
Router
NYSERNet
Buffalo
AppliedTheory
Router
Internet
Commodity Service Restored.
Internet2
R
estoring Service





Restoring commodity service…
To Rockefeller University, Weill Medical Center, and the Hospital
for Special Surgery.
Service restored through Rockefeller’s R&E connection and
emergency installation of a jumper between NYSERNet and
AppliedTheory in Buffalo.
Key - cooperation among campus personnel, commercial service
providers and NYSERNet.
Service restored by Wednesday afternoon, the 12th.
R
estoring Service
Restoring Commodity and R&E Services.
Tim Lance, NYSERNet.
New York Stock Exchange – Photo by Doug Carlson
AppliedTheory
AppliedTheory
Long Island
NYSERNet
Applied
Theory
NYSERNet
Applied
Theory
60 Hudson St.
140 West St.
Verizon
Verizon
NYSERNet Metropolitan OC-48 SONET Ring
Broad St.
Verizon
NYSERNet Metro Ring & AppliedTheory Prior to September 11th.
AppliedTheory
AppliedTheory
Long Island
NYSERNet
Applied
Theory
NYSERNet
Applied
Thory
60 Hudson St.
140 West St.
Verizon
Verizon
NYSERNet Metropolitan OC-48 SONET Ring
Broad St.
Verizon
NYSERNet Metro Ring & AppliedTheory After September 11th.
AppliedTheory
AppliedTheory
Long Island
NYSERNet
Applied
Theory
NYSERNet
Applied
Thory
60 Hudson St.
140 West St.
Verizon
Verizon
NYSERNet Metropolitan OC-48 SONET Ring
Broad St.
Verizon
Service Restored Utilizing NYSERNet SONET Ring.
N
etwork Status - Commodity





Restoring commodity service…
Polytechnic University - commodity service restored on 10/1.
Long Island – commodity traffic still transiting NYSERNet SONET Ring.
White Plains – connected to Garden City and then onto NYSERNet Ring.
Rockefeller Solution - evolving into standby service for all customers
having both NYSERNet R&E service and AppliedTheory commodity service.

West Street Service
– a few commodity T1’s still down.
N
etwork Status – NYSERNet




Restoring R&E service…
American Museum of Natural History
SUNY Stony Brook – service restored on 9/25.
vBNS+ – service restored almost immediately.
– service restored on 9/20.
O
ther Stories

New York University extended service to Mt.Sinai and School of
Medicine.

Hofstra University offered NYSERNet bandwidth on its OC3 if it could
be utilized to serve any impacted institutions.

City University of New York hosted the New York City Board of
Education Web site to ensure that parents could remain informed on the status
of their children’s schools.

American Museum of Natural History staff offered to serve as
remote hands for any institution needing onsite service.

Columbia University provided New York Academy of Medicine with
access via wireless service.

NYSERNet offered to allow AppliedTheory to utilize its Manhattan SONET
ring to restore commodity service, if feasible.
C
onclusions
C
onclusions







Lessons learned…
The value of networks in sustaining communications.
The survivability of the technology.
How much can be accomplished quickly when the
urgency is clear.
The value in multiple independent paths to the
Internet.
The value of government investment in science and
the often unanticipated nature of the return on that
investment.
The value of developing an Emergency Response Plan
C
hallenges Ahead





Networks are necessities, not luxuries…
Networks are now mission critical.
Diverse capabilities must back each other up.
Our constituents are flexible, we must be too.
Merging technologies can lead to redundancy.
C
hallenges Ahead




Industry Contraction and Consolidation
Mergers, acquisitions, business failures.
Continued geographic concentration.
Reduced capital investment.
C
hallenges Ahead





Shifting Public Policy
H.R. 3162, The Uniting and Strengthening America by
Providing Appropriate Tools Required to Intercept and
Obstruct Terrorism (USA PATRIOT) Act of 2001.
Lack of controlling authority viewed as a deficiency.
House Science Committee considering Cyber security
bill.
Senate Commerce Committee considering expanded
government role.
C
hallenges Ahead





Prioritization of Limited Resources
Attracting, retaining and training staff.
Funding.
Don’t chase the bubble.
Base next generation design on controlling the
transport medium, with physical and logical redundancy
and ever more intelligence at the edge.
R
Fordham U.
Marymount
Tarrytown
Upstate NY
16
14C
NYPH (AP)
esponding
Fordham U.
Tarrytown
15B 15C
16A
15
Yeshiva
(Uptown)
Fordham U.
NYPH (CPC)
Yeshiva U.
14
CUNY
(City College)
8B
Bronx
NYSBC
Columbia
13
New Jersey
NYSERNet’s
Manhattan
Project.
Developing a
Dark Fiber
Network in the
Manhattan area.
12
AMNH
MSKCC
(DC)
NYPH (WMC)
14B
MSKCC (PCB) 11D
11
MSKCC (HQ)
9
MSKCC (64th)
11B
10
14E
CUNY
(CIS)
7A
NYPL (PA)
15A
Fordham (LC)
8
14D NYPH (PFS)
NYSERNet POP
Columbia
Controller
13A
NYPL
7
11C MSKCC (53rd)
6
7B
MSKCC
(Admin)
11A
NYPL (Annex)
Arts4All
8A7C
CUNY (Grad.
Center)
NYPH (DC)
14A
Yeshiva
(Midtown-1)
16C
16D
Fiber
Trunk
Loop 1A
NYPL
(Science/DC)
New School
Rockefeller
NYPH (E61)
NYU (MC)
Yeshiva
(Midtown-2)
5
4 16B Yeshiva (Law)
3
Cooper
Union
2
NYU (ACF)
Polytechnic
1
Brooklyn
M
anhattan Project





Advancing Technology for our Members
NYSERNet as catalyst, project manager, business
manager.
Seeking to serve needs of the entire community for:
redundancy, security, flexibility, performance, costcontrol.
Developing a model for future projects.
Urgency has risen since September 11th.
M
anhattan Project







Goals
Implement a network that provides economical, highperformance access to:
the NYSERNet network,
commodity Internet providers,
the public switched telephone network, plus
the ability to implement private intra- and interinstitutional networks, and
unlimited potential for performance upgrades.
M
anhattan Project









Participating Institutions (initial list)
Amer. Museum of Natural History
Arts4All
Columbia University
Cooper Union
City University of New York
Fordham University
Memorial Sloan Kettering
New School University







New York Polytechnic Institute
New York Presbyterian Hospital
New York Public Library
New York University
The Rockefeller University
Weill Medical College
Yeshiva University
M
anhattan Project

Project Objectives
2001




Survey potential participants about their requirements.
Create network design based upon the survey.
Preliminary costs determination based upon design.
Review costs with participants and secure commitment.
2002

Network and colocation construction commences.
2003

First participant online.
A
ctions at NYU




In negotiations with vendor for second commodity
Internet link with diverse routing within Manhattan
Seeking funding for distributing infrastructure in
multiple locations (e.g., network core, key servers,
etc.)
Working with FEMA to determine appropriate
infrastructure upgrades
Risk assessment and response planning
Washington Square Park – 9/23/2001
Photo by Doug Carlson
NY Harbor – 2/2/2002
Photo by Doug Carlson