Transcript Tropical Storm Allison Recovery

Allison 2001
Tropical Storm Allison
Recovery
Charles Chambers
Information Technology
2001
Allison 2001
Time Line
Sa
9th
Su
10th
M
11th
T
12th
W
13th
Th
14th
F
15th
Sa
16th
Su
17th
M
18th
T
19th
W
20th
Th
21st
F
22n
d
• Saturday
– flooding begins (1 A.M.) – IT team on campus
– Additional IT staff arrive – pumping and power are
critical
– Institutional priorities are focused – Non-IT
• Sunday
–
–
–
–
–
Pumping of network node facilities begin
Power restored to UHS data center
Internet connection reestablished
713-743-1000 call center established
Systems and servers returned to service
Allison 2001
Time Line
Sa
9th
Su
10th
M
11th
T
12th
W
13th
Th
14th
F
15th
Sa
16th
Su
17th
M
18th
T
19th
W
20th
Th
21st
F
22n
d
• Monday, June 11th
– WAN services restored except UHV
– IP telephones activated in critical areas
– Relocated some servers to Data Center
• Tuesday, June 12th
–
–
–
–
–
Internet connectivity unstable due to ISP
Servers in Data Center grow by 20
UHV network connection restored
Desktop assessment team inventories campus
Network teams inspect buildings for damage
Allison 2001
Time Line
Sa
9th
Su
10th
M
11th
T
12th
W
13th
Th
14th
F
15th
Sa
16th
Su
17th
M
18th
T
19th
W
20th
Th
21st
• Wednesday, June 13th
– UH-CVN services restored, classes held.
– Payroll processed
– 50% of phone system returned to service
• Thursday, June 14th
– Data backbone restored
– Data connectivity to opened buildings
– Order Equipment for Storm Solutions for
Hofheinz Pavillion
F
22n
d
Allison 2001
Time Line
Sa
9th
Su
10th
M
11th
T
12th
W
13th
Th
14th
F
15th
Sa
16th
Su
17th
M
18th
T
19th
W
20th
Th
21st
• Friday, June 15th
– Majority of phone system restored
• Saturday, June 16th
– Equipment for Hofheinz Pavillion arrives,
installed
F
22n
d
Allison 2001
Time Line
Sa
9th
Su
10th
M
11th
T
12th
W
13th
Th
14th
F
15th
Sa
16th
Su
17th
M
18th
T
19th
W
20th
Th
21st
F
22n
d
• Sunday, June 17th
– Hofheinz Pavillion available
• 270 IP telephones
• 314 data ports
• IT support booth created
• Monday, June 18th
– University Re-Opens
– First occupants moves into Hofheinz Pavillion
Allison 2001
Time Line
Sa
9th
Su
10th
M
11th
T
12th
W
13th
Th
14th
F
15th
Sa
16th
Su
17th
M
18th
T
19th
W
20th
Th
21st
F
22n
d
• Tuesday
– Voice Mail system restored
– Touch Tone Registration restored
– Remainder of phone system restored
• Wednesday
– Majority of SWB trunking problems corrected
Allison 2001
Time Line
Sa
9th
Su
10th
M
11th
T
12th
W
13th
Th
14th
F
15th
Sa
16th
Su
17th
M
18th
T
19th
W
20th
Th
21st
F
22n
d
• Thursday
• Friday
– Reconfiguration of Hofheinz Pavillion network
Allison 2001
Technology Priorities
Infrastructure
•
•
•
•
•
•
Phone System
Internet Connectivity
Inter-campus Network
Campus Data Network
Interactive Video Network
Data Center Operation
Allison 2001
Technology Priorities
Services
•
•
•
•
•
•
E-mail
Web Site
WebCT
IVR for Registration/Records
Payroll
Financial Application
Allison 2001
Technology Priorities
Desktop Computers
•
•
•
•
Inventory
Repair
Replacement Computers
Data restores
Allison 2001
Technology Priorities
Temporary Space - Holfheinz
• Used basketball pavilion
• Boxes became offices
• Basketball floor
– became labs and cubicles
Allison 2001
Holfheinz Pavillion
From Basketball to Academics
• 27 boxes
• Cisco IP telephony – 270 phones
• Switched data network
– 270 ports in offices
– 144 ports on basketball court
– Over 100,000 feet of copper and fiber
• IT support booth
– Voice and data network support
– Desktop computer support
Allison 2001
Recovery Resources
• Over xxx person hours (UH Staff)
• Loss and Business Resumption
– Costs estimated in millions
• Over 250 IT related staff involved
Allison 2001
Lessons Learned
•
•
•
•
Move IT facilities above ground
Move generators above ground
Flood doors on underground entrances
Established plans helped . . .
– The right people at the right places, making
the right decisions was the key.
• Mobilize support efforts early
– Food
– Sleeping Quarters
Allison 2001
Heroic Efforts
•
•
•
•
•
Pumps/Pumping
Saving the Animals
Getting Primary DID in service
Internet Connectivity
Holfheinz Pavillion
Allison 2001
How the Speedy Recovery
•
•
•
•
•
•
•
•
•
Temporary fixes
Great minds and enormous goals
Dedicated staff
Very long hours
Multiple plans
Quick vendor response
Elimination of bureaucracy
Communication of status and goals
Central coordination point
Allison 2001
Starting Recovery
•
•
•
•
Pumping and Power
Starting pumping PGH on Saturday
Moved pumps to animal lab (SR2)
Sunday power restored to
Computing Center
• Internet Connection
– Reconfigured to computing center
– Web and e-mail up first
Allison 2001
Recovery Partners
• Herculean efforts by all!
–
–
–
–
Pumping water
Removing damaged equipment
Designing and installing new systems
Power restoration
• UH Support
– College volunteers
– Facilities
• Vendor response was great!
– Staff on site (over 100)
– Rapid deployment of equipment
Allison 2001
Initial Impact
• Primary Technology Areas Affected
–
–
–
–
UHS Data Center
Network Node in PGH
Network Node in UCU
Headend Facility in Moody
Allison 2001
UHS Data Center
• Lost power
• No flooding or other damage
Allison 2001
Network Node (PGH)
• Services lost
– Voice, data, video for UH
– Internet and interactive video for all
UHS
• Damage
– Water 18 inches above raised floor
– Power systems total loss
Allison 2001
Network Node (PGH)
Telephone and Voice Mail Systems
Allison 2001
Network Node (UCU)
• Services lost
– Voice network for 1/3 of campus
– Trunk lines from SWB
– High capacity circuits from SWB
• Damage
– Telephone Switch corroded
– Wire center destroyed
• 3600 stations
Allison 2001
Network Node (UCU)
Telephone system
Allison 2001
Network Node (UCU)
SWB
Interface
Allison 2001
Network Node (UCU)
Wire Center Frame
Allison 2001
Headend Facility (Moody)
• Services Lost
– CATV Network
Allison 2001
Hofheinz Pavillion
Home of “Phi Slamma Jama”
Allison 2001
Hofheinz Pavillion
Wiring was pretty open…
Allison 2001
An wide shot view of the inner-city of Houston, Texas
Allison 2001
Flood
waters
invade
the City
with no
Mercy
Allison 2001
Freeway
Entrance
ramp into
downtown
from the
North
Allison 2001
Freeway Exit ramp
Allison 2001
A view of Allen-Parkway
Allison 2001
Raging waters in the Tunnel System
Downtown
Allison 2001
Another view of the Tunnel System
Downtown
Allison 2001
And...Another view of the Tunnel
System Downtown
Allison 2001
Canoeing down HWY 59 near the Compaq Center
Allison 2001
A view of 1 of 3 of the Suspension Bridges
that crosses over HWY 59 near the Compaq
Center
Allison 2001
A view of the Houston’s Channel 11 - KHOU Television Studio
Allison 2001
A view of
Interstate
Highway I-10
Allison 2001
Jet Sking in the flood waters on Interstate
Highway I-10
Allison 2001
A view of a 3-way Intersection located at Allen
Parkway and Sanford Street
Allison 2001
Here’s Who did it!
CTS “Crew” at Storm Allison
Luncheon