Transcript Document
Lessons Learned Response and Recovery in a Disaster Robert DiLossi Director, Crisis Management [email protected] Top 8 lessons from DR Scenarios • • • • • • • • Key players not available or missing Communication issues Lack of transportation/lodging/logistics No plan for extended recoveries Documentation/Scripts out dated (i.e. DDA, Technical Scripts, Employee Contact Lists, etc.) Difficulty accessing tapes/data/documentation Incomplete contracts-any vendor (i.e. Media Storage, Insurance, Fuel, etc.) Lack of content “Business Continuity Plans” 2 Copyright 2006 SunGard Availability Services LP, all rights reserved. Lessons Learned – Comparison Grid 3 Copyright 2006 SunGard Availability Services LP, all rights reserved. People - Employee Issues Encountered Key players missing, safety Not focused on recoveries Insufficient funds Who’s in charge? / No communication Lack of personal necessities Transportation unavailable No lodging / logistics Lack of corporate presence at recovery site (i.e. HR, Corporate Communications, etc.) Include employees / people with plans and tests Address employees families in plan Engage HR, Corporate, and Local Government Use alternate staff / alternate locations Establish Employee safe and travel kits Create cash account with debit cards Travel, lodging agreements near recovery locations 4 Copyright 2006 SunGard Availability Services LP, all rights reserved. Incident and Crisis Communication Plans Internal documentation not current Lack of involvement during planning / testing Business unit requirements not accurate Corporate participation minimal Vendor contacts out dated External agencies not engaged (i.e. Local or Federal Governments, Red Cross, EMS, etc.) Enforce change control / mandatory Develop / test your incident and crisis communication plans Engage all business units earlier / ownership of plan Rally Corporate involvement / include while testing Validate vendors contacts quarterly Engage local agencies (i.e. Fire, Police, Red Cross, EMS, Local Government, etc) 5 Copyright 2006 SunGard Availability Services LP, all rights reserved. Business Continuity / Disaster Recovery • Preparedness • Outdated equipment configurations with vendor contracts • Crisis leader or communication not clearly defined • No plans for extended recoveries • Documentation / technical scripts out of date • No vendor agreements in place • Single point of failures Develop and ‘TEST’ your plans Change control/management Designate leads, single contact number and distribute Develop robust plans for longer stays Update scripts as they change Establish vendor agreements Establish backups for key technical personnel 6 Copyright 2006 SunGard Availability Services LP, all rights reserved. Data Availability • Destroyed Tape, DASD and Paper Documents • Tapes lost during shipment • No access to Tapes, DASD or Paper Documents • Travel Restrictions (curfews, government occupancy) • Recovery scripts outdated / unknown Investigate the need for Electronic media Utilize disk to disk mirroring Use alternate locations for off-site storage vendor Store recovery scripts in electronic format off-site Test scripts and add to your BCP plan Rotate technical staff while testing 7 Copyright 2006 SunGard Availability Services LP, all rights reserved. Travel and Communication Challenges • Air travel unavailable • Highways closed • No rentals available • Vehicles broken down • Mass evacuations enforced earlier • Long delays • Limited fuel available • Disabled vehicles Deploy key team members and families earlier Consider alternate routes Utilize alternate travel means Understand and follow evacuation routes-safety Extra gas on hand or delivery Enter into agreements with fuel providers Utilize local government agencies for support 8 Copyright 2006 SunGard Availability Services LP, all rights reserved. Travel and Communication Challenges Cont…. • Cell towers and land lines down • Press/media statements for public sectors Consider alternate communication tools Extra cell phones and batteries Activate stand-by satellite phones Wireless cards for laptops Establish VPN, text messaging, direct connect Corporate e-mail address Augment staff from alternate business locations Establish external communication plan Update local radio stations with reports Establish company wide Crisis Bridge for updates 9 Copyright 2006 SunGard Availability Services LP, all rights reserved. After the Disaster… • Insurance coverage - unknown • Inventory not current • Relocation Understand your policy and risks Test different disaster scenarios Keep inventory current Utilize independent review of insurance Record all disaster activities Use external security agency to monitor 10 Copyright 2006 SunGard Availability Services LP, all rights reserved. Long term recoveries… • Recovery & Business Continuity plan while at the hotsite • New equipment and circuit procurement Cold-site and New Facility • Location for a new facility • Relocating your employees and their families • New contact lists for employees • Damage assessment report/length of stay • No “return home” content 11 Test scenarios for all levels of recovery Third party to search for new location Utilize vendor for facility plan HR and corporate involvement for employees New facility evacuation and fire routes Longer term housing Possible school registration Update communication plans Copyright 2006 SunGard Availability Services LP, all rights reserved. Key Points to a successful recovery • • • • • • • • • Develop and test plan regularly Implement change management Maintain recovery scripts and procedures Assign and rotate backup roles Execute your plan as documented Familiarize yourself with offsite storage vendor Utilize professional organizations to augment staff Engage corporate and external agencies during testing Test the way you recover, recover the way you test! 12 Copyright 2006 SunGard Availability Services LP, all rights reserved. Philadelphia Crisis Management Center 13 Copyright 2006 SunGard Availability Services LP, all rights reserved. Proactive Calling Map 2003 Northeast Power Outage Mt. St. Helens Eruption N. CA. Earthquake Hurricanes California Fires 14 Copyright 2006 SunGard Availability Services LP, all rights reserved. How to Declare a Disaster 866-722-1313 or 215-351-1313 Appropriate Staff Notified - Call Back Verification Disaster Declaration Authorization List Authorization Password (if applicable) Identify Declaration Needs/Alternatives Site Preparation 16 Copyright 2006 SunGard Availability Services LP, all rights reserved. Alert Process What are the advantages? • • • • • • • No cost to You, our Customer Heightened awareness Review last test and DDA on file Logistics Review contract information Document configurations needed Follow up on every alert until issue is resolved 17 Copyright 2006 SunGard Availability Services LP, all rights reserved. What’s new? DDA Online - The Disaster Declaration Authorization (DDA) Form is a list of individuals who are authorized to declare a disaster for their company. • • • • Directly logon to www.mysungard.com Access and Update DDA 24/7, by person and by schedule Delegate DDA Information access by company, person or schedule Data is update within the hour or final changes Alerts Online - SunGard Customers may open an Alert online • • • • Logon to www.mysungard.com Access and submit online Alerts 24/7 Customer will receive call back on all alerts Still no costs to Customer Hotline calls are now handled by the IOC in Denver. 18 Copyright 2006 SunGard Availability Services LP, all rights reserved. Lessons Learned Response and Recovery in a Disaster. Questions?