Hurricanes, floods, fiber cuts keep IT pros on their toes

Many companies are prepared for a disaster, with plans and technologies in place to help sustain operations in the wake of severe weather, hardware failures, human error, and accidents. But not every company is covered. In a survey conducted by Network World and IT management software maker SolarWinds, 25% of respondents admitted their companies don't have a disaster preparedness and response plan in place. Among those without a plan, 57% said they'll create one in the next 12 months.

PAYDAY: Who are tech's highest paid CEOs?

There's good reason to make that happen: Among the 230 IT pros who responded to our survey, 27% said they've been unable to go into the office because of a disaster of some kind. Of those, 34% missed a week or more of work because of the incident.

"During a severe storm an electrical surge caused our UPS to shut down, taking down our data center," recalled one survey participant. "After this we added a whole building surge suppression system."

Another company lost connectivity in its office for a week after a backhoe cut through the building's phone lines and fiber. The company quickly moved its servers and set up shop in an employee's home. "We are a software developer, [so we] were able to bring up our web site at an employee's house within half a day. Clients didn't know we had a problem," the IT pro said, but it was a "big headache for the network guys, cleaning, buying, rebuilding."

Another respondent cited multiple outages at his company. In one instance, a server room was exposed to smoke from a fire and "all the equipment had to be removed and cleaned, stopping the business for a week." Another time, a flood crippled the business -- 1,500 people across 50 sites -- for nearly three days because voice, WAN and Internet were all routed through a single fiber connection to the carrier's exchange, which was powered down due to flooding.

Despite the horror stories, it can be difficult for IT pros to convince senior management of the risks of insufficient business continuity planning. When asked about the attitude of senior non-IT management toward disaster preparedness and response investments, just 31% of survey respondents said it's a top or high priority. Another 29% said it's a medium priority, and 40% said it's a low priority or management is unconcerned.

Sometimes, it takes a disaster to get the message across.

In one instance, corporate finance nixed IT's plans to deploy a backup AC system during data center construction. When temperatures in the data center soared past 90 degrees, it resulted in a shutdown of nearly all server resources. The result? A "new, dedicated backup AC installed [and] connected to the building's backup generator," the respondent said.

"Flooding can be an issue during hurricanes, nor'easters and other similar events," said another IT pro whose company is located in a low-lying coastal area. "We justified our demands that the servers not get shoved in the basement at our main site when it was flooded out during [hurricane] Irene. Nothing learned the hard way, thankfully."

Disaster readiness

Disaster preparedness is a moving target; plans require testing, updates and modernizations on a regular basis. In our survey, 44% of respondents said they update their plans about once a year, and another 15% do updates every two years.

Some companies test plans even more frequently. "All foundation applications are housed off-site with campus network presence. Tapes are sent off daily for other systems and can be recovered within 12 to 18 hours," one respondent said. "We test this twice a year."

In general, respondents have deployed a wide range of technologies that play a role in their disaster preparedness plans, including VPNs, offsite backup/storage, remote access technologies, and company-supported mobile devices.

CEO FIRST JOBS: Licorice maker, housekeeper, scuba diver and more

In particular, virtualizing applications and putting apps in the cloud as a disaster preparedness measure are common tactics among respondents. Email/messaging infrastructure is most often virtualized or put in the cloud (cited by 58% of respondents), followed by web site/ecommerce systems (48%), database servers (46%), Web servers for critical online applications (44%), and CRM/ERP systems (32%). Twenty percent of respondents said none of their apps are virtualized or in the cloud.

On the data-protection front, nearly one-quarter of respondents (23%) said 100% of their company data is backed up at an offsite location at least 50 miles away from the main office. At the other extreme, 23% of respondents said none of their data is backed up offsite. The remainder fall somewhere in between, with the majority backing up at least 40% of their data offsite.

If a disaster were to strike, respondents have varying opinions about how well their organizations would fare. When asked if they're confident their organization could recover in a reasonable amount of time (12 to 18 hours) if a significant disaster were to make their main data center inaccessible, 45% said they're certain or very confident, 25% are somewhat confident, and 30% said they're not at all confident.

"I'm confident we could restore our data and servers. Not so sure about getting hardware shipped and loaded within 12 to 18 hours. I'd say at least a week," one respondent estimated.

Some are fully aware of the limitations in their current business continuity plans.

"It'd take 140+ hours to restore a full backup from tape," one respondent said.

"We have way too much data backed up on tape -- restoration would take days if not weeks," another echoed.

"The biggest problem will be DNS settings. While there are technical 'best practices,' we can't afford many of them, either because of license costs or recurring bandwidth charges," acknowledged another respondent.

Experience with data center disasters, however painful, can sometimes pay off in the future.

"Given that we have already seen one incident, the office has adopted cautious and diligent attention to prevention and management of similar incidents in the future," said one IT pro. "Back-up facilities have been upgraded and a new data center has been constructed."

Ann Bednarz covers IT careers, outsourcing and Internet culture for Network World. Follow Ann on Twitter at @annbednarz and check out her blog, Occupational Hazards. Her e-mail address is

Read more about infrastructure management in Network World's Infrastructure Management section.

Join the CSO newsletter!

Error: Please check your email address.
Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Ann Bednarz

Latest Videos

  • 150x50

    CSO Webinar: Will your data protection strategy be enough when disaster strikes?

    Speakers: - Paul O’Connor, Engagement leader - Performance Audit Group, Victorian Auditor-General’s Office (VAGO) - Nigel Phair, Managing Director, Centre for Internet Safety - Joshua Stenhouse, Technical Evangelist, Zerto - Anthony Caruana, CSO MC & Moderator

    Play Video

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

More videos

Blog Posts

Market Place