These last few years seem to have been wrought with natural disaster. Here in Austin, Texas, where I live, we've been ravaged by wildfires the last few weeks. The East Coast has recently been hit hard by hurricanes and flooding, and last year, California was hit with torrential rains and earthquakes. If you aren't doing adequate disaster recovery plans and designs, let this serve as proof that you should be.
Disaster recovery, or disaster preparedness as we used to call it back when I was in the military, is a subject that not many of us like to talk about. This sort of planning and the resiliency that these concerns require can significantly increase our network design in terms of both complexity and cost. System redundancy takes on a different meaning when you think in terms of disaster recovery.
I was recently at Fort Bliss in El Paso, Texas, working with the U.S. Army, and it got me to thinking about some of the disaster recovery work I did when I was active duty in the U.S. Air Force. Many times during an exercise we would simulate a building suddenly "gone". You can imagine lots of military scenarios that would lead to that, but it just as well could've been a natural disaster. How would you react if your boss walked in tomorrow, randomly picked a building and said, "This building is now gone as far as you're concerned. It's vanished from the face of the Earth. Explain the problems this causes us and how we're going to recover." How would you react? What if that site was one of your major network hubs or primary data centers?
One thing I've learned about disaster recovery planning is that it's significantly cheaper to build in plans for disaster up front than it is trying to retrofit them later. Most vendors offer redundancy solutions for disaster recovery, and they're more than happy to help you with planning because this means more revenue for them. Whether you're thinking about your network management tools or your company's web properties, most technology solutions today have fail over technologies built into them.
In terms of best practices, here are five tips to keep in mind when doing disaster recovery planning:
- Assess how your organization would function during a disaster. Would it run at half speed, full speed, or 2x?
- Document which systems will be most important during a disaster scenario and invest accordingly.
- Plan from the core outward - meaning infrastructure first.
- Be thorough. Much like with IPv6 planning, don't forget support systems like trouble ticketing and IT system monitoring.
- It's not just about the technology. In most disaster type situations you'll also be missing people. Include personnel availability scenarios within your planning.
Do you have a specific tip for disaster recovery planning or a horror story to share? What would you add?
Flame on...
Josh
Josh Stephens is Head Geek and VP of Technology at SolarWinds, an IT management software company based in Austin, Texas. He shares network management best practices on SolarWinds GeekSpeak and thwack. Follow Josh on Twitter@sw_headgeek and SolarWinds @solarwinds_inc.