In 1987, I was an IS manager working for a bond house employing more than 300 people in Bloomington, Minn. Our computer systems were Wang 2200s, very old systems at that time. On a Friday night in July, a massive rainstorm hit the Twin Cities area and the poor design of the building we were in became evident.
The entire parking ramp and open parking area sloped to the building. When the sewers could no longer handle the volume of water, they kept backing up until the water reached the level of the parking areas. The entire basement of the building was flooded, including our form storage, media storage, power and phones. All of our computer systems were fine on the 22nd floor, but we had no electricity or forms for them.
Luckily this happened on a Friday night, and we had Saturday and Sunday to prepare for what we were going to do Monday. Here was the plan:
Determine if we could put a generator on the roof to operate ventilation and power PCs.
Try to locate a warm site where we could bring our Wang 2200s, terminals and a few PCs and where we would have a phone connection to our California office to do data entry.
Transfer our phone numbers to our branch office in St. Paul that was still functional.
Set up our accounting functions in St. Paul.
Determine if our forms company had our forms or if they could quickly make more.
Have all employees meet at our Bloomington site to discuss the situation.
- Explain that we couldn't sell securities until the computers were operational, per Securities and Exchange Commission rules (48-hour turnaround on tickets). We were the only major bond house that couldn't move our inventory during the outage.
I found a commercial real estate agent who had open office space with a small computer room. We had a former phone technician on our payroll who was able to reroute a phone line from another tenant of the building to use with our California office. (The "rental" of that line wasn't cheap. The other company charged us a multiple of what they paid for the line -- I don't remember the figure -- to cover their loss of use.) Our forms supplier did have some forms that could tide us over until more could be created.
The local fire marshal and building inspector nixed our plans to either fly a generator to the roof or snake cables up the side of the building for 22 floors. Our building wouldn't be usable to us for another two weeks, and the computers wouldn't be returned from the warm site to our building until one month after the disaster.
The next problem was that there were no spare Wang 2200s immediately available. We had to hand-carry the 20- to 100-lb. equipment down 22 floors (44 flights of stairs) amid high heat and humidity and then drive the parts to our warm site. Then the local Wang support company could rebuild the drives that didn't survive being bumped going down the stairs and we could reload them from our last backup that didn't get to the now submerged vault. Though the vault remained relatively dry, there was no way to reach it until the water receded.
Once everything was rebuilt and back online, we could update our data and start selling securities again. We were able to re-establish our positions by Thursday and start selling securities. It was a very tough time. I worked more than 30 hours straight myself and had to be sent home, as I was no longer making sense!
The lessons I learned during this time:
You might live to see a 300-year flood.
Business-interruption insurance is mandatory. In 1987 dollars for this small company, IT alone burned an additional $10,000 in one week.
Take time to plan your strategy and have multiple options available. You don't know which ones will pan out. For example, we were able to locate the real estate agent, but the fire marshal and the building inspector rejected the generator plan.
Have your people work in shifts, and don't let them burn out.
Having a company account at a good local grocery allowed managers to run in and get food and drinks for the employees without generating a massive expense nightmare after the emergency.
Don't discount food delivery services like pizzas to keep employees going.
You need cell phones, possibly from multiple vendors, to keep in contact.
Be creative. Leasing equipment and services from other companies works in a disaster. Just knock on doors.
Find what assets the company still has functioning and transfer what you can to them while the disaster site is brought online. Phones are especially important because customers are worried about what's going on.
Maybe you can't get all the equipment you need, but you may be able to move other equipment and repair it.
Know what you legally must restore to make sales. Restoring sales is the No. 1 priority, because you'll need cash flow to survive.
Disasters are very expensive and usually happen without notice. You may not be as lucky as we were to have two full days to plan before you need to respond.
Remember that you may be the only one of your competitors to be affected. Good public relations is critical to avoid losing market share.
Keep spare forms in another location or at your supplier in case your forms storage gets destroyed.
- I've lived through one of these disasters in my 21-year career. That was enough!
Columbus is the owner of Columbus Consulting Group in New Hope, Minn.
Preparing For The Worst
Stories in this report:
- Editor's Note: Preparing For The Worst
- Disaster Homework
- Rising From Disaster
- Classic Mistakes
- Get In Sync With Suppliers
- A Dose of Reality
- The Almanac: Disaster Recovery
- Opinion: Hold Outsourcers To High Standards
- Shark Tank: Disaster Recovery
- Data Recovery Planning: The First Step
- A Business Continuity Checklist
- One IT Manager's Story of Recovery After a Disaster
- Survivor's Guide: 10 Disaster Recovery Tips
- Calculating the Cost of Downtime