A rainy day can have you rethinking your contingency plans

The unprecedented impact of Superstorm Sandy is forcing financial institutions to rethink assumptions about disaster recovery, contingency planning, and the vulnerability of on-premise solutions. And as a result, we may see widespread adoption of cloud-based disaster-recovery solutions in which third party data centers in remote, land-locked locations are relied on to conduct mission-critical technology operations while the client institution itself is handling the disaster.

Before Sandy, many Wall Street industry participants and observers believed, from a capital markets perspective, that electronic-trading technology had evolved beyond the point where it was vulnerable to a storm. They didn’t believe a storm could ever cause exchanges to close.

This belief was based on a combination of factors, including the establishment of remote back-up facilities far from lower Manhattan, the development of “all-electronic trading” contingency plans established by the major exchanges, as well as the gradual decentralization of trading activity away from the NYSE to other venues. The assumption was that stock trading could continue as usual as long as individuals were able to remotely access the necessary systems.

Yet despite this certainty, both the NYSE and NASDAQ closed for two days in October -- the first such weather-related closure since the 19th century. What the major exchanges weren’t prepared for was the level of devastation across the entire New York metropolitan area (including large parts of New Jersey and Connecticut) -- precisely those areas where almost all exchange employees live, and where the Wall Street firms they interact with are located. It wasn’t simply that employees couldn’t get to lower Manhattan because of storm-related transportation disruptions; it was also that they couldn’t access electronic trading systems due to storm-related utility and communications failures.

In their rush to prove -- after the experiences of 9/11 -- that trading didn’t have to screech to a halt in the event of a disaster, the exchanges simply did not recognize the potential for both the human element and climate change to significantly impact their operations.

Sure, electronic-trading technology is a definite (if imperfect) advance, but it is important to recognize that every trade involves multiple parties; if the process is not 100 percent automated for all of them, human interaction is required, by definition. In addition, there is always some level of management oversight required -- whether for handling exceptions or monitoring traffic.

So when a storm slams the world’s largest financial center, and its human operators lose both physical and digital access to critical systems (or are simply unavailable as they deal with personal survival), all the automation in the world isn't going to make a difference.

Now contrast the Wall Street debacle with how things played out for the major retail banks. Because these banks’ operations and data centers are not concentrated in lower Manhattan, only the physical branches actually sitting in Sandy’s path suffered significant disruption. Transaction processing as a whole was not affected; banks were able to communicate with customers through a variety of channels; and some banks were able to quickly provide mobile branch and ATM services to customers who desperately needed access to cash.

The lesson? It’s time for institutions to step back and clearly think through not only their own contingency plans, but also how those contingency plans will impact end customers and partners. Cloud-based contingency solutions certainly provide some benefit by offering institutions remote data centers for mission-critical operations, but this is only one part of the equation. Even taking into account the human element of how your own employees access back-up systems isn’t enough. Every financial transaction involves at least two (and usually more) parties. If your customers or partners aren’t available to initiate or respond to transactions, no amount of investment in contingency solutions will make the difference.

We were lucky this time -- the two-day disruption in the markets failed to cause systemic issues of major economic consequence. Next time, it could be a different story.

Fortunately, we don’t have to rely on luck. We can, in fact, take luck out of the equation altogether by abandoning our current -- and in all likelihood, more expensive -- approach to disaster recovery. We can investigate the potential of faster, less expensive recovery options now available with cloud-based disaster-recovery solutions. We can strive to make our technology truly invulnerable to a hurricane or other disaster. Most importantly, we can remember always to include the human element in our analysis and planning, account for the vital importance of customer interactions, recognize the critical need for management oversight of exceptions and overall financial flows, and strengthen the weaknesses that automation alone simply can’t address.

Computerworld's IT Salary Survey 2017 results
Shop Tech Products at Amazon