When Five 9s Aren't Enough

Visa provides extreme protection for the world's largest payment- processing system.

One of the largest financial systems in the world is hidden in a nondescript building near Washington. The owner, Visa International Inc., hasn't put its name on the building, nor will it allow a reporter to say exactly where it is. The secret data center is a fireproof, earthquakeproof concrete fortress with 5,000-pound doors and a basement full of backup gear, but it has fake windows to make it look like any of hundreds of ordinary office buildings in the area.

Paranoia? Not when you consider the stakes. Five minutes of downtime in Visa's worldwide processing system, called VisaNet, would block $55 million in payment transactions, estimates the San Francisco-area firm.

"There is no such thing as 99.9% reliability; it has to be 100%," says Richard L. Knight, senior vice president for operations at Inovant Inc., the Visa subsidiary that runs its data centers. "Anything less than 100%, and I'm looking for a job." The company has had 98 minutes of downtime in 12 years.

Visa fights the battle against outages and defects on two broad fronts: Its physical processing plant is protected by multiple layers of redundancy and backups, and the company's IT shop has raised software testing to a fine art.

There are more than 1 billion Visa payment cards outstanding around the world, spawning $2 trillion in transactions per year for 23 million merchants and automated teller machines and Visa's 21,000 member financial institutions.

"We run the biggest payments engine in the world," says Sara Garrison, senior vice president for systems development at Visa U.S.A. Inc. in the San Francisco Bay area. "If you took all the traffic on all the stock markets in the world in 24 hours, we do that on a coffee break. And our capacity grows at 20% to 30% year to year, so every three years, our capacity doubles."

Visa has four major processing centers to handle that load, but the Washington facility is the largest, with half of all global payment transactions flowing through the building. It shares U.S. traffic with a center in Silicon Valley, but it can instantly pick up the full U.S. load if the California facility goes down.

Indeed, everything in Visa's processing infrastructure - from entire data centers to computers, individual processors and communications switches - has a backup. Even the backups have backups. For example, the Washington center has four rotating uninterruptible power supply (UPS) units (only three are needed) driven by the local utility and backed up by an array of batteries and four 1-megawatt diesel-powered generators. The 24,000 gallons of diesel fuel stored on-site is enough to power the center for a week. The UPS units protect the center from possible power fluctuations. The facility has enough redundant cooling capacity to air-condition 300 homes.

"Visa understood early on that things like triple redundancy and scalability would be the critical, defining factors in a highly competitive landscape," says Randi Purchia, research director at AMR Research Inc. in Boston. "They realized that they are a technology company; it is their business."

The eight IBM mainframes at the Washington data center are rated collectively at 3,000 MIPS. Altogether, worldwide, 7,000 MIPS of processing power can conduct 10,000 payment-authorization transactions per second. Visa's network, one of the largest private networks in the world, consists of 9 million miles of copper and optical fiber, and every Visa customer has two paths into Visa via commercial carriers.

Every operations area at the data center is equipped with a blue light mounted high on a wall. The lights flash when the San Mateo center is down and the Washington facility has picked up the entire U.S. processing load. The lights are a warning to workers not to take any action that might escalate the outage.

"If the light comes on, everyone gets off the floor," says Anthony LaManna, vice president for operations and network services at Inovant. "They go get a cup of coffee or something."

While all these backups and safeguards contribute to Visa's ultrareliable operations, they're only part of the story. Every summer, well in advance of its year-end peak processing season, Visa runs a full-scale stress test at IBM's $1 billion Performance & Scalability Center in Gaithersburg, Md., where IBM has 14,000 MIPS of processing power. The tests cap months of requirements analysis, modeling and testing at Visa's own facilities.

"We introduce failures at that point as well," says Mike Wolfson, senior vice president of engineering at Inovant. "So while we are processing 5,000 messages a second, we'll knock off a storage controller and make sure the system doesn't skip a beat."

This kind of full-volume testing - which Visa doesn't have the capacity to do in-house - has proved itself, Wolfson says. Several applications that ran flawlessly in production at peak loads failed when the test load was increased to reflect volumes projected for the coming holiday season, he says.

And Visa tests more than the impact of higher volumes at the IBM center. New software is tested as well, says Mike McGraw, vice president of systems engineering at Inovant.

"These [legacy] applications have, for the most part, been written in IBM assembler," he says. "But now, with the use of C and C++, we have to see how that's going to behave. You can do all the modeling in the world, but unless you push it to its limit, you won't find out where things break."

1pixclear.gif
Credit Card Authorization Flow
visadiagram.gif

1.

2.

3.

4.

5.

The cardholder presents the Visa card (credit or debit) at the point of sale.

1pixclear.gif

The merchant uses an electronic terminal or the telephone to request an authorization from the merchant bank.

1pixclear.gif

The merchant bank creates a VisaNet integrated payment authorization request message that includes details about the account and the transaction. The message is then switched through VisaNet to the card issuer.

1pixclear.gif

The issuer reviews the request and makes a decision to approve or decline it.

1pixclear.gif

The issuer’s response is sent back through VisaNet to the merchant in a matter of seconds.

1pixclear.gif

NOTE: In some cases, when an issuer is unavailable for authorization, VisaNet will authorize the transaction as a part of a stand-in processing service. This is done to further enhance payment system efficiency.

Source: Inovant Inc.

1pixclear.gif
Inside the Secret Center

Visa’s Washington-area processing center houses 50 million lines of code for some 300 applications. Major functions include the following:

1pixclear.gif

Authorization system. This online, IBM-mainframe-based system propels a payment card request from a cardholder to a merchant, to the merchant bank, then on to the card issuer and back to the merchant.

1pixclear.gif

• Clearing and settlement system. This mainframe batch system runs nightly and settles accounts among merchants, merchants’ banks and card issuers.

1pixclear.gif

• Fraud-detection system. This online systems runs on Sun Microsystems Inc. servers and uses neural networks and pattern-recognition algorithms to look for fraud in each payment transaction.

1pixclear.gif

• Data warehouse. This mammoth storage facility consists of

18 Storage Technology Corp. silos and a 250,000-volume tape library holding up to seven years’ worth of transaction histories.

It grows by 250TB each month.

1pixclear.gif

Read accompanying stories:

Download the 2018 Best Places to Work in IT special report
  
Shop Tech Products at Amazon