Computerworld - One of the largest financial systems in the world is hidden in a nondescript building near Washington. The owner, Visa International Inc., hasn't put its name on the building, nor will it allow a reporter to say exactly where it is. The secret data center is a fireproof, earthquakeproof concrete fortress with 5,000-pound doors and a basement full of backup gear, but it has fake windows to make it look like any of hundreds of ordinary office buildings in the area.
Paranoia? Not when you consider the stakes. Five minutes of downtime in Visa's worldwide processing system, called VisaNet, would block $55 million in payment transactions, estimates the San Francisco-area firm.
"There is no such thing as 99.9% reliability; it has to be 100%," says Richard L. Knight, senior vice president for operations at Inovant Inc., the Visa subsidiary that runs its data centers. "Anything less than 100%, and I'm looking for a job." The company has had 98 minutes of downtime in 12 years.
Visa fights the battle against outages and defects on two broad fronts: Its physical processing plant is protected by multiple layers of redundancy and backups, and the company's IT shop has raised software testing to a fine art.
There are more than 1 billion Visa payment cards outstanding around the world, spawning $2 trillion in transactions per year for 23 million merchants and automated teller machines and Visa's 21,000 member financial institutions.
"We run the biggest payments engine in the world," says Sara Garrison, senior vice president for systems development at Visa U.S.A. Inc. in the San Francisco Bay area. "If you took all the traffic on all the stock markets in the world in 24 hours, we do that on a coffee break. And our capacity grows at 20% to 30% year to year, so every three years, our capacity doubles."
Visa has four major processing centers to handle that load, but the Washington facility is the largest, with half of all global payment transactions flowing through the building. It shares U.S. traffic with a center in Silicon Valley, but it can instantly pick up the full U.S. load if the California facility goes down.
Indeed, everything in Visa's processing infrastructure - from entire data centers to computers, individual processors and communications switches - has a backup. Even the backups have backups. For example, the Washington center has four rotating uninterruptible power supply (UPS) units (only three are needed) driven by the local utility and backed up by an array of batteries and four 1-megawatt diesel-powered generators. The 24,000 gallons of diesel fuel stored on-site is enough to power the center for a week. The UPS units protect the center from possible power fluctuations. The facility has enough redundant cooling capacity to air-condition 300 homes.
"Visa understood early on that things like triple redundancy and scalability would be the critical, defining factors in a highly competitive landscape," says Randi Purchia, research director at AMR Research Inc. in Boston. "They realized that they are a technology company; it is their business."
The eight IBM mainframes at the Washington data center are rated collectively at 3,000 MIPS. Altogether, worldwide, 7,000 MIPS of processing power can conduct 10,000 payment-authorization transactions per second. Visa's network, one of the largest private networks in the world, consists of 9 million miles of copper and optical fiber, and every Visa customer has two paths into Visa via commercial carriers.
Every operations area at the data center is equipped with a blue light mounted high on a wall. The lights flash when the San Mateo center is down and the Washington facility has picked up the entire U.S. processing load. The lights are a warning to workers not to take any action that might escalate the outage.
"If the light comes on, everyone gets off the floor," says Anthony LaManna, vice president for operations and network services at Inovant. "They go get a cup of coffee or something."
While all these backups and safeguards contribute to Visa's ultrareliable operations, they're only part of the story. Every summer, well in advance of its year-end peak processing season, Visa runs a full-scale stress test at IBM's $1 billion Performance & Scalability Center in Gaithersburg, Md., where IBM has 14,000 MIPS of processing power. The tests cap months of requirements analysis, modeling and testing at Visa's own facilities.
"We introduce failures at that point as well," says Mike Wolfson, senior vice president of engineering at Inovant. "So while we are processing 5,000 messages a second, we'll knock off a storage controller and make sure the system doesn't skip a beat."
This kind of full-volume testing - which Visa doesn't have the capacity to do in-house - has proved itself, Wolfson says. Several applications that ran flawlessly in production at peak loads failed when the test load was increased to reflect volumes projected for the coming holiday season, he says.
And Visa tests more than the impact of higher volumes at the IBM center. New software is tested as well, says Mike McGraw, vice president of systems engineering at Inovant.
"These [legacy] applications have, for the most part, been written in IBM assembler," he says. "But now, with the use of C and C++, we have to see how that's going to behave. You can do all the modeling in the world, but unless you push it to its limit, you won't find out where things break."
Free Insider Guide