NYSE Euronext boosts network bandwidth with 10GbE throughout
For the stock exchange, the flatter the network, the better
Computerworld - NYSE Euronext has rolled out 10 Gigabit Ethernet technology throughout its data centers, and with the new networking technology the stock exchange boasts 2.4Tbps aggregate bandwidth or a sub-75 microsecond round trip for network messages.
"It's a screamer," said Andrew Bach, the stock exchange's senior vice president of network services. NYSE's previous Gigabit Ethernet network had a total aggregate throughput of 0.5Tbps. "We've been able to get our wire speed up to 10 gig sustained, regardless of packet size, and we've been able to drop our latency significantly.
The stock exchange, which daily executes tens of millions of trades for 1,200 broker-dealer clients, performed the multimillion-dollar upgrade in networks that are part of its existing infrastructure. The 10Gbps Ethernet was also built into two new data centers, one just outside of London and the other in Mahwah, N.J.
The exchange is in the middle of a data center consolidation project that would reduce the number of facilities from eight to four or five, according to Bach.
The 10GbE upgrade was required to handle the increasing amounts of trade messaging traffic, which is staggering. In the U.S. alone, NYSE exchanges process more than 22.4 billion messages representing $70.8 billion in financial transactions every day. The stock exchange said it handles about 7.5 times more traffic than the number of Internet searches that Google handles daily.
The stock exchange also sells market data to broker-dealers, and that represents a large portion of its network traffic.
Using network switches from Juniper, the exchange deployed more than 1,000 Solarflare SFN5122F server adapter cards from Solarflare Communications as well as the vendor's OpenOnload application acceleration software. NYSE Euronext has about 6,000 servers, but many of them have up to six 10GbE ports, Bach said. Only about 15% of server ports are used for inter-switch linking, something the exchange reduced significantly over the past three to four years.
"We're headed for a data center fabric with top-of-rack switching methodology. It's much more scalable on the bandwidth ... lower latency, and more importantly, ever lower jitter," he said. "In short, the flatter the network, the better."
For its main trading platforms, NYSE explored using InfiniBand, which has a maximum quad data rate of 40Gbps, but it is nowhere near as ubiquitous as Ethernet.
"We still use InfiniBand in some areas ... but the vast majority of the installed base is Ethernet. In terms of good ol' market competition, you can select from any number of different vendors on the Ethernet side, while you have a limited selection on the InfiniBand side," Bach said. "You get to play one vendor against the other."
One of the biggest challenges in the new network deployment was responding to microbursts of traffic -- bursts that in a 1 millisecond window would peak at 5Gbps to 9Gbps on a single data feed. It was an exercise in load balancing to address the issue, Bach said. "You trade off pacing the feeds out, but of course the more pacing you do, the more latency you have. So you have to balance those two off carefully. On the receiver side, obviously we used larger buffers ... to absorb it," he said. "It really forces you to make sure all parts of your network can run at 10 gig wire speed end to end."
While the NYSE has yet to incorporate its data storage traffic into its Ethernet network -- it uses the Fibre Channel devices -- consolidating that traffic is a project Bach is considering now that the bandwidth is there.
Unlike many of today's large enterprises, which pride themselves on consolidating their server environments through hypervisors, virtualization has a downside for the NYSE: Sharing compute nodes among applications increases latency, which affects trade executions.
"Virtualization costs you a penalty of a couple of microseconds. That's something I'd rather give back to the customer," Bach said. "So in the core trading applications, we don't virtualize."
Lucas Mearian covers storage, disaster recovery and business continuity, financial services infrastructure and health care IT for Computerworld. Follow Lucas on Twitter at @lucasmearian, or subscribe to Lucas's RSS feed . His email address is email@example.com.
Read more about Financial IT in Computerworld's Financial IT Topic Center.
- 18 Hot IT Certifications for 2014
- CIOs Opting for IT Contractors Over Hiring Full-Time Staff
- 12 Best Free iOS 7 Holiday Shopping Apps
- For CMOs Big Data Can Lead to Big Profits
- Slideshow: 5 ways to lock down your mobile device
- Slideshow: 10 mistakes companies make after a data breach
- How to rob a bank: A social engineering walk through
- Which smartphone is the most secure?
China says to Chiang Kai-shek all Bitcoins at the door.
China marches long and hard over internal financial institutions, proclaiming that Bitcoins "should not and cannot be used as a currency". The news sent Bitcoin exchanges into a dive like cormorants in the deep China Sea. But all is not lost -- or is it? Bitcoins have been bubbling back to the surface, or has China's stance drained all enthusiasm? In IT Blogwatch, fearless leaders and bloggers quote pearls of wisdom from the little red book of Bitcoin. Not to mention: Financial advice from Whiz and Ice...
- IT Certification Study Tips
- Register for this Computerworld Insider Study Tip guide and gain access to hundreds of premium content articles, cheat sheets, product reviews and more.
- Make or Break: New Auto Products Must Go To Market On Time
- This Webcast quantifies the value of time to market for the auto industry and highlights how Primavera Enterprise Portfolio Management can help organizations.
- Stock Shock: The effect of project and portfolio management on share price
- In this independent report, you'll see the intrinsic connection between long-term capital investment and short term market performance -- and how this can...
- Hedge Your Bets
- This report explains how visibility and increased governance is key to reducing risk.
- In the Firing Line
- CEOs Are Increasingly Being Held Accountable; How susceptible is the CEO's reputation to poor performance across the project portfolio?
- The CISO's Guide To Virtualization Security
- This guide describes the security challenges within virtualized environments and shows how to apply the concepts of Forrester's Zero Trust Model of information... All Financial IT White Papers
- Live Webcast Research Report: The Big Data Opportunity for HR and Finance If CEOs, CFOs, CIOs, and CHROs want to drive their businesses forward, they will need to quickly recognize the enormous value of big...
- Live Webcast The Freedom to Run Your Business Your Way Vendors are challenged to create flexible systems that customers can tailor to particular business strategies and industry needs. But the flexibility should not...
- Live Webcast The Business Value of Human Capital Management for Finance View now >>
- HR and Finance Were made for Each Other View now >>
- The Value of Human Capital for Finance Professionals View now >>
- The Business Value of Human Capital Management for Finance View now >>
- The Freedom to Run Your Business Your Way Vendors are challenged to create flexible systems that customers can tailor to particular business strategies and industry needs. But the flexibility should not...
- Research Report: The Big Data Opportunity for HR and Finance If CEOs, CFOs, CIOs, and CHROs want to drive their businesses forward, they will need to quickly recognize the enormous value of big...
- All Financial IT Webcasts
Computerworld's Best Places to Work in IT 2013 list featured Quicken Loans, Securian, Vanguard and other top finance organizations. Honorees say the distinction helps them recruit top talent and boost staff morale.
Want to join this elite group? Nominate your organization for our 2014 list.