Microsoft yesterday announced that each of the three finalists in its $250,000 BlueHat Prize security contest came up with ways to detect and stymie one of the most effective exploit methods now being used by hackers.
The three finalists -- two from the U.S., the other from Croatia -- took different tacks to block return-oriented programming, or ROP, a technique often used to sidestep DEP, or data execution prevention, one of Windows' primary anti-exploit technologies.
"It's an obvious reflection on the most pressing attack vector hitting systems right now," said Andrew Storms, director of security operations at nCircle Security, commenting on the fact that the ROP technique was the subject of each of the finalists' entries.
Microsoft kicked off the BlueHat Prize competition last August as a way to tap into the expertise of top-notch security researchers without setting up a bug bounty program -- a course of action the company has consistently dismissed.
"It seemed to us that to take an approach to block entire classes was the best way to engage with the research community and protect customers," Katie Moussouris, a senior security strategist at Microsoft, said last year during the news conference at which the contest was announced.
The BlueHat Prize competition features a $200,000 award for first place, $50,000 for second place, and a subscription to Microsoft's developer network, valued at $10,000, for third place. The three finalists will be flown to next month's Black Hat security conference in Las Vegas, where Microsoft will reveal the results July 26.
The finalists announced Thursday are Jared DeMott, a security researcher employed by Florida-based Harris Corp., a major defense and aerospace contractor; Ivan Fratric, a researcher at the University of Zagreb in Croatia; and Vasilis Pappas, a Ph.D. student at Columbia University.
All three worked alone -- contradicting earlier speculation that teams might have an advantage in the competition -- and wrapped up their work one to two weeks before the deadline.
And each researcher tackled the same problem -- ROP -- and explained why in much the same way as Storms.
"I focused on ROP because it is the current state-of-the-art in exploit development and a burning issue in exploit prevention," said Fratric in an email reply to questions. "Furthermore, it is a very difficult problem to solve, so it was an interesting challenge."
DeMott echoed Fratric's sentiments. "I targeted ROP because it is currently the most-used technique to exploit fully-compiled software," he said, also via email.
But while DeMott, Fratric and Pappas all attacked ROP, they came up with different solutions.
DeMott, who calls his technology "/ROP" to match the names of Microsoft-made defenses, such as "/GS" and "/NXCompat," said his answer to ROP checks the target address of each return instruction, whether intended or not, and then compares it to a whitelist.
"/ROP is simple to understand and implement [and] it fits the current Microsoft paradigm," said DeMott. "It works with low overhead and finally, /ROP mitigates all known, practical ROP attacks."
Fratric's "ROPGuard" uses a somewhat similar technique to block ROP exploits: His technology also checks each critical function call to determine if it's legitimate.
"Unless [the attacker] wants the attack to stay confined in the current process, [he or she] will need to call some 'special' functions to leverage the attack," said Fratric. "The attacker will need to call these functions from the ROP code, either directly or indirectly, and that makes these functions an ideal place to check if the attack is taking place or not."
Fratric said that ROPGuard could be applied at runtime for any process, even those already running.
Pappas, who called his defense system "kBouncer," took a different approach that involves checking the control path leading to a system call.
"When ROP code is executing, control follows an unconventional path, which makes it easily detectable," said Pappas in an email Thursday. He called kBouncer a "lightweight form of control flow integrity."
Unlike /ROP and ROPGuard, kBouncer relies on a performance-monitoring feature found in newer Intel processors for its efficiency, said Pappas. "Although it does not require any specific hardware, it runs much better on these chips," he said. "Hopefully other CPU vendors will implement that functionality, too."
All three claimed that their solutions would only minimally impact the performance of a Windows PC.
"The effect on memory and the CPU is minimal, about 3% to 4% on average," said DeMott.
Fratric said ROPGuard is even less processor-intensive. "It had an average CPU overhead of just 0.5% in my experiments," he said.
One of the stipulations of the BlueHat Prize competition was that a solution had to have a processor overhead of less than 5%.
Microsoft is expected to update Windows with one or more of the finalists' solutions -- and possibly some of those that didn't make the last cut. Contest participants will retain intellectual property rights to their work, but they must license their technologies to Microsoft on a royalty-free basis.
"I suspect we will see changes in the next Windows 7 service pack," said Storms. "I'd put my money on January [2013]."
Microsoft issued its only service pack for Windows 7 in February 2011. Based on its previous practice, Microsoft will probably ship a second service pack for Windows 7 soon: It delivered SP2 for Windows XP just over three years after that edition's launch, and Vista SP2 two years and three months after Vista's debut.
Although it's possible that Microsoft could squeeze the protections into Windows 8 before the new operating system launches this fall, experts think that's unlikely. The software vendor could, of course, issue an update to Windows 8 after its release to add one or more of the defensive technologies.
The finalists were all optimistic that Microsoft could easily add their code to Windows.
"My guess is that it would be not that difficult," said Pappas. "kBouncer's main idea is straightforward, and its transparency makes it easy to integrate. Even the prototype implementation, which I developed by myself, is already capable of protecting large and complex applications, like Adobe Reader and Internet Explorer."
ROP has been widely used by hackers, sometimes with spectacular results.
The Stuxnet worm, reportedly created by U.S. and Israeli coders to sabotage Iran's nuclear fuel enrichment facilities, used ROP extensively. In late 2010, attack code that exploited IE on Windows 7 went public; the attack exploited ROP vulnerabilities to sidestep Windows DEP and ASLR (address-space layout randomization), the two main anti-exploit defenses in the OS.
Earlier in 2010, a pair of researchers used ROP to hack Safari on Apple's iOS mobile operating system to win $15,000 at the Pwn2Own contest. It was the first time that ROP had successfully been used against a device with an ARM processor.
"Windows users would be safer if /ROP is adopted, because ROP attacks, as they are now, would fail," said DeMott. "Clever attackers will likely move to something more advanced, but that is the cat-and-mouse game we play in security."
Fratric had a similar take. "It will protect users from currently-used exploits. For how long, I can't answer, because as the protection technologies are developed, so are the exploitation techniques," Fratric said. "The protection I proposed is not perfect, but it raises the bar for the attacker and could raise it a bit more if some ideas I proposed are extended a bit."
Storms thought it noteworthy that Microsoft selected ROP defenses as the three finalists for the BlueHat Prize: He said it shows that the company has ROP as a top priority.
However, Storms did wonder why other security threats didn't share the spotlight. "Couldn't they have chosen anything else for at least one of the finalists?" he asked.
DeMott regarded the focus on ROP from a more personal angle: "I did not know that others would also address ROP attacks."
Gregg Keizer covers Microsoft, security issues, Apple, Web browsers and general technology breaking news for Computerworld. Follow Gregg on Twitter at @gkeizer and on Google+, or subscribe to Gregg's RSS feed . His email address is gkeizer@computerworld.com.
See more by Gregg Keizer on Computerworld.com.