How the ANI bug got baked into Vista: Microsoft explains

Computerworld - In a postmortem of last month's Windows animated (.ANI) cursor vulnerability, one of Microsoft Corp.'s security development gurus today spelled out how the bug sneaked into Vista.

Michael Howard, an authority on Microsoft's Security Development Lifecycle (SDL) -- a multipart initiative that aims to get developers to design more secure code -- posted an extensive entry on the brand-new SDL blog that outlined lessons learned from the ANI vulnerability. "SDL is not perfect, nor will it ever be perfect," Howard acknowledged yesterday. "We still have work to do, and this bug shows that."

That bug, which first surfaced late last month and posed enough of a threat that Microsoft went out of cycle to patch it, affected all older editions of Windows as well as the newest, and supposedly more secure, Windows Vista. Some security researchers, in fact, took Microsoft and its SDL process to task for not catching the flawed code as Vista was written, debugged, tested and polished.

Some of those same researchers immediately weighed in on the unusual mea culpa by Microsoft. "This is really out of character," said Jonathan Bitle, the manager of the technical accounts team at Qualys. "Microsoft historically has played security issues much closer to the vest."

Oliver Friedrichs, director of Symantec Corp.'s security response team, was a bit tougher in questioning Microsoft's motives. "They're attempting to be more transparent to explain why this vulnerability was missed. They received a lot of criticism for not catching this earlier and for letting it into Vista, and I think this was one of the only ways for them to explain both to the technical and the management-level communities how they actually missed it."

Specifically, Howard called out flaws that the ANI vulnerability revealed in Vista's security components, as well as in Microsoft's development tools and processes.

The /GS switch, a function of Microsoft Visual Studio's compiler that's designed to protect stack variables from overflows that could result in arbitrary code execution, was one. Some third-party researchers, notably Ollie Whitehouse of Symantec (download PDF), have criticized Microsoft for not /GS compiling all of Vista's binaries. Turns out, however, that in the ANI case, that wasn't the problem.

"Because there are no candidate buffers on the function's stack, there is no /GS cookie added to the stack, even though the code is compiled with /GS," said Howard. "This is not the first time we've seen code with no cookie, and this has made us rethink the heuristics used by the compiler when it determines whether to place a cookie on the stack or not."

Another Vista security feature, Address Space Layout Randomization (ASLR), which is supposed to randomly assign data to memory to make it tougher for attackers to determine the location of critical OS functions, also didn't have the intended impact on the ANI vulnerability. "If the vulnerable code is wrapped in an exception handler that catches many errors [as was the animated cursor code], a failed attempt will not crash the component and the attacker can try again with a different set of addresses," Howard said. David LeBlanc, also of Microsoft and the co-author with Howard of the just-released book Writing Secure Code for Vista, blogged about the danger of using exception handlers on April 3, the same day that Microsoft patched the ANI bug.