Slow down the security patch cycle

May 3, 2004 (Computerworld)

There are many myths surrounding computer network security that are counterproductive to finding a true solution to the problem. One of these is the belief that vendors should speed up the process of producing and releasing patches for security vulnerabilities that have been discovered by security researchers. Instead, we need a completely different solution to the patch management problem, and part of the solution involves slowing down, not speeding up, patch releases.
Slow them down? What about hackers taking advantage of the vulnerability in the meantime? What about those "zero-day" exploits? To answer this, we need to know how the researcher/patcher/exploiter cycle really works and the motivations of each party in the cycle. This cycle is where researchers discover vulnerabilities, software companies patch the vulnerabilities and hackers exploit the vulnerabilities.
First, let's define zero-day exploit. An exploit is a method devised to take advantage of a specific software vulnerability using a software virus, Trojan horse or worm. When the exploit is done without a virus, Trojan or worm, it's using an undocumented feature. The zero-day type of exploit is discovered, not as part of the security research process, but when an active exploit is using a vulnerability the software developer was previously unaware of. Many different groups at that point rush to reverse-engineer the exploit to document the vulnerability. Antivirus vendors compete to be first to announce a method to detect and fix the exploit, and the software vendor must devise and release a patch immediately to combat the exploit.
By far, the most common type of exploit is the buffer overflow, and software vendors are spending millions of dollars to find and prevent these types of vulnerabilities. These vulnerabilities still exist -- they are getting fewer in number, however, and finding them is now much more difficult. Part of my consulting practice to software vendors and their major customers is finding and reporting these types of vulnerabilities. Where I used to be able to do the "find vulnerabilities blindfolded with one arm tied behind my back" routine, I now actually have to work to find them in major software products.

Researchers are motivated to find these remaining vulnerabilities either by a desire to make a name in the software security industry ("Discoverer of the wack-a-mole vulnerability in Linsoft 2004") or, as in my case, because they are getting paid by the vendor or major customers for this research. Independent researchers will report their findings to CERT, and the vendor and researchers will report the vulnerability to the vendor and the client. Working from a black-box approach, this work is painstakingly detailed and involves searching and reading through millions of lines of disassembled software code. Even when you find a buffer overflow the vendor has missed, it may not lead directly to an exploitable vulnerability.
The easy way to find a vulnerability
Code exploiters (some would call them hackers) know that this is long, detailed, potentially thankless work. Their goal is not to discover if or where a flaw exists. Their goal is to create software that takes advantage of a flaw that already exists. They know there is a much easier way to determine the details of a particular vulnerability than slogging through millions of lines of assembly. The code exploiters use the same tactics against the software vendors that the software vendors and antivirus companies use on them. They wait until the patch for the vulnerability is released, then they reverse-engineer the patch. This is orders of magnitude easier than finding the vulnerability directly.
For example, the patch for the SQL Slammer worm was released six months before the exploit was launched. This long delay enabled blame to be placed on lax systems administrators for not properly patching systems. This entirely misses the point. It enabled some to place the blame on Microsoft Corp. or allowed critics to claim the superiority of some other system that supposedly doesn't need patches. Again this entirely misses the point. Code-exploit developers are becoming much more sophisticated about using the reverse-engineering method to quickly develop exploits. Recent developments show that code exploiters can reverse-engineer a patch in hours, not days or months.

Despite their detractors, patch management isn't only a Microsoft problem. Every system, including Linux and proprietary Unix systems that are built with C/C++, are faced with the buffer overflow security issue. Speeding up release of patches, without improving the underlying process, won't help; it only gives the code exploiters the information they need more quickly. Ultimately, more systems will be developed using managed code (for example, Java and C#). This will narrow the problem to the bootstrap code those systems rely on without every application developer needing to be hypervigilant about buffer overflows.
Until managed applications become the norm, however, we need a better process for distributing patches. The Witty worm, released March 19, provides a good example of why this is true. The Witty worm affects products produced by Internet Security Systems Inc. (ISS). It exploits a vulnerability in the Protocol Analysis Module software component used across the ISS product line. Affected software products included the BlackIce firewall products and RealSecure security products (see story).
There are several interesting things to note about this worm attack. First, these are not Microsoft products. All vendors developing with C/C++ are subject to this type of vulnerability, even nonvendor open-source products. Second, the users affected by the Witty worm were security-conscious and had installed software designed to increase their system security. They were attacked through the very systems they had deployed to make them less vulnerable. So the "I'm safe, I have a firewall" approach is insufficient; in this case, the firewall was the victim. Third, the worm has a destructive payload; it randomly erases sectors on the affected system's hard drive.
Lastly, and most importantly, once the patch was released, the exploit was released the very next day. This wasn't a coincidence where the exploiters just missed having a zero-day exploit. If the patch had been released a week earlier, the worm also would have come out a week earlier.
The patch had the specific information embedded in it that the exploiters needed, and the exploiters already had the expertise and tools required to rapidly make use of the information. The fact that the users of these products are much more likely to patch their software immediately made it critical for the exploiter to act quickly to achieve the effect he desired.
According to the Cooperative Association for Internet Data Analysis, once the worm was released, it had infected its peak number of hosts within 45 minutes. At that point, the number of infections began to decline as the destructive payload began to take the affected systems down. If you had not installed the patch from ISS within 24 hours of its release, odds were your system was infected and damaged. It is clear that the tools required to create an exploit are more sophisticated. This speeds the process and allows those with little or no expertise to create these exploits. These are script kiddies on steroids.
Reworking the patch process
ISS was very responsive in developing and releasing the patch. EEye Digital Security notified ISS of the vulnerability on March 8, and the patch was released within 10 days. The rapid release of the worm in conjunction with the release of the patch and subsequent infection of target systems demonstrates that the patch process itself must be overhauled. For the foreseeable future, we will continue to have software that requires vulnerability patching. End users are in a dilemma, however, because the current method of deploying patches doesn't allow them enough time before an exploit based on reverse-engineering of the patch can be deployed. The problem will have to be addressed by vendors.

The solution will be complex because of the different environments a system may live in, including relative degrees of isolation from the Internet. In one possible scenario, software owners would subscribe to an automated patch service. Those without a subscription would receive the patch through current means, but it would expose those users to greater risk. Subscribers would receive a predeployed, encrypted version of the patch. At a predetermined point, a decryption key would be passed to a patch installer on all subscribed systems. Since the size of the key is small compared to the patch itself, the key distribution could be considered nearly instantaneous, and all affected systems would have the patch installed simultaneously with the official release of the patch. By the time the code exploiter even begins to reverse-engineer the patch, most affected systems would be immune.
This is just one possible solution. The main idea is that vendors need to rethink the patch distribution process, slow it down rather than speed it up and deliver security patches in a way designed to defeat the reverse-engineering process.

Care to comment? Join a discussion.

Bill Addington is a software entrepreneur and inventor, operating several application service provider companies. His spinoff technologies have been sold to companies including America Online Inc. and Microsoft Corp. He conducts his security business in conjunction with Dyonyx in Houston.