This past winter, a worm known as Slammer rattled the Internet violently enough to become what you might call a "CNN-level virus" -- that is, it burrowed its way into the national consciousness.
Nearly everything about the SQL Slammer was old. It was an old hack that exploited a year-old vulnerability found in an old target, Microsoft Corp. software. There was a patch to block Slammer that was six months old, and that patch suffered from an old patch problem: It was so kludgy to install that the patch needed a patch. Above all, the reaction to Slammer -- the call to use the event to build security awareness -- was so old it called Bob Hope "kid."
But this much was new: Everyone agreed that Slammer was your fault.
How to save $60 billion
The old game was to blame Microsoft. "Microsoft did not protect its customers," read a letter to The New York Times after the Melissa virus hit in 1999. A year later, after the I Love You virus infected Microsoft Outlook, a Washington Post editorial stated, "This is a software development problem." The Nimda worm (2001), according to Forrester Research Inc., required 625 combinations of patches applied to Microsoft's Internet Information Server. Nimda, along with its contemporary, the Code Red virus, eventually compelled Microsoft to implement and market Trustworthy Computing, an initiative aimed at helping Microsoft developers learn how to write secure code.
Slammer, though, hasn't followed the old pattern. A developing consensual wisdom suggests that as woeful as Microsoft's products may be, CIOs have been equally sloppy. A February poll of more than 200 IT professionals, by antivirus company Sophos, showed that 64% of respondents blamed their peers' lax security practices for Slammer. Only 24% blamed Microsoft.
The poll also revealed that only 43% of the respondents said they subscribed to Microsoft's vulnerability mailing list, which provides early alerts of viruses in the wild. Twelve percent said they relied on "mainstream news" -- newspapers and TV -- to learn about new viruses. Three percent said they "don't really hear about them at all." And 19% said they patched software when they "got around to it."
"I've got to look around at my comrades and ask, Why aren't you patching your systems?" says Bob Ferderer, vice president of IT internal operations and security at CUNA Mutual Group, the nation's largest financial service provider for credit unions, with 5,000 employees and $9.3 billion in assets. "There's a relationship between individuals not taking action and how these things spread out of control."
What frustrates Ferderer and other security experts is the fact that this seemingly intractable problem is actually quite tractable. The tools and strategies to prevent another Slammer are just waiting to be used. In fact, the number of tools and strategies available to you -- and available at a reasonable cost -- makes it inexcusable for any CIO to fiddle while the software burns.
There is, after all, $60 billion on the table. A 2002 study by the National Institute of Standards and Technology (NIST) developed that number to describe buggy software's cost to the national economy. Improved software testing alone, NIST suggests, could shave $22 billion off that.
Why can't the software community motivate itself to grab all that cash? The answer lies in software culture. Vendors, for the most part, value time-to-market over security. As long as they can get away with shipping buggy code, they will.
Developers live by deadlines, which compel them to work fast. At the same time, they're being asked to provide ever more features.
And CIOs, as a group, have been passive, assuming there was little they could do to effect change.
They assumed wrong. In fact, a growing number of advocates believe CIOs should be leading the charge for secure software.
"CIOs must take action," says Linda Northrop, director of the product-line systems program at the Software Engineering Institute (SEI) and co-author of Software Product Lines: Practices and Patterns. "I think CIOs have done a deplorable job matching their software decisions to business goals, especially in its security and quality. What we need from the CIO ranks are leaders."
Northrop could be talking about Al Schmidt, vice president of IT and CIO of $939 million Arch Chemicals. "What I've come to realize," says Schmidt, "is that security is really about operational excellence. So why wouldn't I jump on that? I mean, operational excellence -- that's what I'm supposed to be doing, right?"
Know your enemy
The vast majority of vulnerabilities in software arise from the following 10 basic development flaws.
Unvalidated parameters: Anyone can change a URL to access data. For example, switching "admin=no" to "admin=yes" within the URL gives the hacker admin privileges.
Broken access control: Software doesn't check a user's authorization properly, which means that credentials are cached and can be co-opted by anyone.
Broken account and session management: Hackers can take advantage of predictable validation schemes by, for example, manufacturing or altering cookies.
Cross-site scripting: User hits a poisoned Web page, which his browser accepts uncritically, allowing the hacker to access his machine.
Buffer overflows: Miscreant is allowed to input so much data that memory gets junked up and starts accepting malicious commands.
Command injection flaws: User's Web application executes malicious commands because it doesn't know any better.
Error-handling problems: Error messages give away information -- for example, "ODBC Error" signals that a SQL injection (such as the SQL Slammer) is possible.
Insecure use of cryptography: Use of homegrown, weak cryptography is easily compromised by professional bad guys.
Remote administration flaws: Web apps allow remote control of machines on a known port with predictable default configurations, making hacking child's play.
- Web and application server misconfiguration: Unused features that leave ports open make servers vulnerable.
SEI studied 2,500 software products and 56,000 security incidents, and found the vast majority of problems are caused by those defects. Buffer overflows alone account for 40% to 60% of incidents, says CEO Steve Cross. What's more, the flaws are, from a coding perspective, simple to fix.
"The thing that frustrates me is people think that since the problem is so prevalent, it must be complex," says Cross. "It's not. If the public understood that any freshman computer science student knows how to fix these problems, there would be an outcry. And there should be."
The trick with code defects is to find them. A million-line software program, which is about the size of the average CAD/CAM package, comprises 20,000 pages of text. Finding errors -- even in poorly written programs that will have two or so errors per page -- is like finding needles in a binary haystack. And, once you do find the vulnerabilities, you have to make sure the fix doesn't break anything else.
This used to provide a reasonable excuse for letting bad software practices continue. But a new class of application scanning software is emerging that makes the process of finding flaws almost trivial.
How to take responsibility for more secure software
Scan everything ...
Application scanning is based on the proposition that programming errors are reasonably predictable and limited, even if their consequences are not.
Once you buy (one large insurance company recently paid $120,000) and then train your developers and IT staff to use the application (which should take anywhere from two days to two weeks), plan to use it in two ways.
First, make app scans a mandated part of all application audits. Whether it's internally developed code or third-party software, create a checkpoint at which an application must contain fewer than a certain number of bugs, with all egregious errors eliminated.
"We've decided to push back," says Ferderer's partner, Tim Burke, IS manager at CUNA Mutual. "Any new software we're developing or having developed must go through an error-free scan before it's deployed."
Second, integrate the application scanners into the development process by training your developers and mandating that outside developers scan their work at predictable intervals. (This was made easier for .Net developers last February when Microsoft integrated the application scanner from Sanctum into Visual Studio .Net.)
Expect developers to freak out about that. "The first time we had one of our developers run the scan on some sample code he wrote, his eyes bugged out," says Erick Weber, vice president of enterprise information security at IndyMac Bank, which holds $9.6 billion in assets and last year earned $600 million. "He thought he wrote pretty good code, but he had created significant security holes.
"But it made him want to learn how to integrate the scanning into his development. That's what we want. The last thing we want is to have developers continue coding the same way, run scans on their work, and then throw the code back at them and say, Hey, fix this."
...But know scanning's limitations
It would be folly to deploy app scanning and think the security problem is therefore licked. Know the limitations of these tools.
First, they don't know what they don't know. That is, they scan for known vulnerabilities, the most common ones. The tools will have to evolve as vulnerabilities evolve.
Second, the tools don't know whether a flaw is dire or benign.
Third, some of these scanning apps just tell you where the holes in the roof are, some others put buckets under the holes, and still others even suggest how to patch the roof. But none of them actually fix anything.
"My concern is people buy them and think that alone fixes the problem," says Bill Guttman, director of the Sustainable Computing Consortium, a collaborative designed to protect the nation's computing infrastructure and improve the reliability of its IT systems. "It happened with antivirus [software]. Firewalls. People develop a false sense of security."
The next generation of application scanners should address some of those issues. Plan for an ongoing investment in application scanning. A divisional IS officer at one of the nation's largest banks (who asked not to be named) says his company will spend $30,000 to $40,000 per quarter scanning and auditing a major application.
IndyMac's Weber has no trouble justifying the cost. As an ancillary benefit, he uses the tools to convince both developers and his executive peers of the value of security. He says security is like pollution; no one cares until they can see it. "I can rattle off really great reasons we need application security to users and management," says Weber. "But they never really understand until they see, you know, eight buffer overflows, right there on their screen."
Pull rank on the development team
Application scanning is a useful tool, but it's also fixing a problem 10,000 times rather than solving it once. If the 10 basic software flaws are so easy to avoid, why don't developers do so?
The answer is that developers have two masters: features and deadlines. Until recently, security was not a feature. And dealing with it usually threatened a deadline. (There's an adage among developers: Speed to market, number of features, level of quality. Pick two.)
"It's why we published the 10 vulnerabilities," says Mark Curphey, chairman of the Open Web Application Security Project (OWASP), an open-source project that develops Web application and Web services tools. "Developers understand the vulnerabilities technically, but they don't change how they code because of them. But when CIOs get the list, right away they understand."
Bring the list to a meeting with the person in charge of application development. Each of the 10 vulnerabilities can be assigned to either the development group or the IT-operations group, or in a few cases, both.
"As CIO, you need to be pretty intolerant of development that doesn't address these basic issues," says Schmidt, the Arch Chemicals CIO who, as a former developer, has come to appreciate the need for a sit-down between the CIO and the head of developers. "Absolutely get talking. And bring some of the numbers on how much you save by catching bugs early. That data is out there."
Give QA some TLC
The development culture that rewards features delivered quickly also scorns quality assurance (QA). Here's how several developers view the code testing piece of QA:
"Thankless."
"Entry-level button pushing."
- "You know your career is advancing when you don't have to do testing anymore."
It's imperative for the CIO and the head of developers to reward developers who don't feel that way.
At CUNA Mutual, Ferderer has recruited top developers and trained them to write secure code. He created a mentor program in which developers bring their code to their quality mentors and together they work on securing it. Ferderer says it's early, but there's anecdotal evidence that code quality is improving.
Make no mistake, forcing developers to get involved with QA puts you at the fulcrum of some wrenching cultural change in a guild whose practices are 30 years entrenched.