In today's business environment, companies large and small are looking for ways to secure the applications they create.
Most start by reviewing their internally developed software, but many don't realize that underlying operating systems and other third-party components must also be validated to ensure application security.
This article will discuss two approaches for securing critical software infrastructures -- white-box testing and black-box testing -- and describe the expertise needed and challenges involved with doing each effectively.
White-box testing: Looking under the covers
White-box testing, sometimes called "crystal box" testing, involves looking at a system's code to find bugs. Most organizations use a team that reviews code in internal applications. Sometimes this team falls under the category of quality assurance and isn't necessarily focused exclusively on security. Sometimes the team is in the IT group and is more security-focused. The team is usually composed of highly skilled engineers who review code manually and with automated tools.
Two common issues with white-box testing are the following:
- Manual reviews: Everyone has days where they are less effective, and manual reviews require the reviewer's skills to be at their peak at all times. When reviewing code all day, it can quickly become tedious, leading to burnout and reduced effectiveness.
- Source code: Source code generally isn't available for all of the components that make up an enterprise application.
Unfortunately, the expertise required for auditing binary code is even rarer than the expertise required for testing source code. Maintaining a component-reuse repository and reusing binary-only components that have already been reviewed can help by avoiding duplication of the review work. Another option involves using automated binary-code analysis tools.
The major advantages of the binary-code analysis approach include:
- The ability to integrate code analysis into standard automated build processes and procedures.
- The improved ability to cover all produced code within an organization by reducing the dependency on a scarce human resource -- the security analysis team.
- Increased assurance that the analyzed code is the actual code deployed into production -- eliminating the risk of analyzed source-code modifications (and the potential for introduced security exposures) after the security review is completed.
Black-box testing: Making the application run the gauntlet
In most organizations, black-box testing is done by the same security team that does the white-box code analysis, but sometimes it's done by the quality assurance team or the application development team.
Rather than looking at the code (inside the box), the security team pokes at the box from the outside, disregarding any knowledge of the application's internal workings. This dual perspective is essential, which is why most QA departments focus on black-box user interface (UI) testing. The reason that both perspectives are required is that one can be blind to obvious UI problems or how to test for those problems if one has knowledge of the internal code. It can become difficult to think about the software from the user's experience or about the overall state of the application being developed if you are writing code line by line every day.
Likewise, the team developing a Web service or thin-client GUI-based application can look past security exposures that they have introduced. Even the best binary-analysis tools and human review teams aren't going to catch every exposure, especially those that involve intricate data flows through multiple components.
The 80/20 rule applies here, with white-box analysis identifying 80%, or the "low-hanging fruit" (and dramatically reducing deployment and maintenance costs in turn), and black-box testing picking up the remaining 20% of problems.
Black-box testing also has obstacles, however, such as:
- Code coverage: This is the most crucial obstacle to black-box testing. After all, if security teams don't exercise code in the application, they can't observe any bugs it might have.
- Fault defection: This step, which involves issues such as how to determine that the testing has triggered a problem, is also a large obstacle to effective black-box testing. As most C/C++ programmers can attest, just because a user overflows a buffer doesn't mean that the program will crash immediately. If Murphy's Law has anything to say about it, the application will crash or otherwise misbehave weeks later, when all the clues to what went wrong are long gone.
- Scalability: This is the smallest issue with black-box testing, but it's worth noting. Black-box testing over the network generally involves recording network packets sent to -- and sometimes from -- an application, manipulating them (sometimes called "fuzzing" or "fault injecting") and observing whether the application handles the manipulated data. Systematically manipulating the data for maximum coverage, like a bit-walk, isn't practical, of course. But even trying to be practical can take an impractical amount of time, since sending manipulated packets faster might just overload the target's IP stack or network drivers rather than test the application.
One is good, two are better
So, which is better, white-box or black-box testing? The reality is that each set of techniques finds certain bugs more easily than the other, so they are highly complementary and should both be used to fully test proprietary and third-party software.
In my experience, however, it's best to start with white-box testing, since it usually gets the low-hanging fruit out of the way, and there is generally more low-hanging fruit than people realize. However, use of both white-box and black-box methodologies by developers, QA and IT engineers is needed to ensure that applications are secure -- protecting companies and consumers alike.
Matt Hargett is the director of development for LogicLibrary Inc.'s BugScan Division. The Pittsburgh-based company is a software and services vendor that helps companies develop better applications and integrate them faster.