Trusted Operating Systems: The Ultimate Defense

These operating system add-ons are powerful but complex. Use with caution.

Charles Kalko is a big fan of trusted operating systems. With PitBull from Argus Systems Group Inc., he's used them to lock down vital functions that run his business-to-business barter site. The result: tighter security and more stable systems because fewer information technology administrators can make ill-advised tweaks to them.

But trusted operating systems are "nuclear bombs," says Kalko, a senior security engineer at Inc. in Redwood Shores, Calif. "They drop in and they solve a lot of problems very quickly, and they also create some of their own. If you don't know what you're doing, it could make your life miserable." Which means that IT managers should use them only when the benefit is worth the cost in training and management time.

Trusted operating systems are special versions of off-the-shelf operating systems, such as Windows NT and Unix, that are enhanced to be more secure. An IT manager might use a trusted version of Windows NT on a Web server that contains or is linked to sensitive corporate information. But beware: Trusted operating systems are usually harder to learn and administer than standard versions.

For example, because a trusted operating system can seal applications into unbreakable "compartments," one system administrator might think an application has crashed when in fact he just isn't authorized to monitor it. And because they split administrative power among many people, support staffs need to coordinate more than they have in the past. "I scratch my head every day," says Kalko, trying to figure out, "What's going on here?"


Trusted Operating System Products and Pricing

Argus Systems Group Inc.

Savoy, Ill.

(217) 355-6308

Product: Trusted operating systems, which are enhancements to Sun Solaris, IBM's AIX and Linux.

Price: Runs from $5,000 for a trusted operating system running on a single-processor Web server to nearly $50,000 for an enterprise-level implementation.

Computer Associates International Inc.

Islandia, N.Y.

(631) 348-1789

Product: eTrust Access Control, which can be used to harden Windows NT and various Unix variants. It can also control access to files, operation of critical application processes and access to network services.

Price: Servers start at $4,000.

Hewlett-Packard Co.

Palo Alto, Calif.

(650) 857-1501

Products: Virtual Vault( trusted version of HP-UX), which runs only on HP hardware, and HP Praesidium WebEnforcer, which is a tool for continuously monitoring and repairing Windows NT security vulnerabilities.

Price: Virtual Vault starts at $15,000; WebEnforcer costs $3,000 per server.

Qiave Technologies Corp.

(recently acquired by WatchGuard Technologies Inc.)

Waltham, Mass.

(781) 788-8199

Products: QSecure Enterprise Suite for Windows NT, Windows 2000 and Sun Solaris blocks any changes while the operating system is in operational mode; changes while the system is in administrative mode are allowed only after an exhaustive authentication process.

Pricing: Servers start at $1,295.


New World Order

What's going on is a fundamental change in how companies protect their applications and data. Today's Web economy demands that companies keep applications deep within their corporate infrastructure secure from hackers while keeping those same applications available to customers. Competition also requires corporations to bring e-commerce systems online quickly, even if some components have known security bugs.

"You have to know which systems are critical for the business," says Chuck Ryan, director of information security at Molex Inc., an electronics manufacturer in Lisle, Ill. "You can't secure everything today."

Operating systems, especially on servers, can be weak points because of the fundamental role they play controlling basic functions such as how data is organized into files, written to disks or displayed on-screen.

Any off-the-shelf operating system can be made more secure, or "hardened," with simple procedures such as changing the administrator's password from the easy-to-guess "password" or turning off connections to the Web when they're not being used. But these common-sense fixes can be time-consuming and may not protect a critical server from a determined hacker.

A truly trusted operating system is born, built from the ground up with security in mind. IT managers should look for the following three things in a trusted operating system, says Paul McNabb, chief technology officer at Savoy, Ill.-based Argus Systems:

• A mandatory access-control policy. Consider the simple matter of creating and sharing a file - just fine if you're a legitimate user, possibly deadly if you're a hacker. "If you get into NT or Unix, the OS is not going to tell you if you can e-mail or share" that file, says McNabb. But using a mandatory access-control policy such as the one in PitBull, "you can configure the system in advance to say . . . this user can never get access to, or give away access to, certain resources," such as a file.

• An administration and privilege capability, which an administrator can use to control or eliminate the ability of a user or application that manages the system, or part of the system. "On a trusted OS, you can set up a program which does not have the capability to ever administer the system, even if that program should somehow be totally controlled by an attacker," says McNabb. This prevents a hacker who enters a system through one application from, say, disabling the password that protects other applications.

• Evaluation by an independent laboratory, such as the National Institute of Standards and Technology and the National Security Agency under the National Information Assurance Partnership in Gaithersburg, Md. (

By these criteria, most commonly used operating systems such as Microsoft Corp.'s Windows NT and Windows 2000, as well as the various flavors of Unix, aren't trusted systems, although Windows 2000 took an important step forward with its "system file protection," which safeguards some crucial components.

Trusted operating systems from major vendors such as Sun Microsystems Inc. and Hewlett-Packard Co. have been around for a long time, but they have had a poor reputation for being hard to manage, lagging behind their commercial counterparts in key features. They were also incompatible with applications that their less-secure counterparts could run, says John Pescatore, an analyst at Gartner Group Inc. in Stamford, Conn.

They were largely confined to high-risk environments in organizations such as banks and governments that could afford the staffs to manage them.

Newer versions of those tools, such as HP's Virtual Vault (a secure version of HP-UX) and PitBull (which enhances the security features of Sun's Solaris, IBM's AIX and Windows NT) are easier to use, says Pescatore, but are still more expensive than their off-the-shelf counterparts. Still, the need for cost-effective and trusted operating systems is growing as more corporate systems are linked to the outside world.

Not only do operating systems ship with too many vulnerabilities, says Ryan, but many applications also add security holes as they install themselves. Customers are "finding hundreds, if not thousands" of vulnerabilities, he says, ranging from weak password protection to user accounts or file structures that are "wide open" to hackers.

Compartmentalizing Security

Most trusted operating systems split the services they offer (such as file, print or network access) into compartments, or "sandboxes," and allow only certain end users, administrators or applications into those areas.

To be sure only genuine administrators can make such changes, trusted operating systems may require administrators to authenticate themselves using both a password and a secure ID card, and to enter the system only from certain host machines or network addresses, says McNabb.

Limiting the ability to make changes helped limit what Kalko calls "system drift" - undocumented changes to system configurations that not only open security holes but also make the systems less stable.

But creating these multiple levels of control can be confusing. Splintering the power to administer the system and to access the root directory (which allows access to all other directories and files) required 10 days of training for each of the 10 people on Kalko's administrative staff.

"It's not your standard systems administrator view where he can do anything he wants," says Kalko. "It [requires] a lot of communication within the team on who can do what, when, how."

Pulling the Trigger

QSecure from Waltham, Mass.-based Qiave Technologies Corp. (recently acquired by Seattle-based WatchGuard Technologies Inc.) locks down vulnerable portions of servers while they are in operational mode and provides a console for managing security across the network. In operational mode, even an authorized system administrator can't take actions that would compromise the system, says founder and CEO Jack Danahy.

QSecure also uses a "239-bit elliptical curve" encryption to transmit requests to the operating system kernel. "Every time you want to access one of your files on the file system, on your own box, first you have to reauthenticate yourself into the file system," Danahy says. As for ease of use, he claims a basic installation for an NT server requires only "five mouse clicks [and to] type your password twice."


Hacker Attacks on Nontrusted vs. Trusted Operating Systems

A. In an attack against a conventional operating system, the hacker steals, guesses or decodes the administrator's password. B. Posing as the administrator, the hacker is free to create, delete or e-mail files or directories and to open any application on the server to more attacks.

A. During an attack on a trusted operating system, the hacker steals, guesses or decodes the administrator's password. B. But despite appearing to be the system administrator, the hacker can't tinker with operating system features that have been locked down during operation.


Along a similar line, the current version of HP's Virtual Vault divides operating system functions into only four compartments "rather than separate every process into different compartments, [making] it hard to use," says Gary Sevounts, director of marketing, products and services at HP's Internet Security Division in Cupertino, Calif.

This fall, Sevounts says, HP plans an even easier-to-use product called Web Proxy that will have fewer configuration options than the current version but will be easier to use as a secure front end to many popular Web servers.

It was such ease-of-use features that were most important for several IT managers.

"Since we're global, we need to be able to administer the software, potentially, from a centralized place," says Ryan. He says he also wants reports that tell him which vulnerabilities are most important, not just a list of '500 things wrong the system.' "You can't give that back to the support people and say 'Fix this.' " Finally, Ryan says he wants tools that work across NT, Unix and perhaps even NetWare without needing specialized staffs to monitor each platform.

Carl Tianen, director of global IT security at oil-services company Halliburton Co. in Dallas, says he was nervous about the cost of supporting a trusted operating system. "Look at Windows NT and the effort required to administer an NT system. You start adding layers on top of that, and it could become very difficult," he says.

For such reasons, Pescatore suggests using secure operating systems mainly on servers that conduct financial transactions over the Web, and then only if a corporate security group is available to help system administrators support them.

Trusted operating systems become crucial, says McNabb, "when you have different types of people, different classes of users, on the same system, or you have different classes of networks attached to the same machine."

McNabb says examples include servers that are linked to both the Web and to internal systems, systems administering public-key infrastructure encryption systems and servers running firewalls.

"On a front-end Web server, they're pretty crucial," says Kalko. "I wouldn't use them anywhere else."

Scheier is a freelance writer in Boylston, Mass.

Copyright © 2000 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon