Weekend Wasted as Firewall Upgrade Flames

The devil's in the details, Vince finds, as an after-hours infrastructure upgrade goes south

I spent my weekend working with the network team, the Unix systems administrators and some external consultants to try to add a second Internet service provider (ISP) to our infrastructure and upgrade our outer firewall.

Our Internet connections have grown into a critical part of our business processes, with customers in every time zone. There's no good time for an outage. We're even being asked to approve delivery of critical information over the Internet.


This Week's Glossary

Autonomous system numbers (ASN): The Internet comprises groups of IP networks, called autonomous systems (AS), that use exterior routing protocols to communicate among themselves. Each AS has an ASN that identifies it and allows such routing to take place.

Border Gateway Protocol (BGP): An exterior gateway protocol defined in the Network Working Group's request for comments (RFC) 1267 and RFC 1268 for passing routing information.

Internet Control Message Protocol (ICMP): Part of IP, the ICMP is a network routing protocol that allows a gateway or destination host to communicate back to the source host to report errors and other events.

Smurf attack: An ICMP-based denial-of-service attack where a request for a ping reply is sent to the broadcast address on a network. The hacker spoofs the source address as that of the targeted host. Every device on the network then replies to the spoofed address, magnifying the volume of the traffic and overwhelming the host.

Secure Multipurpose Internet Mail Extensions (S/MIME): A specification for secure e-mail, S/MIME adds authentication (using digital signatures) and privacy (using encryption) to e-mail messages.


Written by Bill Burns, this code lets internal hosts "statefully" execute ping and traceroute functions from an internal host to Internet hosts through a Check Point Firewall-1 firewall. The code is written in Check Point's Inspect scripting language.

We can protect the integrity and confidentiality of such files reasonably well using high-grade cryptography, but it's difficult to explain to users that the Internet's availability is beyond our control.

Adding an ISP should decrease the risk of an outage, but it will do little if there's a domain name system or routing problem elsewhere in the Internet.

We like to think big, so we got an autonomous system number and a second provider. Luckily, we went online early, so we have a large IP address space.

Moving from a single provider to multiple providers isn't simple because local systems have to decide which Internet provider to use for each destination and keep updating this information in response to any changes in the Internet.

Fun With Firewalls

With the risk of failure at the ISP reduced, it was also time to replace the single point of failure in the firewall infrastructure, and therefore we had purchased and prepared to deploy Atlanta-based Stonesoft Corp.'s Stonebeat for Check Point Software Technologies Ltd.'s Firewall-1 firewall. Stonebeat lets us recover from a firewall failure by switching over to another one.

The default Check Point rule set is somewhat rigid: If you want to allow your internal staff to be able to extend ping and traceroute functions through the firewall, you have to allow Internet Control Message Protocol replies from the whole Internet.

That sounds reasonable, but it leaves you vulnerable to smurf attacks and the like.

Frustratingly, the firewall is capable of supporting what's called "stateful" inspection of ping and traceroute attempts. This means it can keep track of outgoing requests and allow only the corresponding replies back in. You'd think Check Point would use this as a selling point, but it doesn't enable the function within the firewall's normal code.

A few years ago, your only hope would have been to understand Check Point's Inspect scripting code and write your own fix. However, the Internet is a wonderful place, so you can find such code already written online (see links at right).

While we were checking the fix, we managed without the ability to ping, but we were ready to deploy and double our Internet bandwidth, reduce the chance that a minor failure would disable our Internet connection and become a proper peering network. As a peering point, we become a proper, albeit minor, member of the Internet rather than just an end user.

Or at least we would have been if the network component had worked. When we rolled out the change, the large number of Border Gateway Protocol routes kept overfilling the routers. The Internet has certainly grown. There are now more than 90,000 routes to be stored, and our routers don't have enough memory for that.

A weekend down the drain, a day of the consultants wasted. Of course, we'll have to try it all over again once the routers are upgraded, so another weekend will be sacrificed on the altar of keeping my organization secure.

It would be less annoying if our ISP and Cisco Systems Inc. hadn't told us that the specification would be fine.

In my first column, I explained that I'd been asked to provide a secure e-mail connection to board members at remote companies. Internet e-mail enjoys the same level of protection as a postcard: Anyone involved in delivery can read it, change it or pretend it came from someone else.

Our long-term strategy is to use S/MIME with our Microsoft Exchange Server, but first we need to widely deploy Windows 2000 so that we can store user encryption keys within the Active Directory. In the interim, we deploy point solutions to particular requirements.

There are many ways to allow users to exchange information, protected from prying eyes. Although encrypted e-mail solves a lot of problems, it also introduces other risks.

First, we scan all e-mail within the servers and at the gateway for viruses. If messages are encrypted, they can't be checked. This increases our risk of virus infection. Our network-based intrusion detection system is also blind to attacks within the e-mails.

Encrypted E-Mail Danger

These shrink in comparison with the vulnerabilities that an insider can exploit with encrypted e-mails. They can use the encrypted mail to leak critical information or send abusive e-mails.

If users encrypt all their content and then lose the key, we can't recover the data for them. A malicious user might blackmail us by encrypting our critical data and demanding payment to provide the key. Normally, we'd be able to go to backups, but if all the copies are encrypted, then we're out of luck.

We could store copies of all the keys centrally, but the central store becomes a tempting target for attackers and introduces a high risk that all our keys could be hacked.

There are standards for key exchange and encrypted e-mail, but there are so many to choose from and no obvious manner to agree on them with the organizations with which we need to exchange e-mail. Our board members are already using PGP encryption from Network Associates Inc., so we have to find something that works with that.

Right now, we're looking at London-based GFI Software Ltd.'s Mail Essentials for Exchange/SMTP. It should provide transparent encryption at an infrastructure level. We hope to manually exchange a single key with each organization and then encrypt all user e-mail to that organization. It shouldn't require any change to user desktops, and we won't have to train the users.

But just as we arranged a demonstration for our messaging team, we were given another requirement for secure e-mail. One of our regulators needs to exchange confidential fraud information and has chosen a product that doesn't work with S/MIME or PGP.

So in addition to a standard that we can't deploy until we upgrade to Windows 2000, we'll have two nonstandard e-mail encryption systems spreading through our environment. Does anyone know a way to let S/MIME and PGP translate between each other?

Copyright © 2001 IDG Communications, Inc.

Shop Tech Products at Amazon