Real life: The ghost in the network

Computerworld - This is the true story of a ghost server -- a phantom server that nearly brought down our network. Paranormal? Perhaps not. Simple common sense and a little low tech delivered what fancy equipment couldn't see.

Our current network evolved from massive mainframes to early Windows machines, Unix boxes, dumb terminals, NT and now thousands of smart clients and rooms of powerful dedicated servers. We monitor every facet of their operation. Yet a phantom nearly took us down.

The onset was not ominous. A network printer overran its buffer, and jobs stopped printing. We speculated that it was a hardware failure, swapped out the old printer, configured the new one, rebooted the device, and then it was business as usual.

Then it happened again on another printer. Soon a master console winked out. Just for a moment, but it was definitely gone. Users reported sporadic data header corruption. Strange incidents became more frequent with no set pattern.

A computer virus on a server? A network worm on a shared drive? Not likely. We use a multilevel security approach guarding against blended threats.

NetOps caught some of the signals. They were random but definitely originated from inside the perimeter. We had a phantom server.

TIP: Document what comes in and what goes out

Some of our very old servers had been in operation for many years. They plodded along doing the mundane maintenance tasks for which they were originally assigned. New hardware came and went. Many of the older devices could no longer be traced except by anecdotal memory.

TIP: Scan your entire IP range periodically

Most modern operating systems monitor a wide range of activity. But beware! They cannot detect what they cannot see. Very old hardware can lie below the radar. Our phantom server with no name was undetectable by normal means.

Program consoles can display their clients, but not every machine runs every service. We suddenly realized that we did not have a single, simple comprehensive method to detect everything that was out there on our network, no matter what it was.

The humble ping command came to the rescue. It detects connectivity and lets you capture IP addresses and machine names as it traverses your network.

TIP: Use an informative standardized naming convention

Cute server names may be amusing, but cute names can be problematic when trying to locate a specific item in a hurry. Encoding the location and functionality into the device name saves both time and aggravation. You should label every server with a tag bearing its name and IP address in a conspicuous location when it enters service. This technique may be low-tech, but it saves valuable time when trying to find a box amongst its colleagues.

TIP: Read your logs