Why you should sweat the small stuff

It's never a good night for the IT department when the first person to get hit by a new virus is the CEO.

That's exactly what happened when the W32.Blaster Internet worm slipped onto the notebook of ABM Industries Inc. chief Henrik Slipsager. Slipsager was booting up during a business trip in Los Angeles in August 2003 when the error message that defined the Blaster popped up, paralyzing his machine and millions of others across the globe. The CEO began calling cell phones of top IT staffers in San Francisco looking for help.

"It was 5:30 on a Wednesday," recalls Sean Finley, assistant vice president and deputy director of electronic services at ABM, a $2.3 billion company that provides janitorial, lighting and security services to high-rise buildings. Finley, a 15-year veteran of the company, says he called an ABM Web site administrator in Los Angeles. "I said: 'Listen, you've got to do me a big favor,'" he recalls. Slipsager left his notebook with a hotel bellhop as the employee raced there with antivirus software. The CEO's computer was fixed. But after that night, the way ABM dealt with viruses changed.

Instead of putting out fires, ABM's IT group moved to set up policies that mandate how employees use antivirus software. One user mandate: No network log-on without the latest virus update download.

After disasters like the 2003 blackout in the Northeast and the devastation of 9/11, you'd think CIOs would be wearing hard hats and duck boots to the office. After all, they've been training -- prodded by worried CEOs and boards of directors -- to prepare for the catastrophic: floods, earthquakes, power outages, even terrorist attacks. Not surprising, IT spending on disaster recovery by global financial services companies after 9/11 spiked 19.2% to $3.4 billion -- up from sleepier 3% to 5% annual increases throughout the 1990s, according to Tower Group. Although spending dipped by 6.4% in 2003, businesses are still shelling out unprecedented amounts of their IT budgets on security. An estimated 5.4% in 2003 went to bulk up security compared with 3.1% in 2001, according to Gartner Inc..

Of course, the annoying headaches an IT staff tackles every day might seem insignificant when stacked up against natural disasters. But to the average company, they aren't. The total effect of spam, viruses, software upgrades and other niggling problems is a plague that cost U.S. businesses billions last year. Most CIOs know this. They realize that the real threat isn't Armageddon; it's being nibbled to death by ducks.

"The majority of our time is spent on the little things that prevent the big things from happening," says Dan Yee, CIO of the California Independent System Operator Corp. (the nonprofit organization that manages the state's power grid established to prevent electricity shortages and blackouts). Yee says focusing on the "little things" means, for example, splitting end users into different classes (like executives and other workers), and using automated tools to monitor what software gets onto their PCs in an effort to head off problems before they occur.

CIOs could be excused for delegating these nuisance issues to their staff. It makes sense to divide and conquer, to quash each snafu as it comes up. Many IT executives interviewed for this story continue to follow that approach.

But it's also not hard to see that CIOs who fail to treat these nuisances holistically, as a class of problems that deserve management's attention and a plan of attack, do so at their peril.

Spam, for one, cost corporations $10 billion in 2003, according to Ferris Research Inc. Look at viruses: Computer Economics Inc. estimates that in 2003 the endless parade of 7,064 new viruses, worms and Trojan horses cost companies more than $13 billion. Even seemingly benign problems like employee password changes add up. These requests account for up to half the help desk calls in a given year and cost a company about $38 per annoying reset, according to Gartner. Add password updates to never-ending nuisances such as the employee who never deletes a single e-mail in 10 years or the PC user who crashes his computer during massive MP3 downloading, and the road leads to one all-encompassing term that could use its own army: nuisance management.

The good news is that CIOs have plenty of weapons in their utility belts to fend off many of these recurring problems. Ideas as simple as enforcing a better written policy for e-mail and banning certain kinds of instant messaging applications from the company's desktops can make a big difference. Ultimately, dealing with nuisances is about being proactive and learning from mistakes. The problems might never go away, but they can be controlled.

Engage every nuisance, but avoid Big Brother

CIOs often walk a tightrope: Trusting employees is important. The staff shouldn't be forced to play Big Brother, censoring every software download or Web site visit. But trusting too much can lead to big budget trouble.

Most any tech administrator knows that the sneakiest network bandwidth stealers are often music and video file-sharing programs such as Audiogalaxy, Kazaa, LimeWire, Morpheus and NeoModus's Direct Connect. MP3 files, at only 3MB to 5MB per song, might seem trivial, until 100 people download dozens of them simultaneously. Universities that cap bandwidth use are finding that MP3 downloads can hog up to 40% of network bandwidth at peak times. And it's not just kids doing it. A May 2003 Jupiter Research Inc. survey of 2,835 consumers found that 12.3% of all 18- to 24-year-olds (compared with 4.5% in all age groups in the survey) regularly download MP3s at work.

Tools like Packeteer Inc.'s PacketShaper or Allot Communication Ltd.'s NetReality detect when fat files are causing network slowdown. They examine packets as they move from the local to the wide area network and classify hundreds of applications. Companies can use the data collected to set policies -- for example, allotting half of all bandwidth to Oracle Corp. applications and just a small percentage for specific file-sharing applications. Other tools made by vendors including Blue Coat Systems Inc., SurfControl PLC or Websense Inc. help filter unwanted applications. Evident Software Inc. takes bandwidth nuisance management a step further: It lets corporations track in dollars which corporate departments consume the most bandwidth. Then it's up to the company to decide whether departments will be charged accordingly for their usage.

If charging bandwidth hogs doesn't work, CIOs can always place a bandwidth cap on users who take more than their fair share.

LandAmerica Financial Group Inc. did this after analyzing bandwidth use. The real estate title insurance company has more than 700 offices in the U.S. that access the Internet through data centers in Richmond, Va., and Dallas. LandAmerica initially set out to use Packeteer's network appliance to improve performance of its network, which often crawled because of peer-to-peer applications or if a worker simply opened a 20MB FTP file. Congestion took its toll on critical applications such as e-mail. To remedy the problem, LandAmerica set a 100K bandwidth limit to weed out heavy use of file-sharing applications -- like Gnutella and Kazaa -- and file-sharing on instant chat. "People can use whatever they want up to 100K," says Matt Matin, a systems engineer. LandAmerica figures it's avoiding $500,000 in bandwidth upgrade costs by using Packeteer for application filtering and data compression.

Others handle the bandwidth problem differently. At Oklahoma State University, Michael White, the university's interim director of telecommunications, uses NAT (network address translation) to deter file-sharing. NAT lets him set up network nodes so that many end users share few IP addresses; 750 kids in a dorm might share six IP addresses, for example. That way, the outside machine seeking to copy files can't easily contact an individual machine in the dorm. However, White says a lot of the peer-to-peer software is able to query the network "super node" to find a single user. He concludes the best antidote is educating students to set their computers so that they aren't open for file-sharing 24/7. "Most students just want to download music," not share all their computer files, he says.

Instant chat campaign

Instant messaging might not hog as much network space as multiple Lord of the Rings downloads, but it can pose problems. Aside from bandwidth issues, many managers find it hard to track the panoply of IM software versions on user PCs. (Microsoft Corp., Time Warner's America Online Inc. unit, Yahoo Inc., IBM Corp.'s Lotus division, Sun Microsystems Inc. and Oracle all make a corporate version of IM.) Just 26% of organizations have standardized on a common corporate IM application, according to market researchers at The Radicati Group Inc.

Yet IM software is now installed within 90% of all corporate networks, according to research firm Osterman Research. Often it's used by employees to get real work done. But some CIOs view it as a bandwidth-sucking productivity blaster.

"It's a huge problem," says Richard Ortiz, IT manager at Palace Entertainment Inc., which runs water and amusement parks. Ortiz says he kept noticing strange spikes in traffic on his frame relay routers last year. So he used network reports to hunt down the culprit. It was IM. "The guys are worse than the girls," says Ortiz. "They play poker. They're talking to their friends about the football game." In October 2003, Ortiz ended the fun, installing Akonix Systems Inc. software, which, like similar products including SurfControl's Instant Message Filter, blocks IM use and helps stop end users from downloading pirated software and peer-to-peer file-sharing.

Akonix works by grabbing packets related to the application and blocking them from leaving the network. It also tells Ortiz who is trying to do what. "If Mary Jo in New York is downloading illegal software from Kazaa, it runs a report. She gets a (pop up) message that says what she's trying to do isn't company policy and that it will be reported to a manager," Ortiz says. The reports are working. During the first week of using Akonix, 60 people received warning notices advising them that IM was no longer allowed. "Now we barely have 10 or five" offenders per week, he says.

Operation auto respond

For the worst nuisances -- e-mail maintenance, antivirus updates and server software upgrades -- companies are finding that automation works by saving time and labor. For Ron Rose, CIO of Priceline.com Inc., the biggest headache used to be the hands-on part of software upgrades. Priceline's business, which allows Web users to haggle the prices they pay for airfare, hotel rooms and other services, is powered by a farm of 300 Microsoft Windows servers that require between 100 and 200 software changes each per month, Rose says. "Before, we had a team of six people applying application updates on a machine-by-machine basis to each of the servers," he says. "It would take up to an hour to deploy the software to a small group of the servers manually -- every time we had to do an update."

Now, Rose uses BladeLogic to consistently deploy software upgrades to not only Windows servers but to the company's Sun Microsystems servers too. Others providing similar data-center automation offerings include IBM's Tivoli, CenterRun (acquired in 2003 by Sun Microsystems), Moonlight Systems and Opsware.

Rose figures the technology has made a 50% increase in the efficiency of technicians doing software loading by eliminating all the hours they once spent manually loading software onto servers and debugging machines that were misconfigured during that manual process.

Allies in the spam war

Other nuisances -- like getting employees to delete e-mail regularly or to quit saving 25 versions of the same Excel spreadsheets on their hard drives -- are harder to tackle. But for spam and virus management, more CIOs are looking to outsource the headache.

Spam now makes up about 60% of all messages pouring into corporate e-mail boxes. But it's not so much the spam that strikes fear in the heart of CIOs, it's the potential viruses lurking within the unwanted messages, says Andy Toner, a partner at PricewaterhouseCoopers who advises clients on security policy.

The average company shells out at least $2.5 million a year to deal with spam -- when you add up productivity lost, bandwidth and storage consumption and support costs, according to server software maker NetIQ Corp. Remedies range from simple whitelists and blacklists for filtering approved and disapproved mail, to software that analyzes the algorithms used to write e-mails. Many CIOs are using an army of tools to tackle spam, including CipherTrust Inc.'s IronMail, Brightmail Inc.'s Anti-Spam Enterprise Edition and Postini Inc.'s Perimeter Manager.

1 2 Page 1
Page 1 of 2
7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon