Stupid user tricks 4: IT horror never ends

1 2 Page 2
Page 2 of 2

"Then came the phone bill, which I think had quadrupled for that project," he adds. "And the security audit bill, since the data our Web dev guys were working with was quite a king-size waffle of personal customer data. And the hardware/services bill for moving our data out of the outsource outfit's internal datacenter -- which as far as we could tell was four servers in a closet somewhere -- and into a professional data hosting facility in Europe."

If that weren't enough, the final product -- a redesigned site -- "looked so average it might as well have been beige."

"Just a vanilla template with shoddy JavaScript and Perl behind it," Aubrey says.

As it turned out, the new CIO had outsourced not just development, but project management and QA as well.

"There was literally nobody on our side proofing the work. They just kept showing her screenshots and she kept approving them until the day the redesign flipped," recalls Aubrey.

The volume of customer complaints about the site's new look and lack of functionality was put to a stop by the site itself, which crashed twice on the first day.

"The CEO ordered her to pull the plug and go back to the previous design," Aubrey says. "When they added it all up, she'd spent about 75 percent of the original project budget and had nothing to show for it."

"Normally, we'd have just snickered as they walked her out the door, but this was a down economy and this crap just cost us about five months of competitive advantage," he says.

The company never recovered. And though our intrepid offshorer was the first out the door, the rest of the crew followed by year's end.

"I'm not saying outsourcing doesn't work," Aubrey says. "But it takes a hell of a lot more planning than just comparing staff costs."

Solution: Go back and reread No. 3, and then realize that this submitter didn't go far enough. Web site development doesn't have to be isolated to be cheap.

Moral: If the Web site is a key revenue stream, do not entrust site development to a single exec.

Stupid user trick No. 7: Duct tape doesn't fix everything"This one still makes us laugh over beers," says H. Foreman, an admin at a Midwest-based organization.

"We were growing pretty well in 2004 and 2005, so we opened an office across the street," Foreman says. To connect the two offices, they decided to buy two microwave bridges.

"The setup is easy enough that we were able to do the job ourselves, though we had professional carpenters install the bridges to the walls of each building, just under the roof, pointing through double-paned office glass, so we would have no weather worries."

Success carried over into 2006, when the company decided to extend its leases.

"As part of the deal, they get permission to put up a fancy sign near the top of both buildings -- indoors but facing outward through the windows," Foreman says. "The day the sign goes up, our network goes down for about 15 minutes. We're still doing the basic set of troubleshooting diagnostics when it suddenly comes back up. Our guy shrugs, verifies everything again, and lets it go."

The next morning wasn't as forgiving. The network went down and stayed down.

"The basic software diagnostics aren't working, so we go to physical link monitoring," Foreman says. "Pretty quick, we see that one of the bridges isn't responding anymore. Upstairs we go."

Apparently the bridges had been in the way of the signs.

"The outfit that put up the signs just detached the bridges and moved them -- outside," Foreman says. "There was a balcony on the upper floor and they just moved both bridges out there and then duct-taped both of them to the railings.

"What kills us is that the network somehow recovered the first time," he says. "The duct tape across the street held, but the one on our side slipped off during the night and the bridge fell eight stories, bounced off the dumpster, and landed behind it. The sign installers apparently left a note explaining what they'd done with the receptionist across the street and she hadn't passed it on."

Naturally, Foreman and company had fun pointing the finger at the sign company in front of the CEO, who then ran out to chew out the install rep.

"But as soon as he left the room, the CIO, who is a really good ex-tech, pointed out that if we knew someone was going to be doing construction around a critical piece of network infrastructure, why the hell hadn't we gone up there to check it? Especially after the network went down during the construction process," Foreman says. "He had a point."

Solution: The basic network monitoring software this company was evidently using is as good a technology solution as you need in this instance. Without such software, however, this would have been a much nastier adventure.

Moral: What the CIO said. Construction around network infrastructure requires personal attention from your IT staff. Remote monitoring is no substitute for "eyes-on" during critical times.

Stupid user trick No. 8: Executive privilegeThis one hits close to home, as some tech magazine editors epitomize the worst kind of user an IT admin can encounter: those who have read so much about IT that they simply assume hands-on expertise. What results are "special requests" of IT not unlike those we find dealing with higher-up execs.

Let me set the stage: I was working as a technical editor for an IT magazine some years back and happened to be in the executive editor's office three days in a row when this little drama went down. I can't remember whether Windows 95 or Windows 98 had just come out, but it was one of those two. The executive editor had requested the new OS on his honking Toshiba notebook -- a $6,000 box, the price of which I still can't fathom. IT had obliged and installed it. He'd happily used it for a day, taken the box home, and when he returned the next morning it was dead. Windows wouldn't boot. The conversation went something like:

IT tech: "So what did you do?"

Executive editor: "Nothing, it just didn't reboot."

IT tech: "It couldn't have just stopped for no reason. Did you install something?"

Executive editor: "No. Really. It just wouldn't reboot."

IT tech: [sigh] "OK. Fine. I'll fix it."

The next day, the tech returned the notebook, Win 95/98 fully reinstalled and working fine. The day goes well; no crashes. The next morning, the executive editor returns yet again with a $6,000 paperweight. I'm in his office for this part and had to work hard not to shoot coffee out my nose.

IT tech: "Come on, you had to have done something. Everything was working yesterday!"

Executive editor: "No, really. I didn't install a thing. I was just working and organizing."

IT tech [suspicious]: "What do you mean 'organizing'?"

Executive editor: "You know, just arranging folders so that I can find things more easily."

IT tech [still suspicious]: And which folders were you 'organizing'?"

Executive editor [annoyed]: "What does that matter?"

IT tech [equally annoyed]: "Trust me. Which ones?"

Executive editor: "My personal folder, the issue folders, the system folder --"

IT tech [squeezing his eyes shut]: "What did you do in the system folder?"

Executive editor [slowly dawning]: "Uh, well it was so messy. They had one folder for 16-bit DLLs and another for 32-bit DLLs, so I thought it'd be more efficient if they were all, you know, in a single folder."

I'm not sure who the tech wanted to kill more: the executive editor for what he did or me for sitting there, shoulders shaking, beet red, with my mouth clenched shut and tears coming out of my eyes.

Solution: Don't let your users become case studies for denying administrative access to local machines. Deny them administrative access to begin with. With senior execs, however, it still takes a social engineering degree to keep that rule enforced. That's a line of patter every IT guy needs to develop.

Moral: Even Microsoft computers don't suddenly quit for no reason. There's always a guilty user somewhere on the chain of causality. Find him early and you can avoid a large load of trouble down the line.

Stupid user trick No. 9: User populations are like bacterial ecosystems from distant planetsThis particular stupid user trick hails from my days as an IT consultant, when our clients' CIO types, who had read about Shadow Copy, immediately wanted to engage on it. After all, in many cases they'd paid for it already, so they wanted it up and running right away.

Rolling out Shadow Copy was easy -- once we had Windows 2000 on every desktop and a working Active Directory domain controller. Then I used my vaunted writing skills to pen a short and sweet "Shadow Copy Advisory" memo and e-mailed it to every user. We followed that up with personal visits to all the managers in the company, explaining how the feature worked and what they needed to tell their employees about it.

The upshot was that My Docs was now being shadow-copied for every user, so all those folders they had on their desktops should be moved to My Docs to make sure everything got backed up to the network automatically.

In retrospect, I might as well have been asking them to bite off their own fingers for my amusement.

Everyone nodded excitedly, but nobody had any intention of using it.

To be fair, this was our fault as much as theirs. Assuming that users will put data exactly where they say they will is a newb mistake. But like true consulting newbs, we set up a backup policy to perform daily backups of "data" folders -- the shadow-copied stuff and the file shares users said they were going to be using -- and weekly snaps of the full server. Desktop backups relied entirely on users making My Docs their sole data dump.

Naturally, when a nasty virus hit and took out a large percentage of the desktops and simultaneously dumped two out of three servers, we found only 8 percent of users had been taking advantage of Shadow Copy. The rest were simply screwed. Worse, we found out that they had decided to build new "informal" network shares right off the server's hard disk (exactly where we hadn't expected them to), so those files were lost, too.

Solution: First, realize that you will never get away from users using their desktops as data storage. Ever. That's why it's called the "desktop." Whatever desktop backup strategy you employ, it needs to cover the desktop -- My Docs and any personal folders they've built themselves -- automatically. On the server side, you need a daily snap, so just thank God for block-level change technology.

Moral: Great ideas are fine, but you have to weigh them against every user's inherent resistance to change. User populations are like bacterial ecosystems from distant planets. You can't predict with very much precision how they'll evolve, so things like backup need to use the word "holistic" rather than "targeted."

Related articlesStupid user tricks 3: IT admin folliesIT heroes toil away unsung in miserable conditions -- unsung, that is, until they make a colossally stupid mistakeMore stupider user tricks: IT horror stories reduxIdiot-proof your enterprise with these 10 hard-luck lessons of boneheaded IT miscuesEven dirtier IT jobs: The muck stops hereMore dirty tech deeds, done dirt cheapThe 7 dirtiest jobs in ITSomebody's got to do them -- and hopefully that somebody isn't youTrue IT confessionsSupergeeks fess up to some of the dumbest things they've ever done -- and the lessons they learned as a resultStupid QA tricks: Colossal testing oversightsThe trick to nipping IT miscues is testing, testing, testing, as these hard-luck lessons in boneheaded quality assurance attestStupid hacker tricks, part two: The folly of youthTech-savvy delinquents set the Net aflame with boneheaded exploits that earn them the wrong kind of fameProgramming IQ test: Round 2 Acing last year's quiz was nothing. Only true hacker heroes will survive Round 2Linux admin IQ testHow much do you really know about the free OS?Apple IQ testHow much do you really know about the house that Jobs built?Web IQ testThink you know all there is to know about the World Wide Web? Take our quiz to find out

This story, "Stupid user tricks 4: IT horror never ends" was originally published by InfoWorld.

Copyright © 2009 IDG Communications, Inc.

1 2 Page 2
Page 2 of 2
9 steps to lock down corporate browsers
  
Shop Tech Products at Amazon