Everybody has storage problems, but you can always find room on your hard drive for these funny anecdotes collected from the IT front lines by our Sharky.
Game theory
Users at this hospital each get a directory on a file server, mapped to the I: drive. When this IT pilot fish notices during a routine audit that one user's I: drive has gotten very big, he investigates and finds it contains several hundred megabytes of games -- a major no-no.
User's explanation? "I just got this computer a few months ago, and the games were on it when I got it!"
Sighs fish, "She's still having trouble grasping the concept of a network drive."
That other school in Cambridge teaches math, too
At this printing business, IT has lagged in the past. But now it's coming on strong, says a pilot fish working there.
"So the company hires a 'manager's manager' who can't even turn on any of the workstations," fish grumbles. "Management brags about her pedigree and training, and she drops the Harvard name every chance she gets. I think she went to a seminar there."
But fish isn't impressed, especially when his new boss is arguing for a network-based backup system the company is evaluating on loan from the vendor.
The servers have 4TB of storage that must be backed up. "I point out that the network's a moot point," he says. "The RAID the data is on is pretty old, and will only spin at about 12MB/sec. Backup is going to take a lot longer than overnight at those speeds.
"I'm told I am being difficult, and shouldn't act so negatively toward proposals from this vendor."
So fish is elbowed out of the backup evaluation, and his boss decides to get it working herself. "Harvard spends five days on the phone with the vendor's tech, trying to remotely troubleshoot the problem," says fish.
"Finally she gets a backup started. After it's working for a couple hours, she comes in and points to the monitor and boasts in that I-told-you-so voice, 'See, it's working. 700MB/minute!'
"I politely wait for her to do the math. Then when it's obvious there's no intention to, I say, 'Yup ... or about 11 to 12MB/sec., right?'
"After 15 seconds of staring into space blinking, she leaves the room without saying a word."
What a Concept!
After user's hard drive is replaced, she remembers a file that wasn't transferred to the new disk -- and it has information she needs.
"She knew the old one went to a storage room full of drives," says a pilot fish on the scene. "But when she asked us to retrieve it, we did so in a few moments."
Amazed user: How did you find my old drive so quickly in that hard-disk graveyard?
Fish: "We put your name on it."
What could possibly be hard about managing IT?
There's downsizing. And departmental restructuring. And when the dust settles, this small company's IT team is lumped in with the operations department, and in charge are two managers who have never run an IT staff, says an IT pilot fish on the scene.
"However, they deem themselves outstanding managers and feel managing IT is easier than managing operations," fish says.
But he has more pressing problems to worry about: The company's mail and file servers are running out of storage. So he asks his new bosses for permission to attend a computer storage trade show in a nearby town. It won't cost anything but a little time, and fish and his fellow admin will be able to collect information from all the disk and tape vendors at once.
Fish's new boss responds by e-mail: "To be honest, both your time would be better spent on other, more important stuff, bearing in mind the storage of our excess computers is not a big issue at the moment and I don't envisage it becoming an issue for a long time."
No, thinks fish, not that kind of computer storage. "He thought the recently vacated offices would provide ample storage for all the extra computers previously used by the redundant staff," says fish. "It seems he wished to prevent his newly acquired IT staff from exploring cupboard or Tupperware options."
But it's an easy mistake for someone new to IT-shop jargon to make. So fish begins drafting an e-mail response to clear up the confusion: "Boss, we meant computer storage, not storage for computers" -- when his boss's boss, the department head, passes by and notices what fish is doing.
"When your supervisor says no, he means no," department head tells fish. "Stop writing that e-mail and focus on more important things."
"She then walks off, leaving me to pick my jaw up off the ground and find something more important than an impending server crash," fish says.
"I update my resume and resign the following week. The mail server crashes two weeks after I leave.
"Why? Storage problem."
Space: The Final Frontier
Central IT has just upgraded this site's e-mail server with a much bigger hard drive - it's big enough to give each user an extra 50MB of storage. So how could it possibly run out of space and crash a few days later?
"There was no free space," grumbles local IT pilot fish. "They had created a file share for the help desk, where everyone was storing their games and personal files, leaving the mail system with even less storage than before. I destroyed the file share, and magically my mail server returned to life -- but not before I changed the admin password on the box."
It's what we call 'field testing' a new product
It's the mid-1980s, and there's a hot new feature for this IT shop's mainframes: the ability to let multiple machines access the same disk storage arrays automatically. That seems to be exactly what's needed, says a pilot fish working there.
"The plan is to have one CPU running production, the other testing and development, and both sharing the same disk packs simultaneously," fish says.
So the upgrade is scheduled for one weekend. The team from the mainframe vendor will come in on Friday night, and by Monday the new system will be up and running.
At least, that's the plan.
"I arrive on Monday to find the entire data center raised floor removed and stacked out in the hall," says fish. "Hundreds of cables are draped over the equipment, and most of the disk controllers are disassembled."
There's a problem with the controllers, a vendor technician tells the people from IT. But they're working on it.
"After an emergency meeting between administration and the vendor, a single directly connected CPU is rebuilt to run the must-have business reports while the vendor sorts out the multiple-CPU-connection problems," fish says.
"At night, the vendor techs recable to test. Next morning they unrecable back to a single CPU. This goes on painfully through the week until a new team of techs from the vendor shows up Friday night to fix the problem."
Bright and early Monday morning, the new vendor tech team proudly unveils its solution to the people from IT. "It's a Frankenstein-style open-face switch," sighs fish, "cabled to the top of one of the controllers, allowing the disks to be manually switched between the production CPU and the test CPU.
"That's when administration finally agrees with us to order a new mainframe.
"From a new vendor..."
THAT'S what they're for!
This sysadmin pilot fish is in charge of a group of servers that's spread over a wide area. Some sites have their own IT people; at others, local office staffers do things like load and remove backup tapes.
And one remote site has nothing but problems with backup. "Things happen so often that the hardware vendor's service engineer is on a first-name basis with the office manager who pulls the backup tapes each day," fish says.
"The engineer replaces several tape drives and eventually the library robot components. After one recent tape drive swap, he mentions that a tape was lodged in the drive, and he removed all the gooey stuff from the tape, but we ought to destroy it."
Gooey stuff? Fish is busy dealing with another problem at the time, but the comment about "gooey stuff" keeps running through his mind. There's nothing gooey in a tape or a drive, he thinks.
Meanwhile, the remote office manager e-mails fish with a request: She needs more backup tapes because she's having to throw away about one gooey tape per week, her message says.
Fish calls the office manager and asks about the gooey stuff. And he's startled by her response.
"The off-site storage company puts a bar-code label on the tapes when they pick them up," she says. "They peel them off when the tape is returned. Some of the adhesive stays on the tape."
Fish is furious. He calls the off-site storage company and demands to know why adhesive labels are being put directly on the tapes.
"OK, we'll use a solvent to remove the extra adhesive in the future," the account administrator tells him.
But why not just put the label on the nice plastic tape box instead? fish suggests.
"The site doesn't send the tapes in the plastic boxes," account administrator says.
Fish hangs up and dials again. Why are the tapes going off-site without their protective boxes? he asks the office manager.
"I didn't know we were supposed to use those," the office manager says. "I have hundreds in a cabinet, and I was wondering what to do with them."
The letter of the law
This pilot fish's server is upgraded, which means it's time to review all the processes related to disaster recovery.
"The nature of our transactions required 13 months of data to be recoverable if necessary," fish says. "So we asked questions of our elite backup department about what happened to the data once it was backed up to tape."
The backup guys explain that the data is on the machine for a week, then tapes are held on-site for another week, after which they're sent off-site.
OK, says fish. Now, once those tapes are sent off-site, how long will it take to get access to them in an emergency?
The response: "It shouldn't take long. It's only sent off-site for a week."
Huh? What happens to it then? fish asks.
"It returns to the original site for storage for the additional 12 months."
But what happens to the recovery of the 12 months of tapes if the data center is destroyed along with our machine and the latest two weeks of stored backups?
"Oh, yeah. That could be an issue, huh?"
Turns out that, after yet another round of cost reductions, management decided it could cut the budget significantly by dumping the outside data site, says fish. And now all the backup guys do to meet the corporate "move off-site" policy is move the tapes off-site -- and then back again.
"So we do our own backups," fish sighs. "To a different site -- just in case."
C'mon, why would we need backup?
Flash back to the days when this IT department still uses huge, old disk-pack drives for its storage, and a consultant pilot fish is responsible for helping keep things going.
Each of the 20 drives holds 65MB of data and is the size of a washing machine. So when the CIO announces the department will upgrade to a state-of-the-art RAID system -- four stacks of four drives totaling more than 14GB -- it's a huge leap forward.
"CIO says that since the mean time between failures is rated at years, and one drive per stack could die without a problem, the system would not need to be backed up," fish says. "It's invincible. That's what the salesperson told him."
Fish points out that two drives could fail, or there could be a fire or other disaster. "But CIO says no backup -- this is the wave of the future," says fish.
A few months after the RAID system goes in, a drive fails. Just as it should, the system keeps working. Fish orders a new drive to swap in.
Then he takes a closer look at the drive that died. The metal coating is flaking off due to excessive heat -- a very bad sign.
"The dead drive is replaced," he says. "Then I cross my fingers and power on the stack, one drive at a time. Replacement drive at the bottom goes green OK, second drive goes green OK, third drive goes red -- uh-oh, problem.
"I repeat powering on the third drive several times -- still a red light. This means the RAID is dead, with no backup.
"That's when I start making phone calls up the ladder of command, explaining that we will need another drive but the data is lost."
Soon the management accusations and finger-pointing begin. And the IT manager takes fish aside and asks if there's anything he can try.
"I explain that the heads are probably stuck to the platter on the third drive," says fish. "I tell him the drive can be removed and put in a freezer, but this would void the warranty."
One call to the RAID vendor later, fish has permission to try it. After 30 minutes in the freezer, he removes it, puts it back in the RAID stack and once again crosses his fingers.
"As people stand around holding their breaths, the power switch is flipped -- and the light goes green," fish says.
"Everyone lets out a sigh of relief. The possibility of bringing back the disk packs and losing several months' worth of data is avoided.
"The next day I suggest a backup unit again. It's ordered that day. Other drives fail over the next several months. One more deep-freezer incident happens.
"And the CIO moves on -- no questions asked."
Battling Complexity
Stories in this report:
- Battling Complexity
- Virtual Unity
- Decluttered Data
- Watchful Eye
- Cleaning Out the Attic
- Storing Stuff
- Executive Briefing: The New Storage Paradigm
- Downloadable Data Points on Storage
- Silly Storage Stories
- Webcast: Grid Security Infrastructure
- Storage Automation Part 1: Pot of Gold or Can of Worms?