Shark Tank: Oops!

Consultant pilot fish and his team are at this brokerage firm to install a backup network connection. "We had to do the work after hours, but we got there by 4 p.m. anyway just to drop off our equipment in the data center," fish says.

"While some of us are rack-mounting our equipment, others are locating the workstations that will be affected by this installation.

"Meanwhile, the bored data center administrator is watching us work to make sure we don't impact any critical systems until our maintenance window."

There's not much to watch. But fish notices that the admin is standing by the big primary data-center UPS and idly playing with the switch.

And shortly after that, fish notices that the meat-locker-size UPS has started a steady beeping -- and the admin is struggling with the switch.

Realizing he has fish's attention, admin sheepishly says, "Uh, I think with this type of UPS, you can't just go from 'test' back to 'on' -- I think you have to put it in 'standby' before switching it back to 'on.'" Sure enough, after a few minutes the UPS goes into standby mode automatically.

And the brokerage's data center goes down -- just before the market closes.

Seconds later there's a knock on the door: It's VP of operations, who must have sprinted to the data center from his office three floors away. "So, what happened?" he asks in a remarkably cheerful tone.

Says fish, "It appears that the e-mail announcing we'd be installing the new connections went out just prior to the unplanned outage, and the users are hopping mad, thinking we'd taken their network down just as they were getting ready to finalize their trades for the day.

"We had a tense several hours as we explained ourselves repeatedly to multiple people at various levels, and we had to delay installation for several hours.

"The customer ended up dropping us shortly thereafter, citing 'reliability concerns.'

"As I understand it, the data center admin received a lecture, but his career was otherwise not impacted by the event. But who said life is fair?"

Submit your own true tales of IT life to sharky@computerworld.com. If Sharky uses it, you'll snag a snazzy Shark Tank shirt! You can also add comments by using the form at the bottom of this page.

See more Shark Tank stories at the Sharkives.

Join Shark Bait

Now you can post your own stories of IT ridiculousness. Join Shark Bait today and vent your IT frustrations to people who've been there.
Go to the
Shark Bait homepage
or
Post a story now


Copyright © 2007 IDG Communications, Inc.

Bing’s AI chatbot came to work for me. I had to fire it.
Shop Tech Products at Amazon