After connecting to the server from his notebook at a local Panera sandwich shop (the chain is well-known among business travelers for its free Wi-Fi), Laping determined that the server's network interface card (NIC) had to be powered down.
That meant the server had to be rebooted, which required nothing more than pressing a button. The only problem was that nobody back in the office could find the right button.
The CEO and the power user were mortified that they couldn't figure out which button to push, says Laping, but this particular machine was a Dell rack server with a flat design rather than the tower configuration with which the men were more familiar.
The two kept pushing a button that was for adjusting the display, not turning the unit on and off. When nothing happened, they panicked.
In the end, everyone agreed that the easiest solution would be for Laping to physically fix things himself. "I had to drive two hours back to push a power button," says Laping, recalling that he turned right around and got back on the road once the NIC was up and running again.
Lesson learned: Even with smart devices, wireless services and VPN technology, not every problem can be dealt with remotely; make sure your backups know the basics -- like how to power down servers.
Don't forget to delegate your decision-making authority
On the night before Thanksgiving last year, T.J. Whelan was minutes away from arriving at his in-laws' house with his wife and kids, having driven two hours from Washington, D.C., to Wilmington, Del.
As he pulled into the driveway around 9 p.m., his phone started buzzing with texts from the network manager at the Washington law firm where Whelan is MIS manager. The messages said there was no connectivity to the Microsoft Exchange cluster.
That meant that attorneys in the firm's two U.S. offices and two overseas offices were completely cut off from email. "As I scrolled through the alerts, I saw one message stand out: 'We are f#!*d,'" remembers Whelan.
Whelan called the network manager -- who was also supposed to be on the road heading to his own holiday destination but had never gotten out the door -- and learned there was a major failure in the SAN containing the Exchange information stores.
The network manager contacted Dell support, which confirmed that the disks had failed but also reported that it might be a while before replacement parts could be located, because they were older models. (The law firm had been planning to replace them in the coming months.)
As MIS manager, Whelan was the only one able to oversee the process of getting replacement disks from Dell, since he alone had all the purchase and warranty information. He spent most of Thanksgiving morning on the phone with Dell, ate a quick holiday meal and packed his family back in the car to return early to Washington.
Whelan headed to the office, where he and the network manager decided to begin rebuilding the Exchange environment on a spare server rather than waiting for Dell to locate parts. By noon on Black Friday, Exchange was back up.
"The irony is that most of the [employees] in the U.S. never knew that there was anything wrong due to the holiday," Whelan says.