Data center moving day: REJIS makes it a smooth one

1 2 3 4 Page 4
Page 4 of 4

When Frank heard where he had to go, he was a bit worried. "I am going to pull up to this building in the middle of the night and every cop in St. Louis is going to be there," he told me as he was sitting on the floor of the service room, trying to debug the elevator circuitry. "Sure enough, here I am at this high-tech cop shop."

The reason he is sitting on the floor is because there is this huge pile of manuals next to him. "The manuals aren't well written, and the documentation on the error message that I am getting is very obscure." Sounds a lot like what most of us have had to do with computer manuals.

While Frank puzzles over the manuals, Gorham is downstairs in the break room, giving everybody their instructions for the night. We are sitting next to a large stack of our own -- a large collection of pizzas, cookies (carefully home-baked and labeled by flavor by some volunteer), sodas and plenty of coffee. It is going to be a long night ahead and the movers need their fuel. "If you are going to ask people to come to work in the middle of the night, you want to make sure that they are going to be fed and keep them happy while they are working," says Gorham.

To get people juiced, Gorham draws a few random names for prizes -- this being St. Louis, more than two dozen people have donated pairs of Cardinals baseball tickets, and he has also purchased several restaurant gift certificates. Then we all go to work.

clear.gif
clear.gif
 
clear.gif
Tip:  Use triage to determine the most mission-critical machines to move first.
clear.gif
 
clear.gif
clear.gif

At 2 a.m., Frank gets the elevator going and the moving crew begins to take apart the control consoles that are needed in the new building. The crew starts to roll the IBM e-Server and virtual tape system across the floor and upstairs into the new building. Everything will sit on top of WorkSafe Technologies' ISO-Base seismic isolation platforms that can dampen vibrations in case of earthquakes.

As each machine is taken off the old rack, it goes by Chassidy's station. She is in charge of blowing out and clearing the accumulated dust with a reverse vacuum.

A pile of old bus and tag cables await recycling, showing how times have changed from the old System/370 days when the data center was first built in the 1970s.
A pile of old bus and tag cables await recycling, showing how times have changed from the old System/370 days when the data center was first built in the 1970s. (Click image for larger version)

REJIS isn't moving everything out of the basement; some of it's being recycled or tossed. Staffers are dismantling two robotic tape libraries that have outlived their usefulness, along with 10,000-plus tapes. "Our new library has the same storage with just 1,500 tapes and is compatible with the state data center formats, making it better for disaster recovery purposes," says Gorham. And REJIS plans on recycling a huge pile of bus and tag cables, and other old equipment to EPC Recycling.

Within a couple of weeks, the basement will be completely empty of all equipment, cabling, and even the raised floor will be removed, too. "We don't have any current plans for the basement, but are thinking about a few ideas for it," says Gorham.

By 2:30 a.m., "All of our red servers are in place," says Gorham. "We thought it was going to take longer." REJIS tagged each server by color, with red being the most critical (law enforcement servers, for example) that needed to have the least downtime. REJIS has a lot of nonlaw enforcement clients, such as doing the general data processing for a local city government -- running payroll jobs and processing other HR data -- that was triaged as not quite so mission critical.

By 3:58 a.m., the huge IBM eSeries servers are connected and back online. "I'm happy about the limited downtime," says Gorham. "We were really only down for 118 minutes for mission-critical services."

clear.gif
clear.gif
 
clear.gif

IMG_4636_inStory.jpg

Tip:  Tag every server with information about its IP address, old and new rack locations and server name. Place the tags near where the cable is connected on the back. When you cut the cables, you still have the tags to guide where to hook up the new wires.
 
clear.gif
clear.gif

Seeing how things are going well, I head home for a few hours' rest. I am back by 6:30 a.m. and walk into the server room, to see only one cable laying across the floor: "We just had one bad connection that we needed to fix," says Gorham. That was it; all other systems are up and running.

By 7 a.m., as breakfast is served in the break room, most of the crew is finishing up and getting ready to head home. But not before a few lucky ones are picked at another random lottery to get some Cardinals tickets as their reward for a good night's work.

Looking back with the benefit of a couple of weeks of hindsight, Gorham says he is very satisfied with the overall move. "I would have done some of the building's system tests earlier on -- such as making sure that major electrical and air conditioning systems were working, and maybe we should have had the elevator guy on site," he says. "And it would have been nicer if we had more new racks to spread out our servers."

But that's about it. All in all, it was a good night for a move.

David Strom is the former editor-in-chief at both Tom's Hardware.com and Network Computing magazine, and is now a podcaster, blogger, public speaker and freelance writer for numerous IT pubs and The New York Times, among others. His blog is located at http://strominator.com and he can be reached at david@strom.com.

Copyright © 2007 IDG Communications, Inc.

1 2 3 4 Page 4
Page 4 of 4
  
Shop Tech Products at Amazon