Questions abound as San Francisco struggles to repair locked network

Computerworld - IT managers and analysts are expressing surprise at the amount of time it appears to be taking San Francisco officials to regain full control of the city's FiberWAN network after a disgruntled network administrator allegedly locked access to it by resetting administrative passwords to its switches and routers.

With few details publicly released on what exactly happened, many are suggesting that the network lockout and the city's response to it point to a failure to implement and manage fundamental security controls.

Terry Childs, a network administrator at San Francisco's Department of Telecommunications and Information Services (DTIS) was arrested on July 13 for allegedly tampering with the network, which carries almost 60% of the city government's traffic. He was also charged with planting network devices that enabled illegal remote access to the FiberWAN network.

He was jailed last week on $5 million bond after refusing to divulge the passwords he had used to block access to the network. Childs, 43, pleaded not guilty to the charges against him at a hearing in San Francisco Superior Court last week. He is scheduled for a bail hearing tomorrow. If convicted on all charges, Childs faces a maximum of seven years in prison.

As of late Monday, the city's efforts to repair the network still remains a "work in progress," according to Ron Vinson, deputy director at the DTIS. Despite round-the-clock efforts with vendors and staffers, the city is still only in the process of determining "what level of illegal access or what level of tampering" Childs was responsible for, Vinson said.

According to Vinson, the WAN is operating normally, but the city has not yet regained full administrative control of all routers. What also remains unclear is whether IT officials have discovered all of the network devices that Childs is alleged to have illegally installed on the WAN, he said. Vinson did not say what exactly the network devices were or what they did, but he claimed that Childs had gone to great lengths to hide them from detection. "We have 60-plus departments that are clients of ours," Vinson said, and the task now is to find out if Childs managed to install the devices at any of those departments.

"That is why we need to do a systemwide analysis of where we are with access," Vinson said. "We don't know what he had access to." He added that the focus right now is on damage containment. "We want to make sure we maintain full operability if [Childs] was to be released on Wednesday" and were to try to gain illegal access to the network, he said. "We want to make sure we are up and running."