IT managers and analysts are expressing surprise at the amount of time it appears to be taking San Francisco officials to regain full control of the city's FiberWAN network after a disgruntled network administrator allegedly locked access to it by resetting administrative passwords to its switches and routers.
With few details publicly released on what exactly happened, many are suggesting that the network lockout and the city's response to it point to a failure to implement and manage fundamental security controls.
Terry Childs, a network administrator at San Francisco's Department of Telecommunications and Information Services (DTIS) was arrested on July 13 for allegedly tampering with the network, which carries almost 60% of the city government's traffic. He was also charged with planting network devices that enabled illegal remote access to the FiberWAN network.
He was jailed last week on $5 million bond after refusing to divulge the passwords he had used to block access to the network. Childs, 43, pleaded not guilty to the charges against him at a hearing in San Francisco Superior Court last week. He is scheduled for a bail hearing tomorrow. If convicted on all charges, Childs faces a maximum of seven years in prison.
As of late Monday, the city's efforts to repair the network still remains a "work in progress," according to Ron Vinson, deputy director at the DTIS. Despite round-the-clock efforts with vendors and staffers, the city is still only in the process of determining "what level of illegal access or what level of tampering" Childs was responsible for, Vinson said.
According to Vinson, the WAN is operating normally, but the city has not yet regained full administrative control of all routers. What also remains unclear is whether IT officials have discovered all of the network devices that Childs is alleged to have illegally installed on the WAN, he said. Vinson did not say what exactly the network devices were or what they did, but he claimed that Childs had gone to great lengths to hide them from detection. "We have 60-plus departments that are clients of ours," Vinson said, and the task now is to find out if Childs managed to install the devices at any of those departments.
"That is why we need to do a systemwide analysis of where we are with access," Vinson said. "We don't know what he had access to." He added that the focus right now is on damage containment. "We want to make sure we maintain full operability if [Childs] was to be released on Wednesday" and were to try to gain illegal access to the network, he said. "We want to make sure we are up and running."
However, Vinson did not provide any details on what exactly Childs did or the extent to which he may have compromised the network. He said that the reasons for the slow recovery would become apparent once those details were publicly released.
Meanwhile, news of the city's continuing struggles, combined with a relative lack of publicly available details on what exactly happened, is fueling questions and theories about what may have happened in some quarters.
"I am completely floored that it is taking so long to restore access to the equipment," said Jim Kirby, senior network engineer at Dataware Services, a Sioux Falls, S.D.-based service provider. "Unless they have some crazy uptime requirement that prevents them from rebooting gear, it's hard to understand." In most cases, he said, passwords can be reset with a reboot and some keyboard combinations.
But, Kirby added, with some networking gear from Cisco Systems Inc., it is possible to issue a command that renders password recovery impossible, making it necessary to send the equipment back to Cisco to be "reflashed to new." "Even so, it's merely a matter of replacing locked gear with new gear created from the backup configurations they surely must have somewhere. One box at a time, if needed," he said. The apparent fact that San Francisco city officials took longer than 48 hours to restore access suggests that they were not following "even basic network-administration standards," Kirby said.
Similar sentiments were expressed by Johannes Ullrich, chief technology officer at the SANS Internet Storm Center in Bethesda, Md. "It is odd that it took so long to recover. The only reason I can think of is that they did not have good configuration backups," he said. Though it is unclear what networking equipment was involved, typically it is possible to reset passwords with physical access to the equipment, though all configuration changes would be lost, Ullrich said. Good configuration management practices also would have allowed for unauthorized changes such as those made by Childs to be detected quickly, thereby limiting damage. "In addition, it can help with recovery as it is easier to revert to an old known-to-be-good configuration," he said.
Lou Michael, director of network and infrastructure services in the Department of Technology Services in Arlington County, Va., said that based on media reports alone, he did not want to come down against the IT officials in San Francisco. "But it is a long-standing best practice to make sure that access passwords are never in one place or with one individual," Michael said. "I have worked on systems where root level, hidden accounts were maintained so that if a rogue individual changed all the passwords or attempted to lock the accounts, they would not know of these accounts' existence," he said.
He theorized that even if Childs had used just one password to lock others out of the network, San Francisco IT officials would probably need to initialize and rebuild each locked device to gain access to them. "Depending on how many people you have working on it and when you are allowed to bring the device down, it could take a while," he said.
The entire incident points to a management failure at DTIS, said Arshad Noor, CEO of StrongAuth Inc., a vendor of compliance and ID management products in Cupertino, Calif. For starters, a single employee should never have had exclusive control over the WAN, in the manner Childs appears to have had, Noor said. There appears not to have been much of a configuration or change management process or a disaster recovery plan, judging from the recovery time, he theorized.
"All in all, the IT management of the city are responsible for this mess, because it was within their authority -- and in their mandate -- to avoid this situation, but they did not," Noor said. "While Terry Childs might pay for this situation through jail time or fines, management cannot be absolved of their responsibility for the situation."