Verizon router glitch slams parts of U.S.

IDG News Service - A Verizon router failure in the northeastern U.S. caused an outage that Internet engineers and Twitter users reported on Friday afternoon.

Verizon is still investigating the problem, but it apparently involved a card in a router in New York City that failed, according to Verizon spokesman John Bonomo. The problem occurred between about 3:15 p.m. and 3:50 p.m. Eastern time, according to Verizon.

The router was taken offline and traffic seems to have recovered, Bonomo said. Problems seem to have been concentrated around New York City, though they may have been broader, Bonomo said. Verizon had no estimate of the number of customers affected.

The problem seems to have affected users of DSL (digital subscriber line), leased lines and Verizon's FiOS fiber-to-the-premises service, according to comments on the North American Network Operators Group mailing list, as well as on Twitter.

Around 4:30 p.m. Eastern, Verizon Senior Vice President Eric Rabe posted on Verizon's Twitter feed that the carrier's engineers were aware of the problem and working on it. About 15 minutes later, Verizon said a peering router between its network and the Internet for the Northeast, which handled traffic for DSL and FiOS, had failed. The router has been restarted and is now handling traffic, Verizon said.

Because the router went into a hung state, it appeared to the rest of the network as if it didn't have a problem, Verizon's Rabe wrote in a blog post later on Friday. And because the router couldn't communicate its failed state, traffic from some customers kept flowing to the router but couldn't be processed, so they lost their access, he wrote.

The structure of the Internet and carrier data networks normally prevents such problems through the ability to route traffic around a failure in seconds. In addition, routers on critical paths are typically deployed in pairs for failover, said Burton Group analyst David Passmore.

"It's highly unusual that a single router could take down substantial portions of the network," Passmore said. Such failures are more often caused by "backhoe" incidents in which a work crew digs in the wrong place and cuts a cable used by several carriers, he said. Misconfigured routers or network settings, or software bugs that are distributed to multiple routers, can also cause routing problems that don't get resolved immediately, he said.