Leap second bedevils Web systems over weekend

IDG News Service - Despite precautions by system providers, an extra second added to the official timekeeping record Saturday triggered several popular Internet services to crash over the weekend, including LinkedIn, Reddit and Quantas airline's reservation system.

Among systems that reportedly experienced difficulty with the extra second were unpatched Linux OS kernels, Hadoop instances, Cassandra databases and Java-based programs. Enough problems were documented around the globe that some even likened leap second change to another Y2K.

Leap seconds were introduced in 1971, as way to synchronize official digital time keeping, called Coordinated Universal Time (UTC) and mean solar time, namely by adding or removing a second from the official time. Overseen by the International Telecommunications Union, UTC is defined by the accumulation of seconds, which are defined with great precision by atomic clocks. Solar time, in contrast, measures the day by the time it takes the Earth to do one complete rotation, which can fluctuate slightly due to tidal effects, the slowing of the earth's rotation and other factors.

On Saturday, the ITU added a second onto the end of June 30 UTC, at midnight Greenwich Mean Time, or 8 PM eastern daylight time, the 25th adjustment to UTC since 1971.

Typically, UTC is coordinated across the Internet using the Network Time Protocol (NTP), in which the official time, supplied by the U.S. Navy and other sources, is passed among Internet servers. The client software for NTP can be configured to accommodate the leap seconds when they occur. Additional solutions have been offered to better arm servers against possible leap second failures, including those introduced by Opera and Google engineers.

For this leap second however, numerous problems were experienced. Those with servers running Debian Linux had reported several servers going offline after the leap second occurred. The servers were restored after NTP was temporarily disabled.

Reddit reported failures, stemming from its use of the Cassandra database and programs written in Java (although Reddit may have also been affected by storm related power outages that also took place Saturday in the Northern Virginia area). Mozilla also reported on leap second issues, stemming from Java. Mozilla's deployments of Hadoop, which heavily use Java, needed to be restarted. DataStax, which manages development of the open source Cassandra database, did not immediately respond to a request for comment.

The reservation system for Australian airline Quantas also reportedly went offline due to a leap second crash. LinkedIn also experienced uptime difficulties following the leap second.

The issue with Linux stemmed from a deadlock, a patch for which was released in March.

The ITU has considered eliminating leap seconds because of the problems they can cause IT systems.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com