Fragmented databases behind Scottish election debacle

Computerworld UK - The sheer volume of spoiled ballot papers in the Scottish elections last week led to technical problems with electronic counting systems that delayed the results and caused angry debate in both the Scottish Parliament and House of Commons, according to the technology provider.

Supplier DRS, which attributed the delays in five areas to data consolidation problems and "fragmentation" of databases, has given Computerworld UK a more detailed explanation of the problems it encountered, though its internal inquiry continues.

The DRS e-counting machines are designed to read and recognize both the X marks used by voters in the Scottish Parliament elections and the hand-printed numbers used on the transferable voting ballot papers for the local government poll.

In the immediate aftermath of the elections, DRS said the scanners used to read each vote -- or capture an image of unclear ballot papers for adjudication by the returning officer -- had worked without problems.

But a spokesperson added, "The issue involved a blockage at the end of the counting process, which prevented consolidation of the data." This meant the results could not be announced.

The company has now offered a more detailed explanation of the glitch. A spokesperson said, "The DRS e-counting systems encountered significant fragmentation (up to 99%) of a number of the database indexes, which caused the complex queries required to produce final results and some status summary reports to time out.

"Whilst investigations are still under way to understand the precise reason for the index fragmentation, at this early stage it remains likely that the number of rejected ballots was the major contributory factor."

The level of rejected -- or "spoiled" -- ballot papers had been far higher than expected, with figures obtained by the BBC from returning officers at each counts showing a total of nearly 142,000 rejected papers -- almost 7% of the votes cast.

The DRS spokesperson said, "The database had been tuned for the expected levels of workflow which had not included the unforeseeably high level of ballots that ultimately had to be rejected."

She added, "DRS is a Microsoft Certified Partner and the e-Counting system was designed around the latest Microsoft SQL 2005 Server technology. Over the past 18 months, DRS has been thoroughly testing the system using a team of ISEB quality engineers.

"Numerous load tests were conducted to 135 percent of maximum predicted capacity and at no time during these tests was index fragmentation an issue."

Returning officers at five affected sites decided to adjourn the election counts for several hours to allow a fix to be thoroughly tested before proceeding. The problem was eventually remedied by index defragmentation, which restored all five systems to full working order.

DRS said it deeply regretted the delays and the frustrations experienced by returning officers, staff and candidates.

The company's technology was also used for electronic counting of the London Assembly elections in 2000 and 2004 without any problems.

The Scottish election delays are to be investigated by the Electoral Commission. The inquiry will also look at problems with the elections unrelated to the technology, including the high number of rejected ballots.