This big international airport has a huge computerized central control system that’s maintained by an on-site field engineer — with help from the vendor half a world away.
“The software engineers relied heavily on the knowledge and expertise of the field engineer to install our software on the running system,” says a support pilot fish working for the vendor.
One day, the field engineer calls the vendor to say that the system’s BitBus device — which relays data between a network of sensors and the central control server — has a bug in its software.
According to the engineer, the BitBus device runs fine for a day or two. But then it begins “jabbering” — spewing corrupted messages onto the network — and has to be reset.
And since the device is the only way for the central control system to listen to thousands of sensors and to control remote devices, that’s very bad news.
Worse still, the malfunction makes no sense. “Other than having added some new hardware devices to its inventory, the BitBus software had been stable for quite some time, so its recent misbehavior was a mystery,” fish says.
Fish asks the field engineer to report the checksum for the embedded software. The engineer does, and fish checks it against the current version’s checksum. They don’t match.
Fish has the engineer reload the software onto the device and then reboot it. Now the checksum matches, and the BitBus device appears to be working correctly.
But how could the loaded software get corrupted in the first place? fish asks engineer.
“I didn’t do anything,” engineer tells him. “All I did was run the patch that you gave me a few months ago.”
Um, the latest version of the software we shipped you was newer than the patch, fish tells engineer. That patch could only be run on the previous version of the software.
“What he had done in effect was poke a random change into the newer program,” sighs fish. “This caused the BitBus device to slowly develop the malfunction anytime the randomly patched code was executed.
“I had him discard all copies of the previous program, as well as any patches, so he couldn’t make that mistake again.”
It’s always a good time to send Sharky your true tales of IT life. Email me at sharky@computerworld.com. You can also subscribe to the Daily Shark Newsletter.