So THAT'S how it could happen!


Consultant pilot fish has 25 years of experience as a mainframe programmer and database administrator when this company asks him to come on the payroll.

"I transferred into a Unix system administrator position to expand my skill set, and things looked set for an exciting few productive years before I retired," fish says.

"But there was a departmental reorganization and Barney, the new head of sysadmins, prefers his young staff to older heads. I was shifted to administering an application we support. Oh well."

But one day fish finds that a log file is unreadable. The Unix "cat" command, which dumps the contents of a file to the terminal screen, makes it clear that the log file is now type binary.

Puzzled, fish restores logging and tries to research the issue with no luck. The next morning, the log file has become type binary again. Every morning after that, too.

But with some careful poking around in the server, he discovers that, over the root file system, a volume has been defined using third-party volume manager software. Aha! fish thinks, a classic case of virtual storage and file-type corruption.

"And this machine is one of our busiest production servers," says fish. "So People Will Care."

He asks for help from Barney, clearly describing the problem. Eventually a bored-looking sysadmin named Fred wanders in and announces he knows the problem: "You can't use cat to read binary files."

Fish: It didn't start as a binary file. It's a log file. Fred: "You can't use cat to read binary files."

Several rounds later, fish gets strident enough that Fred actually listens when fish says the file starts out as a text file but somehow changes in the night to a binary file.

Fred: "How could that happen?"

Fish describes what he's found.

Fred: "That's impossible."

Friday evening, fish sets up logging again, along with a second job that checks the log's file type every ten minutes. Monday morning, he has solid evidence that the file type turns binary between 1 a.m. and 1:10 a.m. on Sunday.

Fish writes another note to Barney -- and as he hits "send," an entire filesystem simply vanishes from the server.

"Now there's action," fish says. "Our online system is down at 9:30 a.m. Monday morning and some Very Important People are shouting.

"When the dust settles, it turns out that before the volume was laid down, the sysadmin forgot to run the file-system checking utility fsck on the root filesystem. Everything was OK until the volume manager flushed its in-core data down to disk. It was the problem I'd nailed days before.

"And the sysadmin who screwed up? None other than 'How could that happen?' Fred."

Feed the Shark! Send me your true tale of IT life at You'll score a sharp Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.

Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.

The march toward exascale computers
View Comments
Join the discussion
Be the first to comment on this article. Our Commenting Policies