After a rash of ill-conceived IT projects, with servers that are either under-provisioned or over-provisioned, company management decides to split the sysadmin group into operations and architecture teams, reports a pilot fish on the scene.
"The arch team would be the gatekeeper, vetting the project details and, in theory, preventing this happening again," fish says. "It seemed to be going well. The arch team gathered and scrutinized and sometimes modified the plan from the systems analyst."
But as time goes on, discipline starts to slip -- and communication between the teams starts to fall apart. Calls and email go unanswered. Changes are made by one side without notifying the other and with no audit trail. Worst of all, in some cases approvals and change control never happen at all.
And as communication collapses, the arch and ops teams start working under different assumptions about how day-to-day operations should go.
The problem comes to a head after the arch team sends out a browser update that's been tested -- but not thoroughly -- and ends up breaking a few ancient Windows servers that control scientific instruments.
Realistically, breaking a few servers out of a thousand isn't bad. Unfortunately, those servers have been treated as if they were embedded controllers -- they haven't been updated in years.
Worse, there aren't any backups for the control software, which now needs a reinstall from the vendor -- which may not may not exist anymore.
"Those servers' files were changed without any notification, and we're not sure if they were tested or not," says fish. "Who did it is all in the log, but since the communication breakdown, it's hard to tell whether they ignored it, or are just afraid to admit it."
Sharky won't name names -- I just want your story. Send me your true tale of IT life at firstname.lastname@example.org. You'll score a sharp Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.