All employees of this government agency carry a Personal Identification Verification smart card -- or PIV for short -- to log into computers and get into agency facilities, according to a pilot fish on the scene.
"On Thursday at 3 p.m., an agency-wide email about PIV cards was broadcast," fish says.
It reads: If you were issued a PIV card after March 15, 2013, you will need to update the certificates (the electronic data elements on your card that allow you to gain access to data systems) on your card at your earliest opportunity, and do so on a regular basis, thereafter.
Even though your PIV card MAY PHYSICALLY show a five-year expiration date, the electronic certificates on the card expire after three years. You can update your certificates by going to any PIV light activation station and running the MAINTENANCE function on your card. This will update your certificates and, in effect, extend the certificates' validity for another three years.
Forty-five minutes later, everyone gets a follow-up email from the office's executive assistant: We are experiencing problems with the PIV Activation Station. Several team members have had their PIV card disabled while making the attempt to run the maintenance as directed in the email. Please hold off on running the maintenance until we advise the PIV station is working properly.
Fifteen minutes after that, this from the division director: WARNING!!! We just had someone go through the process, and it does not seem to be working at this time and it throws an error stating 'certificates not ready, please try again later' and now his PIV card is not working.
"Thursday at 6 p.m., the 3 p.m. agency-wide email was rebroadcast," sighs fish.
But it's not over. Friday at 7 a.m., an agency-wide email tells users: It is IMPERATIVE that no action be taken to perform MAINTENANCE on your PIV Card until further notice. Performing maintenance will corrupt the internal certificate on your PIV Card rendering it unusable. Please DO NOT perform the described MAINTENANCE until further notice.
Friday morning, 8 a.m. report to CIO: Users received an email informing those with PIV Cards that were issued in the past three years that they needed to perform 'maintenance' on their cards due to expiring certificates. The email caused thousands of people to simultaneously attempt to update their cards and the PIV Card Certificate Server crashed. Incident began Thursday at 3:35 p.m. and remains unresolved. Technicians are working to get the server back online.
"The division director reported that over 6,000 users' PIV cards had been disabled by Friday morning," fish says. "They had to be reissued before the employees could regain access to information systems and access-controlled work sites."
You can get access to Sharky anytime -- just send me your true tale of IT life at email@example.com. You'll snag a snazzy Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.