Office 365 email conks out twice within a week
Some customers in North and South America were affected
Network World - Microsoft's Office 365 service has suffered two email outages within a week of each other that affected some customers in North and South America that stemmed from different causes but ended in the same result: failed email delivery.
The first outage Nov. 8 stemmed from an overwhelmed antivirus engine and the subsequent backup that caused the service degradation. The second on Nov. 13 resulted from the failure of unspecified network elements, routine maintenance and increased load that combined to degrade service, according to the Office 365 blog posted by Rajesh Jha, the corporate vice president of Microsoft's Office division.
He didn't say how many customers were affected or where they were located other than somewhere on the two continents. Both outages affected just Office 365 Exchange Online mail services.
Affected customers are entitled to a service credit. Jha apologizes and promises a post mortem on the outages as well as an update on how the Office 365 service level agreement was affected.
The Nov. 8 incident started when an antivirus engine bogged down as it processed emails that the engine determined carried a particular virus. That delay processing emails led to retries that further bottlenecked email flow including legitimate emails, he says.
The issue was resolved by intercepting the tainted messages and quarantining them directly.
To head off similar problems down the line, the company has set a lower threshold for diverting problem emails and implementing faster remediation tools. It is also adding unspecified safeguards that automate remediation of this type of problem, Jha says.
The second incident Nov. 13 started with some scheduled maintenance that required shifting some of the load out of those data centers undergoing maintenance. During this work unspecified network elements failed but sent no alerts of their failure, he says. And finally the entire infrastructure was handling more traffic from new customers, all of which resulted in some customers being unable to access email services.
Traffic for affected users was shifted to healthy data centers while the issues were dealt with.
Jha says the company is in the midst of increasing capacity and is automating how equipment failures are handled to speed up recovery time.
In addition, the company is reviewing its processes to head off future outages.
"As I've said before," Jha blogs, "all of us in the Office 365 team and at Microsoft appreciate the serious responsibility we have as a service provider to you, and we know that any issue with the service is a disruption to your business - that's not acceptable. I want to assure you that we are investing the time and resources required to ensure we are living up to your - and our own - expectations for a quality service experience every day."
(Tim Greene covers Microsoft for Network World and writes the Mostly Microsoft blog. Reach him at email@example.com and follow him on Twitter https://twitter.com/#!/Tim_Greene.)
Read more about wide area network in Network World's Wide Area Network section.
- Social Media Education: The New Edge for Success Failure to train for social media will cost your business money. A recent report showed how digitally prepared companies can unlock up to...
- Social Media in Technology: A Unified Strategy for Success Find out how social media is sparking a new era of customer and industry-understanding in technology enterprises and how industry leaders are overcoming...
- How Network Connections Drive Web Application Performance Users around the globe, on all sorts of devices, expect Web applications to function as seamlessly as desktop applications. This paper discusses the...
- IDC ranks IBM #1 in market share for enterprise social software IDC ranks IBM #1 in worldwide market share for 3 consecutive years. IDC also shares its Social Software 2013 - 2017 market forecast....
- NSS Labs & Cisco Present: Evaluating Leading Breach Detection Systems Today's constantly evolving advanced malware and APTs can evade point-in-time defenses to penetrate networks. Security professionals must evolve their strategy in lockstep to...
- Will the Real Endpoint Threat Detection and Response Please Stand Up? This webinar explores new technologies & process for protecting endpoints from advanced attackers as well as the innovations that are pushing the envelope... All Internet White Papers | Webcasts
Our new bimonthly Internet of Things newsletter helps you keep pace with the rapidly evolving technologies, trends and developments related to the IoT. Subscribe now and stay up to date!