RIM global outage caused by core switch failure; fix under way
Backup system also didn't work, RIM explains
Computerworld - BlackBerry service delays experienced by users around the world on Tuesday were caused by a core switch failure within the infrastructure of Research In Motion (RIM), the company said late Tuesday.
A RIM spokesman said service was beginning to be restored to normal around 2 p.m. Eastern time, although there would be further delays as backlogs in data are cleared. It was the second outage -- or "delay," as RIM put it -- in two days affecting users in numerous countries.
RIM's system is designed to failover to a backup switch, but the failover system "did not function as previously tested," according to a statement issued by RIM at 5 p.m. Eastern time.
When the failover did not function, a backlog of data was generated. The company is working to clear that backlog.
"RIM has failed again at what plagued them in past outages, which is to provide a comprehensive disaster recovery solution," Ken Dulaney, an analyst at Gartner, said after the cause of the outage had been made public.
While it's true that switches can fail, "there should be automatic ways in which the system recovers from this type of event," Dulaney said. "Any vendor who runs this type of mission-critical service must constantly be reviewing disaster recovery solutions."
The latest problems occurred in two phases, with a 12-hour outage Monday morning affecting some BlackBerry users in Europe, the Middle East and Africa, according to RIM. The company said that problem was fixed, but it didn't explain the cause.
Then at about 10 a.m. Eastern time Tuesday, wireless carriers in the U.K. and Egypt reported outages that continued for hours.
RIM said an hour later that the delays affected some customers in South America, Europe, the Middle East, Africa and India, but it didn't immediately offer an update about the underlying problem.
Tweets and other reports blamed a server outage in Slough, England, where RIM operates a data center, but the company would not comment on those reports. The Slough data center would serve much of Europe and the Middle East, analysts said. RIM also runs a data center near its headquarters in Waterloo, Ontario.
But a data center outage in the U.K. or Canada probably wouldn't explain service problems in South American countries, such as Brazil, Chile and Argentina, analysts noted.
RIM doesn't usually explain the cause of its outages and disruptions. In the past, those outages have lasted one or two days and have only affected a certain region of a country or a portion of a continent, not several continents as happened Monday and Tuesday.
In March 2010, there was an outage in both North America and the U.K. on Wi-Fi-ready BlackBerry devices that were not connected to Wi-Fi. A more severe December 2009 outage in North America was related to a BlackBerry Messenger update.
Wireless carriers, including T-Mobile UK and Vodafone Egypt had informed their customers via Twitter at 10 a.m. Eastern time Tuesday that RIM was working on the problems. T-Mobile UK said the question of whether or not customers were owed refunds would be a matter for RIM to address.
Users on BlackBerryForums.com also took note of the problems Monday. One contributor, MrTuck, reported that "50% of the population with BlackBerries across Europe, Middle East and Africa are unable use their Internet, BlackBery Messenger, Facebook Twitter, Email and other applications." He noted that calls and texts were working normally at the time.
Some U.K. reports said the Monday problem seemed to be related to BlackBerry Internet Service customers, who are mostly individual users and small businesses.
Dulaney said the problem Tuesday seemed to be related to both BIS and BlackBerry Enterprise Server, which is used by larger businesses with email and other functions routed through a server placed inside a corporation and its firewall for added management and security. An IDG News Service editor based in Paris who uses a BES server in Boston reported that he was not affected by the Monday outage. However, he said that he was affected by the Tuesday problems and could not receive email.
RIM has not said whether BlackBerry BIS or BES or both were affected in either Monday's or Tuesday's delays. RIM also didn't explain what it meant by a "delay," since some users posted comments indicating that they could not receive certain services at all.
Matt Hamblen covers mobile and wireless, smartphones and other handhelds, and wireless networking for Computerworld. Follow Matt on Twitter at @matthamblen, or subscribe to Matt's RSS feed. His email address is email@example.com.
Read more about Mobile/Wireless in Computerworld's Mobile/Wireless Topic Center.
- 15 Non-Certified IT Skills Growing in Demand
- How 19 Tech Titans Target Healthcare
- Twitter Suffering From Growing Pains (and Facebook Comparisons)
- Agile Comes to Data Integration
- Slideshow: 7 security mistakes people make with their mobile device
- iOS vs. Android: Which is more secure?
- 11 sure signs you've been hacked
- The Critical Role of Support in Your Enterprise Mobility Management Strategy Most business leaders underestimate the importance of tech support when they choose an EMM solution. Here's what to put on your checklist.
- Separating Work and Personal at the Platform Level: How BlackBerry Balance Works BlackBerry® Balance™ separates work from personal on the same mobile device, right at a platform level. Find out how it can work for...
- Protection for Every Enterprise: How BlackBerry Security Works Get an IT-level review of BlackBerry® Security, addressing data leakage protection, certified encryption, containerization and much more.
- Future Focus: What's Coming in Enterprise Mobility Management (EMM) Find out why Enterprise Mobility Management (EMM) solutions that are truly future-ready must be designed to enable Machine-to-Machine (M2M) capabilities and much more.
- Live Webcast On-demand webinar: "Mobility Mayhem: Balancing BYOD with Enterprise Security" Check out this on-demand webinar to hear Sophos senior security expert John Shier deep dive into how BYOD impacts your enterprise security strategy...
- Live Webcast Unmasking the Differences between Consumer and Enterprise File Sync & Share The consumerization of IT combined with the rapid pace of the modern mobile workplace is forcing enterprise IT teams to evaluate file sync...
- Live Webcast Workforce Mobilization for Improved Productivity A mobility research director from Aberdeen discusses reasons for extending legacy applications to mobile devices, and an integration strategist from Attachmate shows how...
- Getting Ready for BlackBerry Enterprise Service 10.2 Find out how BlackBerry® Enterprise Service 10 helps organizations address the full spectrum of EMM challenges, while balancing the needs of both the...
- Containerization Options: How to Choose the Best DLP Solution for Your Organization This webcast outlines a framework for making the right choice when it comes to containerization approaches, along with the pros and cons of... All Mobile/Wireless White Papers | Webcasts