Even Consumer Reports finds smartphone ratings are tough
Rates iPhone 3GS on top, but not by much, in latest evaluation
Computerworld - When it comes to rating new smartphones, even one of the toughest and most objective rating sources, Consumer Reports, concedes that the task is difficult.
In its latest ratings available to subscribers, Consumer Reports listed the new iPhone 3GS on top but noted that other smartphones were close.
In making those ratings, however, Consumer Reports appears to have gone through plenty of late-night soul-searching about how to rate complex products like smartphones, recognizing that personal taste about which features matter most to users comes into play, as well as testing methodology.
"Smartphones are among the most complex products we test, and perhaps most subject to personal preference," Consumer Reports blogger Paul Reynolds wrote about the smartphone ratings.
Reynolds also said the magazine recently revamped its rating methods and scrapped its previous tests of wireless reception because the tests could "no longer adequately replicate real-life reception experience."
Again, the complexity of the smartphone technology came into play in that decision, he added. "Creating reliable reception tests has become steadily more challenging as smartphones add more tasks and use a growing range of reception technologies," he wrote. Consumers Reports is developing tests to more accurately evaluate reception, he noted, and is "wrestling with ways to evaluate a smartphone's speed and versatility -- growing concerns as the devices acquire more sophisticated operating systems and a host of third-party applications."
As for the actual ratings, according to the magazine's blog, the iPhone 3GS has "strong performance in everything but voice quality."
In an earlier head-to-head review of the iPhone 3GS and the Palm Pre, the magazine had found that the Pre and the BlackBerry Storm bested the iPhone 3GS in messaging, while the Pre was considered a "superior multitasker" with its card deck handling of multiple applications.
On its 100-point scale, the iPhone 3GS earned a 73, besting the iPhone 3G, the second-generation model, which got a 70. The BlackBerry Storm got 69 points, as did the T-Mobile G1, BlackBerry Bold and Samsung Epix. The Palm Pre and BlackBerry Curve were both rated 67.
Some of the smartphone problems mentioned in recent days on user forums apparently didn't come into account in the Consumer Reports ratings, which looked at display, navigation, voice quality, phoning, messaging, Web browsing, multimedia and battery life. For example, the iPhone 3GS got an "excellent" rating on battery life, even though users are seeing iPhone 3.0 software as a drain on batteries.
And the Pre has recently been rapped by users in forums for hardware glitches. Hardware quality isn't a major factor in the Consumer Reports ratings.
Consumer Reports rated voice quality "fair" on nearly all the top-rated smartphones but commented in its blog that it had recently changed its rating attributes to reduce the value of voice quality, "in part to reflect the growing importance of non-voice use of smartphones." As a consequence, Consumer Reports put more emphasis on display, ease of navigation, and multimedia and messaging capability.
Like any reviewer, critic or ratings group, Consumer Reports recognizes that one rap on a particular brand of smartphone will bring out objections from its fans in force on Internet forums. Past smartphone ratings, Reynolds wrote, "have prompted debate and sometimes disagreement."
Consumer Reports seems willing to leave those comments to its blogs, although the tone is more civil than comments on almost any other smartphone forum.
Read more about Mobile/Wireless in Computerworld's Mobile/Wireless Topic Center.
- 15 Non-Certified IT Skills Growing in Demand
- How 19 Tech Titans Target Healthcare
- Twitter Suffering From Growing Pains (and Facebook Comparisons)
- Agile Comes to Data Integration
- Slideshow: 7 security mistakes people make with their mobile device
- iOS vs. Android: Which is more secure?
- 11 sure signs you've been hacked
- The Critical Role of Support in Your Enterprise Mobility Management Strategy Most business leaders underestimate the importance of tech support when they choose an EMM solution. Here's what to put on your checklist.
- Separating Work and Personal at the Platform Level: How BlackBerry Balance Works BlackBerry® Balance™ separates work from personal on the same mobile device, right at a platform level. Find out how it can work for...
- Protection for Every Enterprise: How BlackBerry Security Works Get an IT-level review of BlackBerry® Security, addressing data leakage protection, certified encryption, containerization and much more.
- Future Focus: What's Coming in Enterprise Mobility Management (EMM) Find out why Enterprise Mobility Management (EMM) solutions that are truly future-ready must be designed to enable Machine-to-Machine (M2M) capabilities and much more.
- Live Webcast On-demand webinar: "Mobility Mayhem: Balancing BYOD with Enterprise Security" Check out this on-demand webinar to hear Sophos senior security expert John Shier deep dive into how BYOD impacts your enterprise security strategy...
- Live Webcast Unmasking the Differences between Consumer and Enterprise File Sync & Share The consumerization of IT combined with the rapid pace of the modern mobile workplace is forcing enterprise IT teams to evaluate file sync...
- Live Webcast Workforce Mobilization for Improved Productivity A mobility research director from Aberdeen discusses reasons for extending legacy applications to mobile devices, and an integration strategist from Attachmate shows how...
- Getting Ready for BlackBerry Enterprise Service 10.2 Find out how BlackBerry® Enterprise Service 10 helps organizations address the full spectrum of EMM challenges, while balancing the needs of both the...
- Containerization Options: How to Choose the Best DLP Solution for Your Organization This webcast outlines a framework for making the right choice when it comes to containerization approaches, along with the pros and cons of... All Mobile/Wireless White Papers | Webcasts