Even Consumer Reports finds smartphone ratings are tough

Computerworld - When it comes to rating new smartphones, even one of the toughest and most objective rating sources, Consumer Reports, concedes that the task is difficult.

In its latest ratings available to subscribers, Consumer Reports listed the new iPhone 3GS on top but noted that other smartphones were close.

In making those ratings, however, Consumer Reports appears to have gone through plenty of late-night soul-searching about how to rate complex products like smartphones, recognizing that personal taste about which features matter most to users comes into play, as well as testing methodology.

"Smartphones are among the most complex products we test, and perhaps most subject to personal preference," Consumer Reports blogger Paul Reynolds wrote about the smartphone ratings.

Reynolds also said the magazine recently revamped its rating methods and scrapped its previous tests of wireless reception because the tests could "no longer adequately replicate real-life reception experience."

Again, the complexity of the smartphone technology came into play in that decision, he added. "Creating reliable reception tests has become steadily more challenging as smartphones add more tasks and use a growing range of reception technologies," he wrote. Consumers Reports is developing tests to more accurately evaluate reception, he noted, and is "wrestling with ways to evaluate a smartphone's speed and versatility -- growing concerns as the devices acquire more sophisticated operating systems and a host of third-party applications."

As for the actual ratings, according to the magazine's blog, the iPhone 3GS has "strong performance in everything but voice quality."

In an earlier head-to-head review of the iPhone 3GS and the Palm Pre, the magazine had found that the Pre and the BlackBerry Storm bested the iPhone 3GS in messaging, while the Pre was considered a "superior multitasker" with its card deck handling of multiple applications.

On its 100-point scale, the iPhone 3GS earned a 73, besting the iPhone 3G, the second-generation model, which got a 70. The BlackBerry Storm got 69 points, as did the T-Mobile G1, BlackBerry Bold and Samsung Epix. The Palm Pre and BlackBerry Curve were both rated 67.

Some of the smartphone problems mentioned in recent days on user forums apparently didn't come into account in the Consumer Reports ratings, which looked at display, navigation, voice quality, phoning, messaging, Web browsing, multimedia and battery life. For example, the iPhone 3GS got an "excellent" rating on battery life, even though users are seeing iPhone 3.0 software as a drain on batteries.

And the Pre has recently been rapped by users in forums for hardware glitches. Hardware quality isn't a major factor in the Consumer Reports ratings.

Consumer Reports rated voice quality "fair" on nearly all the top-rated smartphones but commented in its blog that it had recently changed its rating attributes to reduce the value of voice quality, "in part to reflect the growing importance of non-voice use of smartphones." As a consequence, Consumer Reports put more emphasis on display, ease of navigation, and multimedia and messaging capability.

Like any reviewer, critic or ratings group, Consumer Reports recognizes that one rap on a particular brand of smartphone will bring out objections from its fans in force on Internet forums. Past smartphone ratings, Reynolds wrote, "have prompted debate and sometimes disagreement."

Consumer Reports seems willing to leave those comments to its blogs, although the tone is more civil than comments on almost any other smartphone forum.

Read more about Mobile and Wireless in Computerworld's Mobile and Wireless Topic Center.