"Facebook has lost millions of users per month in its biggest markets, independent data suggests, as alternative social networks attract the attention of those looking for fresh online playgrounds" says a story in yesterday's Guardian by Juliette Garside. True? Or mis-use of data?
According to the source of that "independent data," SocialBakers, the numbers show no such thing. "The bottom line here is that there is no story," SocialBakers CEO Jan Rezab posted this morning. In fact, SocialBakers said journalists should not use their data this way three months ago when the Guardian wrote a similar story.
Rezab outlines several problems with "jumping to conclusions" based on SocialBakers usage data, which, in Facebook's case, comes from Facebook's advertising platform:
* "Nobody has yet done any proper seasonality analysis" on social networking usage. In other words, usage may naturally drop during certain times of year and pick up at other times; but because growth has been so large over the past few years, it's been difficult to spot any time-based trends.
* Facebook updates usage data on its ad platform fairly regularly while most other social networks do not regularly release fresh data. Thus, trying to compare data from different networks at regular intervals is difficult.
* Facebook's data comes from its Monthly Active User (MAU) measure, which Rezab calls "one of the fairest methods of measurement in the industry." I'm not quite sure I agree on that: Is one check-in per month really a reasonable measure of a "user" for a social network? If someone checked their email once a month, would you find that a useful way to reach them? Probably not. Same goes for a social network: Advertisers may be interested in those eyeballs; but if someone only checks in once a month, chances are that their circle of friends in the real world wouldn't consider Facebook a useful platform to reach them.
In any case, Rezab says the count of monthly users is "statistically vulnerable to more casual users ... [who] might fall out of the 30-day range from time to time." I definitely disagree on this one, since a) if you're only using Facebook once every two or three months, you're no more a regular user than someone who looks at the Guardian once every two or three months can be called a "Guardian reader"; b) such casual users are also likely to not appear during one measurement period and then re-appear during another time period -- those users would be showing a growth in usage, not a decline.
Nevertheless, it's important to know how much variation is normal "noise," as Nate Silver might put it, versus how much is actually "signal" and an important trend. When the data source is telling you not to use the data to trend-spot, that certainly should be mentioned in the story.
Bottom line: It's fair to mention the drop in Facebook users shown by the SocialBakers data, but it's simply poor journalism not to mention that the source of your data disagrees with your conclusions -- especially since the source complained about this three months ago. If the writer disagrees with Rezab's analysis, she has a responsibility to both report and attempt to refute it, not simply ignore it.
See more from my Data Avenger series.