Breaking open Google's big black box

Stop me if you've heard this one.

Big publishers are complaining that Google gives short shrift to the originators of content, favoring the blogosphere over traditional mass media. A story by Nat Ives in today's Ad Age summarizes the complaints thusly:

Many publishers resent the criteria Google uses to pick top results, starting with the original PageRank formula that depended on how many links a page got. ..."You should not have a system," one content executive said, "where those who are essentially parasites off the true producers of content benefit disproportionately."

In other words: It's the repeaters, not the reporters, who get all the hits.

I think they're absolutely right. There's something wrong with a system that ranks a blog with "hey, here's a great story about nuclear-powered chickens" along with a few lines of vapid commentary at the top of the Google hit parade, while the original story about nuclear-powered chickens is buried on page 3 of the search results.

Naturally, some of the beneficiaries of this system - aka, the "A List Bloggers" -- beg to differ. Says Steve Rubel at Micro Persuasion:

I thought it was 2009. But when I read this I felt like it was 2004 all over again. ... First, the lines between media and blogs have been obliterated. What's TechCrunch or Engadget? Sure they are blogs but they run ads. So are they social media or media? ....[T]here's no greater friend to media companies than Google and bloggers... [I]n a world where links rule, the media companies need bloggers as well for traffic, credibility and more.

Now TechCrunch does produce a lot of original content, even if some of it seems pretty suspect at times. Engadget? Much more dependent on other sources for its "news."

As I write this, for example, the lead story on Engadget is "3D scanner made entirely from Legos." Engadget didn't originate this story, they found it on Make. But Make didn't make the story either; they source it to Dan's Data Blog (no relation to this Dan). That Dan, in turn, found it on Philohome.com. That seems to where it originated.

Who does Google reward with the hits for this? A search for "Legos 3D scanner" puts BoingBoing's Gadget pages at the top of the list, followed by Make. Philo's site? Number 29. The odds of anyone besides me tracking this piece back to the original source: Not bloody likely.

I know. This is how the Web works. And when big sites pick up your stuff, even a small blog will see a traffic spike. But this is very much like "trickle down" economics; in the end, most of the wealth stays at the top.

Rubel adds this:

A neutral Google is a good Google. They should continue to deliver an algorithm that rewards the highest quality sources that have earned a following, interest and links from other sources.

Absolutely. But let us not pretend for a moment that Google is a) entirely neutral, b) rewarding the highest quality sources, or c) doing this for the greater good. This is not a benevolent dictatorship. The blogosphere is much more dependent on Google AdSense than traditional media, which still mostly relies on banner and display advertising. Featuring secondary sites ends up being better for Google's bottom line.

Google News is equally screwy in its own way, though most bloggers are shut out of that search (GN doesn't consider one-person outfits a valid news organization and thus doesn't index them for News searches).

Several times a day I scan the Google News headlines, and I am continually amazed and appalled at the stories that grace the top pages. It's rarely the original story from the original source; it's even more rare to find a story that adds value to the original. I know editors who pull their hair out (what's left of it) because their staff consistently produces original stories but can't catch a break from Google News. Meanwhile, someone else's brain-dead summary gets all the Google juice.

I think the solution is more transparency. I think Google needs to crack open the black box a bit and show us how the machine really works. As I've said before, without mainstream reporters doing the heavy lifting, much of the blogosphere would disappear overnight. That's not good for anybody.

When not gorging himself on March Madness online, Dan Tynan tends his wee blogs, Culture Crash and Tynan on Tech.

Copyright © 2009 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon