Grub-by open source searching (and weirdest mating ritual)

It's Ruby Tuesday's IT Blogwatch: in which Wikia buys Grub, in Jimmy Wales' bid to take over the world's knowledge. Not to mention the courtship dance of the waved albatross...

Ryan Paul reports:

Wikia, the company created by Wikipedia founder Jimmy Wales, has acquired the Grub distributed indexing system from LookSmart and is preparing to distribute Grub's code under an open-source license. Wikia plans to use Grub for its user-driven search engine, which is still under development.

Originally created in 2000, Grub leverages the distributed computing model to crawl the web and index pages. Users install a specialized client application on their computer, which then automatically performs indexing while idle and transmits page data back to a centralized repository. In this manner, volunteers will contribute the raw computing power that performs the indexing.
In addition to leveraging volunteer computing power for automated indexing, Wikia's search engine will also attempt to take advantage of human power for index editing and refinement. According to Wales, users of the Wikia search engine will be involved in adding and removing links, removing spam, and policing other users much like the participatory model used by Wikipedia today. [more]

Farhad Manjoo adds:

You can download the Grub client to make your own computer pitch in on the indexing work. While you're not using it, the machine will scan the Web and send back its index to a central server; your scan, combined with input from others running the Grub client, will form the index that will power Wikia's open-source search engine.
Unlike Wikipedia, Wikia's search engine will run as a for-profit venture ...  Wikia sets out several guidelines for its open engine: It will be transparent -- the algorithms determining how results are ranked will be visible to all. Google and other engines invest huge sums to develop these algorithms, and they guard them extremely closely. But that's precisely why Wales believes we need an open search engine -- the world, he says, must have an alternative to a Web that's ranked by "invisible rules inside an algorithmic black box."

But Wales isn't looking for transparency for transparency's sake: the project rests on the idea that community involvement will actually improve upon today's search results. Whether that's possible seems a gamble. [more]

With the ugliest Web page ever, it's Doug Caverly:

You may not have heard of Grub, but the Web crawler company was just acquired by Wikia.  And now Jimmy Wales, founder of Wikipedia and co-founder of Wikia, thinks he may “change the balance of power from the search companies back to the publishers.”

Wales isn’t throwing around the term “Google killer” or anything like that, however ... [but] if anybody can challenge Google and its $160 billion market cap, Jimmy Wales has to be one of the leading candidates. [more]

Kord Campbell pops his head up:

I'm the guy that started Grub back in 1999. In 2003, after getting a little bit of press, I sold the company to LookSmart. I was hoping for a continuation of the OS license for Grub, and the financial backing of a larger company that could help develop the product out to it's logical conclusion - distributed, open search.

Unfortunately that didn't happen with the situation, and I decided to move on to other opportunities. Now here I am again, and I fully support what Wikia is doing with Grub, and what their resources can do for the project and the problem it can solve.

Myself (Kord Campbell), Igor Stojanovski and Ledio Ago (both who work at Splunk BTW) are three original founders of Grub. We are now helping Wikia out with getting it up and running, and explaining how things work (or don't) and will continue spending a bit of time helping out where we can as the project matures. [more]

The self-styled ComputerGuru rants:

It’s official, Jimmy Wales has gone too far. The founder of Wikipedia is famous for tapping into users to generate content and traffic, building the internet’s largest chain/web of user-generated content and information ... Wales has clearly mastered the art of human exploitation more than anyone ever has.

Most people are content with things the way they are: after all, it’s for a good cause! Go ahead and exploit humans, we agree, if the end result brings as much benefit and knowledge to the world as Wikipedia did and does. However, earlier this year, Wikipedia did a couple of things that created an outrage in the online community ... Now Wales is at it again, this time though, it’s not the greater good that’ll be benefiting: Jimmy Wales is now using end-users to bring in the cash! ... Instead of doing the web-crawling and indexing on Wikia’s servers, Wales wants end users to do the crawling for him!

Yep, you read that right. Wales ... intends to use [Grub] (and the internet connections and CPUs of end-users) to power his for-profit corporation. Users could be doing something useful with their hard-earned CPU cycles, electricity, and internet connections such as finding a cure for cancer or even aiding the search for alien life, Wales wants users to focus on making him more money.. [more]

Et Andy Beal, Brute?:

Of course, Wales forgets to mention that Wikipedia wouldn’t have anywhere near the success if its content didn’t appear in the top 5 of Google for just about every generic keyword you can imagine. It also slipped his mind that he has a track record of not exactly giving back to those that helped make it a success - they’ll happily “assimilate” your web content, just don’t expect a link back in the process.

I wish Jimmy Wales the best of luck, but if he really wants to change the world, why not improve the non-profit Wikipedia? If his plans involve Wikia - which means he’s in this for the money - spare us the sanctimonious spin. [more]

Buffer overflow:

Around the Net Around Computerworld Previously in IT Blogwatch

And finally... The ridiculous mating dance of the waved albatross

Richi Jennings is an independent technology and marketing consultant, specializing in email, blogging, Linux, and computer security. A 20 year, cross-functional IT veteran, he is also an analyst at Ferris Research. Contact Richi at

Copyright © 2007 IDG Communications, Inc.

Shop Tech Products at Amazon