Google faces more than just a new rival in Wikia

IDG News Service -

Google Inc. and other search engines face far more than just a new rival in Wikia Inc.; they face the prospect of hundreds, even thousands of new competitors.

The entire search-engine project that Wikia is working on will enter the open-source domain, drastically reducing the cost for just about anyone to make a search engine, said Gil Penchina, the company's CEO. Instead of paying millions of dollars to index the Web and create the software to build a search page, new search companies will find these items free online thanks to the open-source and free software communities.

"In search, it still costs about $5 million to $10 million to build a site," said Penchina during an interview in Taipei. "We want to make it possible for anyone to build a search site for $500. We don't view Google as the competition; we view cost as the competition."

The project, which was started by Wikipedia co-founder Jimmy Wales, consists of four components: the indexing of the Web, developing a search engine application, developing an algorithm, and having people filter sites and rank results.

One of the most expensive components of a search engine is the effort needed to index the Web. Companies have to buy servers and software to crawl the Web looking at every page to create a comprehensive list of what's on the Web.

"Your average search start-up will spend over $1 million buying servers and collecting data. That's bad for a couple of reasons. One is that everyone spends millions of dollars doing what is essentially the same work, which is like writing an encyclopedia all over again. Well, what if all of that data was available over the GNU Free Documentation License, which is the free-content license? So our goal is to make a crawl of the Web publicly available," Penchina said.

The cost of indexing the Web is one of the main hurdles to starting a search engine, and for-profit companies have raised the bar year after year by indexing the Web more and more often. It used to be cataloged once a week, or once a day. Now it's once an hour or even more often. The high cost of running these crawls has become a competitive weapon.

Wikia believes its crawl of the Web will cost nearly nothing because it's asking Internet users to help by downloading Web crawling software from Grub, which will use their computers during idle time to crawl the Web, and send results back to Wikia for the index. So far, a thousand people have downloaded the application, and Penchina is hoping for 100,000 or more. The goal is to post the entire index online, as well as regular updates, so anyone can use them.