New search engine takes 'DeepDyve' into the Dark Web
Engine offers eye into area of Web that traditional search engines can't penetrate
Computerworld - DeepDyve Inc. today announced that it has launched a free search engine that can be used to access databases, scholarly journals, unstructured information and other data sources in the so-called "Deep Web" or "Dark Web," where traditional search technologies don't work.
The DeepDyve search engine enables searches of the Dark Web to more easily find life sciences, patent and Wikipedia data. The new engine indexes 500 million pages, said DeepDyve, which was known as Infovell before changing its name today.
The company said it will soon start indexing physical sciences content in the areas of information technology, clean technology and energy. That will help it meet its goal of expanding its index to more than 1 billion pages by the end of the year.
Because much of the content on the Deep Web is made up of technical publications, databases, scholarly publications and unstructured data, it has been difficult for traditional search engines to access it. To tackle this problem, DeepDyve is partnering with publishers and other providers of those sources of information to gain access to content overlooked by other engines, the company added.
Google Inc. announced earlier this month that it is ratcheting up its focus on the Dark Web by adding the ability to search PDF documents. In April, Google had announced that it was trying to find a way for its search engine to index HTML forms such as drop-down boxes and select menus that are typically part of the Dark Web.
"According to IDC, more than 42 million consumers spend 25 hours per month online researching business and personal information, and they are frustrated with the results they get back and the tools they have to use," said William Park, CEO of DeepDyve, in a statement. "DeepDyve gives information-savvy consumers unparalleled access to quality information found only in the Deep Web, with features and functionality that make it easy to find, filter and organize their results."
The company said that its technology is designed to allow users to type in a few words or copy an entire article into a query box to find all related articles located in the Deep Web.
Chris Sherman, a blogger at Search Engine Land, said that DeepDyve's approach to scouring the Deep Web is innovative. He credited the company's chief scientists, who are veteran genomics researchers. To crack the genetic codes contained in DNA sequences, he noted, researchers must understand the hidden patterns in of massive amounts of data.
"DeepDyve takes a similar approach to understanding information on the Web," Sherman added. "Going far beyond basic keyword-based search, DeepDyve indexes every word in a document, but also computes the factorial combination of words and phrases in the document and uses some industrial-strength statistical techniques to assess the 'informational impact' of these combinations. In essence, this approach looks at the meaning of an entire document and uses that to compute relevance, rather than factors like snippets of text or anchor text in links pointing to documents."
DeepDyve isn't a threat to Google now, nor is it likely to be in the future, but its tool is good for people who want to do research in areas that DeepDyve indexes.
"DeepDyve also offers a genuinely different 'second opinion' of the Web if you're wanting to look beyond the top results returned by Google and the other major search engines," Sherman noted. "With its limited initial offering, DeepDyve has just scratched the surface of what's available on the invisible Web, albeit in a very useful way. However, truly cracking the invisible Web problem still seems like a distant dream."
Read more about Web Apps in Computerworld's Web Apps Topic Center.
- 15 Non-Certified IT Skills Growing in Demand
- How 19 Tech Titans Target Healthcare
- Twitter Suffering From Growing Pains (and Facebook Comparisons)
- Agile Comes to Data Integration
- Slideshow: 7 security mistakes people make with their mobile device
- iOS vs. Android: Which is more secure?
- 11 sure signs you've been hacked
- Face Time Anytime Real-time communications facilitates team collaboration from nearly anywhere in the world. With facts and figures you can use to justify an investment
- Riverbed Stingray Application Firewall: Securing Cloud Applications with a Distributed Web Application Firewall Responsibility over IT security is moving away from the network and IT infrastructure and to the application and software architecture itself. IT organizations...
- Now is the time to implement a video conference solution Video conferencing is getting a lot of buzz lately due to the recent cost decrease, making it tangible for many law firms. It's...
- Video drives engagement Achieving maximum results means building a solid platform and network infrastructure. As digital age unfolds, it's clear that the ability to communicate effectively...
- Meg Whitman presents Unlocking IT with Big Data During this Web Event you will hear Meg Whitman, President and CEO, HP discuss HAVEn - the #1 Big Data platform, as well...
- The New Way to Work Knowledge Vault This Knowledge Vault focuses on how, in today's increasingly virtual world, it's more important than ever to engage deeply with employees, suppliers, partners,... All Web Apps White Papers | Webcasts