Mining the Deep Web: Search strategies that work
How to become an enlightened searcher
Computerworld - Just because a Web search engine can't find something doesn't mean it isn't there. You may be looking for info in all the wrong places.
The Deep Web is a vast information repository not always indexed by automated search engines but readily accessible to enlightened individuals.
The Shallow Web, also known as the Surface Web or Static Web, is a collection of Web sites indexed by automated search engines. A search engine bot or Web crawler follows URL links, indexes the content and then relays the results back to search engine central for consolidation and user query. Ideally, the process eventually scours the entire Web, subject to vendor time and storage constraints.
The crux of the process lies in the indexing. A bot does not report what it can't index. This was a minor issue when the early Web consisted primarily of static generic HTML code, but contemporary Web sites now contain multimedia, scripts and other forms of dynamic content.
The Deep Web consists of Web pages that search engines cannot or will not index. The popular term "Invisible Web" is actually a misnomer, because the information is not invisible, it's just not bot indexed. Depending on whom you ask, the Deep Web is five to 500 times as vast as the Shallow Web, thus making it an immense and extraordinary online resource. Do the math: If major search engines together index only 20% of the Web, then they miss 80% of the content.
What makes it deep?
Search engines typically do not index the following types of Web sites:
- Proprietary sites
- Sites requiring a registration
- Sites with scripts
- Dynamic sites
- Ephemeral sites
- Sites blocked by local webmasters
- Sites blocked by search engine policy
- Sites with special formats
- Searchable databases
Dynamic Web sites are created on demand and have no existence prior to the query and limited existence afterward (e.g., airline schedules).
If you ever noticed an interesting link on a news site, but were unable to find it later in the day, then you have encountered an ephemeral Web site.
Webmasters can request that their sites not be indexed (Robot Exclusion Protocol), and some search engines skip sites based on their own inscrutable corporate policies. Not long ago, search engines could not index files in PDF, thus missing an enormous quantity of vendor white papers and technical reports, not to mention government documents. Special formats become less of an issue as index engines become smarter.
- CIOs Deliver Productivity Breakthroughs with Intelligent Digital Signage Retailers have long recognized the influence that digital signage provides over a shopper's point-of-purchase decision making process.
- Improving Business Value of WAN Optimization Want to achieve faster ROI with WAN optimization? Read the latest IDC report and discover how you can cut IT costs without compromising...
- Four Little-Known Ways WAN Optimization Can Benefit Your Organization Read this white paper to learn how far WAN optimization has come, and how to make this most of your investments by using...
- IDC ROI Infographic Trends such as evolving communication patterns, connection types, applications and bandwidth can have an impact on enterprise organizations. Learn how IT organizations can...
- Live Webcast IBM FlashSystem V840: Leveraging Software-Defined Flash to Drive Your Business With end-to-end, tightly integrated functionality and super-fast flash technology, products like IBM FlashSystem V840 Enterprise Performance Solution empower businesses to leverage the efficiency...
- IBM FlashSystem V840: Leveraging Software-Defined Flash to Drive Your Business With end-to-end, tightly integrated functionality and super-fast flash technology, products like IBM FlashSystem V840 Enterprise Performance Solution empower businesses to leverage the efficiency...
- Top 4 Digital Signage Fails Join RMG Networks for a look at four of the most common reasons digital signage fails in corporate businesses. Learn about strategies to... All Networking White Papers | Webcasts
Our new bimonthly Internet of Things newsletter helps you keep pace with the rapidly evolving technologies, trends and developments related to the IoT. Subscribe now and stay up to date!