Mining the Deep Web: Search strategies that work
How to become an enlightened searcher
Computerworld - Just because a Web search engine can't find something doesn't mean it isn't there. You may be looking for info in all the wrong places.
The Deep Web is a vast information repository not always indexed by automated search engines but readily accessible to enlightened individuals.
The Shallow Web, also known as the Surface Web or Static Web, is a collection of Web sites indexed by automated search engines. A search engine bot or Web crawler follows URL links, indexes the content and then relays the results back to search engine central for consolidation and user query. Ideally, the process eventually scours the entire Web, subject to vendor time and storage constraints.
The crux of the process lies in the indexing. A bot does not report what it can't index. This was a minor issue when the early Web consisted primarily of static generic HTML code, but contemporary Web sites now contain multimedia, scripts and other forms of dynamic content.
The Deep Web consists of Web pages that search engines cannot or will not index. The popular term "Invisible Web" is actually a misnomer, because the information is not invisible, it's just not bot indexed. Depending on whom you ask, the Deep Web is five to 500 times as vast as the Shallow Web, thus making it an immense and extraordinary online resource. Do the math: If major search engines together index only 20% of the Web, then they miss 80% of the content.
What makes it deep?
Search engines typically do not index the following types of Web sites:
- Proprietary sites
- Sites requiring a registration
- Sites with scripts
- Dynamic sites
- Ephemeral sites
- Sites blocked by local webmasters
- Sites blocked by search engine policy
- Sites with special formats
- Searchable databases
Dynamic Web sites are created on demand and have no existence prior to the query and limited existence afterward (e.g., airline schedules).
If you ever noticed an interesting link on a news site, but were unable to find it later in the day, then you have encountered an ephemeral Web site.
Webmasters can request that their sites not be indexed (Robot Exclusion Protocol), and some search engines skip sites based on their own inscrutable corporate policies. Not long ago, search engines could not index files in PDF, thus missing an enormous quantity of vendor white papers and technical reports, not to mention government documents. Special formats become less of an issue as index engines become smarter.
- Capabilities You Need in an IP Address Management Solution A mismanaged IP space can cripple an otherwise healthy network. Take a moment to understand what you need in an enterprise-ready IPAM solution.
- IPv6 Fundamentals IPv6 is needed to sustain the growth of the Internet. The transition from IPv4 will require planning and likely some degree of support...
- Fixing Intermittent Performance Problems Intermittent performance problems are among the most frustrating and time-consuming issues IT administrators face. Read this white paper and learn how technology advances...
- 3G/4G Digital Signage Guide Today, the widespread availability of 3G and 4G cellular or wireless broadband networks enables digital signage to be deployed virtually anywhere.
- Live Webcast 5 Steps to Assuring Quality of Experience In order to align monitoring and management practices with the true demands of the business, IT professionals must expand beyond traditional comfort zones...
- Live Webcast Master the Changing SAP Landscape with Performance Management SAP landscapes are not getting simpler. Gradually, business processes that used to be contained on a single SAP system now involve a range...
- Navigating the New Wireless Landscape Thriving in the new wireless landscape View Now>>
- Deep Dive into Advanced Networking and Security with Hybrid Cloud Security and networking are among the top concerns when moving workloads to the cloud. VMware vCloud® Hybrid Service™ enables you to extend your... All Networking White Papers | Webcasts
Our new bimonthly Internet of Things newsletter helps you keep pace with the rapidly evolving technologies, trends and developments related to the IoT. Subscribe now and stay up to date!