I started using the Internet in the 1970s. It didn't look anything like it does today, and our search tools were, well, awful.
Still compared to what we started with, they were great. Before I ever turned my hand to writing, I put myself through graduate school by doing research on the very first online database systems: NASA RECON, Dialog, and OCLC. These systems, which are still around, are part of what's called the Matrix, and, no, I don't mean the movie. The Matrix, as defined by John S. Quarterman, is the superset of all interconnected networks. Now, unlike then, you can get to these networks over the Internet, but you'll find yourself blocked from getting very deep into them without permission.
As for the Internet itself, it didn't really have search tools then. It wasn't until the late 1980s that the Internet became searchable. For example, today, if you want to find a particular file, Google is your friend and sites like Mininova make finding BitTorrent files easy. When I started, we had to go through ftp file directories screen by screen and hope that the file was in there somewhere.
A major advance was Archie, which beginning in 1990 made it possible to search through a site's file directories. It was painful to use but, compared to what we had been dealing with it, using Archie was sheer pleasure.
An even bigger advance was Gopher, which made it possible to search through online databases and text files. With Archie, you really first had to have a clue that a file was somewhere in a given site. With Gopher, you could simply search and let the server worry about finding which site had the information you wanted.
While Gopher was being built in 1991, the Web was also being created. By 1993, just as Gopher was reaching its maturity, I thought the first real Internet search engine, WAIS (Wide Area Information System) was going to be more important then the Web! I was putting the cart before the horse. WAIS, like Archie and Gopher, exist now only as network archeological artifacts.
Then, the Web roared on to the scene, knocking out the earlier Internet search programs and online services like Prodigy, CompuServe and GEnie. As it happened it also blew away the last major attempt to set up an online service: Ziff Davis and The Washington Post's Interchange Online Network, where I was cheerfully working as a writer and community manager. Oh well, by then, late 93, I knew the Web was going to wipe away all the online services. Only AOL remains as any kind of major player.
Of course, someone had to make sense of the Web, and so the first generation of Web search engines made their appearance. By 1997, there were over half-a-dozen important ones. These included Alta Vista, my personal favorite of the early search engines; Excite; and Yahoo. While most of them are still around, only Yahoo is still important.
What happened to all of them was that Google simply did a better job of making search easy to do and manageable. The key to Google's overwhelming success was PageRank. With PageRank, Google rates the relevancy of Web pages to queries based not only on whether the pages contain the search terms, the technique used by all search engines, but also by how many relevant pages link to it.
It sounds simple. Well, now it sounds simple. Then, it was a groundbreaking advance and, for more than a decade, no one has managed to even come close to catching up with Google.
There are efforts afoot, such as Steve Ballmer's idiotic failure of an attempt to buy Yahoo, but that's rearranging the deck furniture on the Titanic. Even if Microsoft had acquired Yahoo, it wasn't going to seriously hurt Google.
There are other, more serious, attempts to unseat Google as the top search engine. For example, Powerset is trying to bring AI (artificial intelligence) to bear on search by enabling you to look for information with natural language inquiries. So, you might type in "Who is Sarah Palin?" Your answer will be a mini-bio of the Republican VP candidate from Wikipedia and then Web site links.
Then, of course, there are efforts like Cuil, where, the best you can say is, "Well, they tried."
Still some day, Powerset, or some new search engine will upset Google, I just don't see it happening anytime soon. One thing I do know though, you really can see the world as A.G. (After Google) and B.G. (Before Google). The Web changed the online world. Google gave us all maps to this new world and that's why it's so much more than just another search engine.