Archive for April, 2009

“The Search” by John Batelle — Chapters 1-3

April 20, 2009

In The Search, John Batelle does a great job of putting the concept of search into perspective.  He explains it largely against the backdrop of Google, but emphasizes that it is a bigger and more important concept than any one company.  He explains the importance of search through his self-coined metaphor of the “Database of Intentions,” which he defines as “the aggregate results of every search ever entered, every result list ever tendered, and every path taken as a result.”

            Indeed I think that the biggest thrill for everyone the first time they used the internet was the ability to search for any topic of interest to them and instantly find information about it that previously would have required a trip to the library, a call to a subject expert, or some other much more laborious, time-consuming, and potentially expensive method of inquiry.  Indeed, ordinary people might never have gained access to the endless sources of information that are now available with just a few keystrokes’ worth of effort.

            In introducing these ideas and the promise of search, Batelle also foreshadows some of search’s potential complications and downsides – namely the fact that the ability to track and record our personal “clickstreams,” which we once took for granted to be private and fleeting (and the incentive to do so from a commercial perspective, by providing desired products and services based on preferences inferred from those clickstreams) raises significant privacy concerns.  Interestingly, a company called Yauba is now marketing a privacy-sensitive search engine to people concerned with such privacy issues. 

            He then briefly surveys the basic mechanics of search – who does it; where; when; why, (i.e. for both “recovery of information we know and “discovery” of information that we do not);, how it works; and how much money is at stake (which turns out to be quite substantial).  He observes that, although most of the money generated by search so far has revolved around “matching text ads with the intent of a search query,” other methods are being developed that could accelerate the search-generated spending, from local search (which Google is now using even for generic [non-local] searches) to behavioral targeting to search personalization.

            Batelle recounts the development of the first effective search engine and the irony of the fact that it was developed by a hardware company (DEC) in an effort that was focused on selling more hardware, not on developing search capability.  Ironically, the company management didn’t recognize the larger potential of its AltaVista search engine, even as it became #1 in search in 1997 and was competing with AOL and Yahoo to be the most important Web destination.  Following DEC’s acquisition by Compaq, there was a change in focus to portal capability (competing with Excite and Yahoo) as opposed to search capability.  A few ill-timed and aborted IPO attempts later, AltaVista missed its moment and was overshadowed by Google.

            Other companies made similarly notable contributions to the development of search as well as similar corporate missteps that led them off the path of search dominance – from Lycos (which first incorporated links to a website as a measure of relevance, as well as the use of not just links but also web page summaries in search results) to Excite (which first offered web page personalization and customization and free e-mail).

            On the other hand, Yahoo gained its foothold based on a hierarchical approach to organizing information on the web – organizing into categories and subcategories, a passive approach that made sense before people were fluent with the web and came to it proactively seeking information.  Like many others, Yahoo focused its efforts on being a portal for web traffic, only gradually incorporating search capabilities, leaving the way clear for someone – ultimately Google – to come along and focus on search.

            Ironically, what all of these companies have done and continue to do is in itself a search – a search for how best to search .  Google and their competitors are trying to make our search for information better, and to do so they are trying to mimic the way we think.  Understanding how we think will allow them to anticipate how we search and what we’re searchng for, which in the end is what the promise of the internet is all about.  This has been the internet’s appeal since the beginning – the ability to search, to find the information we need, when we need it, without ever leaving the comfort of our own home or office.