Search Industry Races to Uncover the Deep Web

February 24, 2009 05:59 PM
by Liz Colville
The “Deep Web”—billions of Web pages either not indexed or buried deep in search results—could yield better results and more satisfied Web users if harnessed effectively.

Mining the Deep Web

The Deep Web, also called the Invisible Web, is something few people come across in their daily Web surfing. But several companies are making inroads into the Deep Web to harness and deliver more of the Internet to users, reports Alex Wright in The New York Times.

The goal is to make the entire Web available to more people in a way that is organized, intuitive and useful. Currently, a standard Web search will reveal just a small amount of the information that might actually be relevant to the searcher. Changing search terms, trawling through pages of search results, and using other search engines and databases can uncover some of the Deep Web, but still only a fraction of it.

Computer scientist and consultant Mike Bergman, who is credited with coining the term “Deep Web,” told the Times that perfecting Deep Web search may have more to do with improving business than with “satisfying the whims of Web surfers.”

But the two are intricately connected, as businesses and advertisers try to make meaningful connections with Web users by providing them with more of the information they are looking for, rather than the guesses of a traditional search engine algorithm or that of a semantic search engine, which tries to intuit search results using natural language.

President Barack Obama has even alluded to the Deep Web in his efforts to make the government more transparent. As The Washington Post noted in December, “for years, the U.S. government, one of the world’s largest depositories of data, has been unwilling or unable to make millions of its Web pages accessible,” thus banishing the pages to the Deep Web.

Opinion & Analysis: Who will conquer the Deep Web?

According to an article in Huliq, search engine optimization (SEO) must either be changed or embraced by all Web sites in order for the Deep Web to be conquered. In the article, the SEO firm Buzzwords argues that the current search engine process is “flawed” because the “the best site content for a particular search may appear on Page 10 of a search engine. … What surface instead are the seriously massaged results of the SEO-aware.”

Many Web technology experts acknowledge Google is poised to tackle the Deep Web, but others see the company as being in a difficult place. Yahoo chief search strategist Prabhakar Raghavan told Randall Stross in The New York Times that Google searching often result in a “digression,” adding that, “People want to complete a task.”

As Sarah Perez of ReadWriteWeb wrote earlier this month, there is no clear winner emerging in the Deep Web race. Perez notes that, because of how engrained Google is in so many people’s online search activity, “if better search technology for indexing the Deep Web comes into existence outside of Google, the world may not end up using it until such point Google either duplicates or acquires the invention.”

Reference: Semantic search, the Deep Web, SEO, Invisible resources


Most Recent Beyond The Headlines