Even as the major search engines have expanded their indexes of the Web to as many as 8 billion documents, they are increasingly acknowledging that much of the worlds information is nowhere to be found in search results.
To combat that problem, search companies as large as Yahoo Inc. and Google Inc. and as small as Groxis Inc. and Mamma.com Inc. are attempting to put a more public face on Web content previously stuck behind subscription-based sites or hidden deep within a Web site.
The sudden interest in searching subscription and other hidden content may say more about the state of the online content industry than it says about search technology.
Making premium Web content available through search is far from a new concept, since even Yahoo experimented with it a few years ago.
What is changing are the attitudes of the providers of premium content, from magazines and newspapers to research databases and traditional aggregators, say analysts and search engine executives.
“I see this as part of an emerging need to go beyond what is available on the Web today,” said Allen Weiner, a research director at Gartner Inc. “Search has plateaued.”
Yahoo earlier this month began an experimental service for searching subscription-based content from publisher sites and databases, including the Wall Street Journal Online, Consumer Reports, the IEEE (Institute of Electrical and Electronics Engineers Inc.) and Forrester Research Inc.
Groxis on Tuesday plans to launch a pilot of search site called Grokker Research that uncovers results from premium business and research content. Groxis is making the service available as a free trial for about six months and then plans to charge for access to the premium content, said R.J. Pittman, president and CEO of the San Francisco-based company.
Grokker Research initially will draw subscription content from Ebsco Information Services Inc.s database and the IEEE.
Meanwhile, speculation spread last week that Google Inc. also is talking with content providers about its plans for searching subscription-based content. BetaNews reported on the plans, citing unnamed sources, but Google didnt return a request for comment on the report.
While Mamma.com is not focusing on subscription content, the company earlier this month did venture into an area being called the “deep Web,” with a health search site that returns detailed results with on-the-fly crawling of selected health sites.
Called Mamma.com Health Search, it is the first of between two and four additional specialized search sites the company plans to launch this year, said Guy Faure, CEO of Montreal-based Mamma.com.
The health search site focuses its crawl on about seven health sites chosen by Mamma.com. These include WebMD, Health AtoZ, MedlinePlus and NHS Direct. Mamma.com then essentially excerpts and categorizes information from the sites into sections such as symptoms, treatments and news on the results page.
To Weiner, the sudden resurgence of interest in making subscription and more hidden content accessible through major search engines represents an industry shift. Even six months ago, many of the major premium content providers remained reluctant to work with companies like Google or Yahoo.
“I see this as a major, major deal,” Weiner said. “To me the other shoe has dropped. Search companies knew that to increase quality of their results they needed access to this content, and, on the other hand, the premium content companies realized that the Yahoos and Googles of the world were not going away and that they would be an important channel for them.”
In the future, Weiner expects the search engines to do more than simply act as a conduit for current subscribers to reach premium content from other sites. In Yahoos beta test of its Yahoo Search Subscriptions, for example, users must be subscribers to publications in order to click through to an article.
Eventually, search sites are likely to also provide one-time, paid access to subscription content, Weiner said. The benefit for premium content providers would be the potential to gain more subscribers.
As Groxis has worked to form partnerships with publishers for its subscription search service, the company also has noticed a shift in the perception of search engines, Pittman said.
Groxis is targeting its service to librarians and corporate researchers, and Pittman said both publishers and researchers are growing more interested in working with search engines.
“The world has evolved, and it will continue to evolve around search and research,” Pittman said. “People are finally understanding the value of going beyond the top 10 search results.”
