Google Responds to Cloaking Accusations

By Matthew Hicks  |  Posted 2005-03-09 Print this article Print

Google drops AdWords support pages from its Web index after they appeared to use a search-optimization approach that Google largely bans.

Google has taken the unusual step of removing some of its own Web pages from its Web index after reports that the pages used a search-optimization method frowned upon by the search engine. Search-engine watchers this week widely noted the appearance of so-called "cloaking" on a set of user support pages on Googles AdWords advertising program. The pages were displaying different title information to Googles Web crawler than to regular visitors to the pages. In one specific example cited in discussions at the Threadwatch Weblog, a support page about an AdWords traffic-estimating tool had displayed a title to the Google crawler that included keywords such as "traffic estimator" and "traffic estimate" and ranked high in Google search results.
Some Webmasters use cloaking as a way to rank higher in search-engine results, often by feeding keywords to crawlers. But Google, in particular, is critical of the practice. In its guidelines for Webmasters, the Mountain View, Calif., company warns that it may permanently remove sites from its index that engage in cloaking.
Late Tuesday, Google officials confirmed the discrepancy between what the AdWords pages were displaying to the Google crawler compared to what other visitors saw and attributed it to an internal mistake. The additional keywords were meant only for the internal crawler serving Googles site search, spokesman Barry Schnitt said. "We inadvertently showed additional information on product support pages to both Googles site search crawler and Googles main web crawler," Schnitt said in a statement. "We are in the process of making a technical change so that the pages show only the information available to users."
Click here to read more about the ranking battle between Webmasters and search engines. By Wednesday, Google instead had begun removing the AdWords pages from its Web index. A Web posting from "GoogleGuy" appeared on a series of search-engine marketing forums explaining the error and confirming the removal. A Google spokesman verified the authenticity of the posting. "To be consistent with our guidelines, were removing these pages from our index," the Google representative wrote in the posting. "Once the pages are fully changed, people will have to follow the same procedure that anyone else would." Webmasters who find their pages removed from Googles index must send Google a request to be included again. Presumably, the Google employees overseeing the AdWords pages will need to follow the same procedure. Googles mishap appears to be more of an embarrassment for the search engine than a concerted effort to game its own technology, said Danny Sullivan, editor of Search Engine Watch. Still, the episode raises the question of whether Google should rethink its policy on cloaking. While cloaking can be abused as a way to spam search crawlers, it also can have legitimate uses, Sullivan said. Google has made exceptions to its cloaking rule in the past. For example, its crawler views password-protected content in the Google Scholar program that cannot be viewed by broad Web users, and indexes National Public Radio audio transcripts that are unavailable to Web users, Sullivan said. "The bigger issue is not that Google is breaking its own rules but that Google already allowed people to break the rules and should just get rid of the rules," Sullivan said. "Theyre so black-and-white on it right now." He suggested that Google tweak its guidelines to let Webmasters use cloaking if they receive Google permission when it improves results. Google officials did not say whether they plan to revisit the search engines cloaking policies. Check out eWEEK.coms for the latest news, views and analysis on enterprise search technology.
Matthew Hicks As an online reporter for, Matt Hicks covers the fast-changing developments in Internet technologies. His coverage includes the growing field of Web conferencing software and services. With eight years as a business and technology journalist, Matt has gained insight into the market strategies of IT vendors as well as the needs of enterprise IT managers. He joined Ziff Davis in 1999 as a staff writer for the former Strategies section of eWEEK, where he wrote in-depth features about corporate strategies for e-business and enterprise software. In 2002, he moved to the News department at the magazine as a senior writer specializing in coverage of database software and enterprise networking. Later that year Matt started a yearlong fellowship in Washington, DC, after being awarded an American Political Science Association Congressional Fellowship for Journalist. As a fellow, he spent nine months working on policy issues, including technology policy, in for a Member of the U.S. House of Representatives. He rejoined Ziff Davis in August 2003 as a reporter dedicated to online coverage for Along with Web conferencing, he follows search engines, Web browsers, speech technology and the Internet domain-naming system.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel