eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Google has taken the unusual step of removing some of its own Web pages from its Web index after reports that the pages used a search-optimization method frowned upon by the search engine.
Search-engine watchers this week widely noted the appearance of so-called “cloaking” on a set of user support pages on Googles AdWords advertising program. The pages were displaying different title information to Googles Web crawler than to regular visitors to the pages.
In one specific example cited in discussions at the Threadwatch Weblog, a support page about an AdWords traffic-estimating tool had displayed a title to the Google crawler that included keywords such as “traffic estimator” and “traffic estimate” and ranked high in Google search results.
Some Webmasters use cloaking as a way to rank higher in search-engine results, often by feeding keywords to crawlers. But Google, in particular, is critical of the practice. In its guidelines for Webmasters, the Mountain View, Calif., company warns that it may permanently remove sites from its index that engage in cloaking.
Late Tuesday, Google officials confirmed the discrepancy between what the AdWords pages were displaying to the Google crawler compared to what other visitors saw and attributed it to an internal mistake.
The additional keywords were meant only for the internal crawler serving Googles site search, spokesman Barry Schnitt said.
“We inadvertently showed additional information on product support pages to both Googles site search crawler and Googles main web crawler,” Schnitt said in a statement. “We are in the process of making a technical change so that the pages show only the information available to users.”
By Wednesday, Google instead had begun removing the AdWords pages from its Web index. A Web posting from “GoogleGuy” appeared on a series of search-engine marketing forums explaining the error and confirming the removal. A Google spokesman verified the authenticity of the posting.
“To be consistent with our guidelines, were removing these pages from our index,” the Google representative wrote in the posting. “Once the pages are fully changed, people will have to follow the same procedure that anyone else would.”
Webmasters who find their pages removed from Googles index must send Google a request to be included again. Presumably, the Google employees overseeing the AdWords pages will need to follow the same procedure.
Googles mishap appears to be more of an embarrassment for the search engine than a concerted effort to game its own technology, said Danny Sullivan, editor of Search Engine Watch.
Still, the episode raises the question of whether Google should rethink its policy on cloaking. While cloaking can be abused as a way to spam search crawlers, it also can have legitimate uses, Sullivan said.
Google has made exceptions to its cloaking rule in the past. For example, its crawler views password-protected content in the Google Scholar program that cannot be viewed by broad Web users, and indexes National Public Radio audio transcripts that are unavailable to Web users, Sullivan said.
“The bigger issue is not that Google is breaking its own rules but that Google already allowed people to break the rules and should just get rid of the rules,” Sullivan said. “Theyre so black-and-white on it right now.”
He suggested that Google tweak its guidelines to let Webmasters use cloaking if they receive Google permission when it improves results. Google officials did not say whether they plan to revisit the search engines cloaking policies.