Web site operators are clamoring to understand what can best be described as an ongoing disturbance in the Google Force.
Google’s search engine, once a clean, lean indexing machine, from a Webmaster’s perspective has been slipping badly lately.
Starting about two months ago, there have been scores of people complaining that their Web sites have suddenly disappeared from Google’s index for no reason, tantamount to disappearing from the Internet.
Another common gripe is how a Web link mysteriously drops from a prominent place in search results to page 43 or so. The list of complaints goes on.
Something’s clearly going on, on which Google hasn’t yet commented, and likely won’t on Press Day, its annual rite scheduled May 10 in which Google usually makes a number of impressive announcements. This year, there are some “product demos” to consider, it promised.
But in the backdrop of all the shiny new Google stuff is the 800-pound gorilla of a question: What’s going on? Many people suggest that Big Daddy’s to blame.
Big Daddy is, in effect, a brand-new data center, which Google uses to perform core search engine tasks like cataloging Web sites, or serving up localized features. Google operates thousands of data centers across the globe.
The Big Daddy project stands out because search engines infrequently upgrade the computing and networking hardware. Rather, the focus is on new features to lure more site visitors, which translates to more search engine revenues.
By upgrading, Google can better compete against rivals Yahoo, the world’s leading Internet destination but second to Google in search, and Microsoft, whose online MSN unit manages the world’s No. 3 search engine.
The ultimate goal is a bigger slice of the $13.8 billion expected to be spent on Internet advertising in 2006, and even more in years going forward.
Every search engine is constantly tinkering, or buying new companies and absorbing their intellectual property. It’s all in order to find new ways of attracting an even bigger audience, which translates into higher advertising revenues.
But as all the Big Daddy disruptions seem to indicate, Google’s going through some growing pains now as it sharpens its competitive edge.
As screen-name JohnW wrote in a Searchenginewatch forum: Google “rolled out some new technology to fix some things, and in the process they seem to have broken some other things. There isn’t much to say about it other than it’s broken – it is what it is and will be over when it’s over.”
Because of Big Daddy, the theory goes, Google must completely erase the databases where it keeps its 8 billion plus Web pages. Then it’ll add Big Daddy and kick-start the indexing.
That explains why sites disappear, for instance, for a few days or weeks at a time. It’s Big Daddy knocking, said several posters at this Web operator forum (Password, username required).
Another theory about the disruptions is based on comments about a Google “server crisis” from Google CEO Eric Schmidt to the New York Times.
From those words spun a rumor that Google’s plumb out of data storage. That too explains the indexing issues.
There’s been lots of speculation, and some quasi-confirmation from a prominent Google engineer, but Google didn’t immediately respond to an e-mail seeking comment on the topic.
Much of what’s known about Big Daddy comes from Google engineer Matt Cutts, who occasionally writes about the topic on his blog.
There’s an oft-cited March 27 Cutts-agram sent at the height of the initial wave of complaints. In it, he claimed that Big Daddy’s been fully deployed and he mentioned how it’ll “stabilize.”
He also warned that sites finding themselves off Google’s reservation can expect it to take a few days to show up in the index again.
