eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Google unleashed its Caffeine Web indexing system June 8, promising 50 percent fresher search results from its last index and more content than ever.
Caffeine comes after 10 months of rigorous testing, first within Google, then within a single data center before being rolled out across all of Google’s data centers. The reason for this is that it represents a fresh approach from the previous indexing system.
This so-called Decaf index used several layers, some of which were refreshed faster than others, with the main layer updating every couple of weeks. Refreshes took awhile to process before users would see the fresh content.
Caffeine analyzes the Web in bits and pieces, processing hundreds of thousands of Web pages in parallel and updating regularly around the world so new pages, or new information on existing pages, are added straightaway.
For search industry watchers, a fresh indexing system is bad news for rivals such as Microsoft Bing and Yahoo, both of which are combining forces to pit 30 percent market share versus Google’s 65 percent.
While Caffeine makes Google search faster than that of Bing or Yahoo, Google has resisted the notion that its work has anything but Google users in mind.
Indeed, Google software engineer Carrie Grimes wrote in a blog post:
“Content on the Web is blossoming. It’s growing not just in size and numbers but with the advent of video, images, news and real-time updates, the average Web page is richer and more complex. In addition, people’s expectations for search are higher than they used to be. Searchers want to find the latest relevant content and publishers expect to be found the instant they publish.”
Google said Caffeine takes up nearly 100 million gigabytes of storage in one database and adds hundreds of thousands of gigabytes of fresh content per day.
That’s a lot of storage, but for practical perspective, Grimes noted: “You would need 625,000 of the largest iPods to store that much information; if these were stacked end-to-end they would go for more than 40 miles.”
Read more about Caffeine on TechMeme here, and see Caffeine mastermind and Google search quality specialist Matt Cutts take on the new Web index on Search Engine Land here.
The arrival of Caffeine on the back end means Google has completely overhauled its search service, as it comes one month after Google launched its fresh search user interface.
The UI adds several more search options in the left-hand rail to let users slice and dice results. Check them out here.