Google Testing New Storage System for 'Caffeine'

By Chris Preimesberger  |  Posted 2009-08-17 Print this article Print

This new storage system, the back end of the new Caffeine search engine that Google introduced Aug. 10 and is now testing, will include more diagnostic and historic data and autonomic software, so the system can think more for itself and solve problems long before human intervention is actually needed.

Google has been ahead of its time in more than just Web search and online consumer tools. Out of sheer necessity, it's also been way ahead of the curve in designing massive-scale storage systems built mostly on off-the-shelf servers, storage arrays and networking equipment.

As the world's largest Internet search company continues to grow at a breakneck pace, it is now in the process of creating its second custom-designed data storage file system in 10 years.

This new storage system, the back end of the new Caffeine search engine that Google introduced Aug. 10 and is now testing, will include more diagnostic and historic data and autonomic software, so the system can think more for itself and solve problems long before human intervention is actually needed.

Who knew 10 years ago, when it was the newbie on the block to Yahoo's market-leading search engine, that Google would grow into a staple of Internet organization that is relied upon by hundreds of millions of users each day?

Just before Rackable sold Google its first 10,000 servers in 1999 and started the company on a server-and-array collection rampage that may total in the hundreds of thousands of boxes, Google engineers were pretty much into making their own servers and storage arrays.

"In 1999, at the peak of the dot-com boom when everybody was buying nice Sun machines, we were buying bare motherboards, putting them on corkboard, and laying hard drives on top of it. This was not a reliable computing platform," Sean Quinlan, Google's lead software storage engineer, said with a laugh at a recent storage conference. "But this is what Google was built on top of."

It would be no surprise to any knowledgeable storage engineer that this rudimentary file system had major problems with overheating to go with numerous networking and PDU failures.

"Sometimes, 500 to 1,000 servers would disappear from the system and take hours to come back," Quinlan said. "And those were just the problems we expected. Then there are always those you didn't expect."

Eventually, Google engineers were able to get their own clustered storage file system-called, amazingly enough, Google File System (GFS)-up and running with decent performance to connect all these quickly custom-built servers and arrays. It consisted of what Quinlan called a "familiar interface, though not specifically Posix. We tend to cut corners and do our own thing at Google."

What Google was doing was simply taking a data center full of machines and layering a file system as an application across all the servers to get open/close/read/write, without really caring where the data is in the machine, Quinlan said.

But there was a big problem. The GFS lacked something very basic: automatic failover if the master went down. Admins had to manually restore the master, and Google went dark for as long as an hour at times. Although failover was later added, when it kicked in it was annoying to users because the lapse often was several minutes in length. Quinlan says it's down now to about 10 seconds.

Eventually, the growth of the company and its subsequent IPO in 2004 spurred even more growth, so a modification to the file system was designed and built. This was called BigTable (developed in 2005-06), a distributed database-like file system built atop GFS with its own "familiar" interface; Quinlan said it is not Microsoft SQL.

This is the part of the system that runs user-facing applications. There are hundreds of instances (called cells) of each of these systems, and each of those cells scales up into thousands of servers and petabytes of data, Quinlan said.

That's, ahem, a lot of storage space to govern.

At the base of much of this are Rackable's Eco-Logical storage servers, which are clustered to run on Linux to produce storage capacity as high as 273TB per cabinet. Of course, Google now uses a wide array of storage vendors because it's all but impossible for one vendor to supply the huge number of boxes needed by the search monster each year.

The Eco-Logical storage arrays feature high-efficiency, low-power consumption and intelligent design intended to improve price performance per watt, in even very complex computing environments, Geoffrey Noer, Rackable's senior director of product management, told eWEEK.

Chris Preimesberger Chris Preimesberger was named Editor-in-Chief of Features & Analysis at eWEEK in November 2011. Previously he served eWEEK as Senior Writer, covering a range of IT sectors that include data center systems, cloud computing, storage, virtualization, green IT, e-discovery and IT governance. His blog, Storage Station, is considered a go-to information source. Chris won a national Folio Award for magazine writing in November 2011 for a cover story on and CEO-founder Marc Benioff, and he has served as a judge for the SIIA Codie Awards since 2005. In previous IT journalism, Chris was a founding editor of both IT Manager's Journal and and was managing editor of Software Development magazine. His diverse resume also includes: sportswriter for the Los Angeles Daily News, covering NCAA and NBA basketball, television critic for the Palo Alto Times Tribune, and Sports Information Director at Stanford University. He has served as a correspondent for The Associated Press, covering Stanford and NCAA tournament basketball, since 1983. He has covered a number of major events, including the 1984 Democratic National Convention, a Presidential press conference at the White House in 1993, the Emmy Awards (three times), two Rose Bowls, the Fiesta Bowl, several NCAA men's and women's basketball tournaments, a Formula One Grand Prix auto race, a heavyweight boxing championship bout (Ali vs. Spinks, 1978), and the 1985 Super Bowl. A 1975 graduate of Pepperdine University in Malibu, Calif., Chris has won more than a dozen regional and national awards for his work. He and his wife, Rebecca, have four children and reside in Redwood City, Calif.Follow on Twitter: editingwhiz

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel