NASAs Open Data Policy

By Chris Preimesberger  |  Posted 2008-07-28 Print this article Print

This is all part of NASA's relatively new "open data" policy to share almost everything in the public domain, Bester said. This is a radical change from the Cold War years, in which any kind of scientific information was kept sacrosanct so that countries behind the so-called Iron Curtain couldn't use it for their own advancement.

"In fact, the entire data set is available to read on the Web site," Bester said. "It's a big flood of data that we have to handle, so it's rather complex. If somebody wants to analyze the data on their own, they are welcome to it."

Bester said the THEMIS project just bought a set of new, dedicated Linux-based RAID storage arrays and data servers from Silicon Mechanics-several terabytes' worth of raw storage-to handle the additional influx of satellite data from the five satellites. About 40GBs worth of film and photos is processed per evening, Angelopoulos said.

No custom-made software is being used for the data transfers, Bester said. It's all being beamed over the Internet using standard industry software from SGI and other companies.

"We are using some specially developed tools to provide access to the data for the general community," Bester said. "What you see on the Web site is sort of an overview of what the scientists see each day; it's compiled a bit for the general public."

The Berkeley operations center also is in the process of installing a new 20TB server from Sun Microsystems to store the THEMIS data.

"Our 10TB server nearly filled up with THEMIS science within about
18 months, averaging at about 130GB per week," Bester told me. "Most of those data are from the all-sky imagers. The five spacecraft combined generate about 3GB per week. Data from the five spacecraft are downloaded through ground stations at a rate of about 1Mb/ps when they pass near Earth."

The newest addition to the system is a SunFire X4500 Thumper 64-bit server running Solaris 10 OS on 2x AMD Opteron Model Dual-core 290 2.8 GHz Processors. It features 48 3.5-inch 1TB SATA hard drives.

Understanding Space Weather

Angelopoulos said the entire study of space weather is of increasing interest to governments all over the world.

"We are at the beginning of our understanding of space weather, much [as] 100 years ago we were at the beginning of our understanding of atmospheric weather," Angelopoulos told me. "Just like back then, scientists were using weather stations to track storm fronts, today we need to be using fleets of satellites to track space weather fronts.

"THEMIS ushered [in] a new era of satellites, launched to track 'weather fronts' in space and [that] have already paid off by their discovery of what triggers substorms. This will pay great dividends in modeling space weather and in developing tools for accurate and timely space weather prediction."

Editor's note: This story was updated Aug. 2 to include more detail about the IT system at the Berkeley operations center.

Editor's note (2):
To access general information and the data set of the THEMIS Project, go here. To view still photos, go here.

To view video of the aurora borealis, follow these instructions: Start here. Select "All Sky Imager" under "Source." Pick a date earlier this year and click display.

A pop-up appears, like this, if your computer is set to allow pop-ups. Click "Movie." You will see the aurora displayed against the image of

Chris Preimesberger Chris Preimesberger was named Editor-in-Chief of Features & Analysis at eWEEK in November 2011. Previously he served eWEEK as Senior Writer, covering a range of IT sectors that include data center systems, cloud computing, storage, virtualization, green IT, e-discovery and IT governance. His blog, Storage Station, is considered a go-to information source. Chris won a national Folio Award for magazine writing in November 2011 for a cover story on and CEO-founder Marc Benioff, and he has served as a judge for the SIIA Codie Awards since 2005. In previous IT journalism, Chris was a founding editor of both IT Manager's Journal and and was managing editor of Software Development magazine. His diverse resume also includes: sportswriter for the Los Angeles Daily News, covering NCAA and NBA basketball, television critic for the Palo Alto Times Tribune, and Sports Information Director at Stanford University. He has served as a correspondent for The Associated Press, covering Stanford and NCAA tournament basketball, since 1983. He has covered a number of major events, including the 1984 Democratic National Convention, a Presidential press conference at the White House in 1993, the Emmy Awards (three times), two Rose Bowls, the Fiesta Bowl, several NCAA men's and women's basketball tournaments, a Formula One Grand Prix auto race, a heavyweight boxing championship bout (Ali vs. Spinks, 1978), and the 1985 Super Bowl. A 1975 graduate of Pepperdine University in Malibu, Calif., Chris has won more than a dozen regional and national awards for his work. He and his wife, Rebecca, have four children and reside in Redwood City, Calif.Follow on Twitter: editingwhiz

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel