Online Video: Answers to Storage There Are

By Chris Preimesberger  |  Posted 2006-07-17

Online Video: Answers to Storage There Are

Everyone has a favorite wide-screen movie memory. Maybe it is Darth Vader breathing loudly through his words in "Star Wars"; Yoda offering sage advice to Luke Skywalker in "The Empire Strikes Back"; or Rhett Butler carrying Scarlett O'Hara up the stairs in "Gone with the Wind."

These images might be etched forever in our minds, but where do those film characters actually live now? And how are movie studios, television and cable networks, and video production companies handling the crushing influx of both raw and polished video that is filling up archives at an alarming rate?

The production and archiving of movie and television images is well into an industrial sea change. The characters that once lived in cans of film or videotape stored in dark vaults are now moving to safer and far more accessible lodging inside disk drives and digital tape archives.

That trend will only continue as video content-ranging from feature films to commercial surveillance video to homemade videos on YouTube-is increasingly distributed via the Internet.

Researcher IDC predicts that online video services will generate $1.7 billion in revenue in the United States by 2010, up from $230 million in 2005.

Companies ranging from YouTube, which inked a deal with NBC June 27; to Guba and Sony Pictures Home Entertainment, which partnered July 11; and ABC and Apple Computer have teamed on video distribution projects.

Add it up, and the demand for video storage is immense.

But before video can be distributed, it has to be digitized. That would be a lot easier, though, if it weren't for the work broadcasters and video producers must  do to get disparate storage systems working together. As it is, storage hardware manufacturers all have different proprietary APIs (application programming interfaces for digital engineers), and most are not in a hurry to standardize.

Still, the need to get these technologies playing nicely together is great. As it is, Yoda and all those other characters are quickly becoming "ingested"-the industry term for converting rolls of film into high-resolution digital files-into new digital homes that require unprecedented amounts of digital storage space. Yoda himself-along with all the other "Star Wars" characters, Indiana Jones and others-is well-ensconced in the vast digital archive at Industrial Light & Magic, located at Lucasfilm's new headquarters at the San Francisco Presidio overlooking the Golden Gate Bridge.

When a film is in production at ILM, about 500 full- and part-time artists pour out up to a terabyte or more of raw video each week that will eventually be edited into a finished product. All the pieces must be saved somewhere-and made easily accessible for editors. There's no such thing as a "cutting room floor" anymore.

Imation joins growing crowd of Blue-ray disc media makers. Click here to read more.

"Everything we do here at ILM is stored away somewhere, from the smallest storyboard sketch to a final finished scene, complete with all the special effects," said Mike Thompson, ILM's IT storage manager. "We throw exactly nothing away, ever."

Eventually, all of ILM's content will be digitized. Thompson is keeping up with the rest of the United States, a nation of digital pack rats, mainly out of necessity.

Businesses of all sizes are being required to save everything-every e-mail, instant message, photograph, data sheet and spreadsheet-mostly because of recently enacted regulatory and auditing measures such as the Sarbanes-Oxley Act. On the nondocument side, videocams, digital cameras, MP3 players and camera phones are so popular and relatively inexpensive that consumers are building up enormous digital loads on their home computers.

The law enforcement and private-security sectors are becoming video storage hogs, with 24/7 security cameras piling up enormous amounts of mostly uneventful-yet important-digital footage that must be retained.

Add to all this the fact that high-definition video -- which requires 4 gigabytes of digital capacity per frame, compared to standard video's 2GB -- is steadily moving into the mix, taking up more space and bandwidth to access it, meaning even more storage will be necessary in the near future to handle the multiplicity of content.

Worldwide revenues of external disk storage system factories continue to spiral up each quarter, growing by 10.3 percent-or $4.2 billion-in the first quarter of this year compared with a year ago, according to IDC.

Next Page: Storage casting call.

Storage Casting Call

The 2006 projection of nearly a $20 billion market is on a fast track to the $65 billion neighborhood by 2010, according to analysts.

Storage Casting Call

With that much money at stake, it is no surprise vendors are auditioning for a big storage role. Top-tier companies such as EMC, Hewlett-Packard, Quantum, Dell, Network Appliance, Seagate Technology, Sun Microsystems and Sony all have professional-level hardware and software to handle the task of storing video for ready-to-use, backup and archival functions.

Smaller companies such as Thomson/Grass Valley, ProMax Systems, G-Tech, Pinnacle, MedeaVideo and others also have plenty of customers.

Sun is banking heavily on selling a high volume of professional digital video and digital tape storage, backup, and archiving hardware and software in the next few years. CEO Jonathan Schwartz identified data and video storage as one of the three businesses he expects to lead Sun back into profitability after nearly five years of red ink.

Sun's new Sun Fire X4500 "Thumper" storage system, a NAS (network-attached storage) product package that includes "Galaxy" servers powered by Advanced Micro Devices Opteron chips and StorageTek backup, was announced July 11.

One 19-inch-wide, 7.5-inch-deep Thumper server contains 48 hot-swappable disk drives totaling as much as 24TB of storage.

To go with Thumper, Sun also has developed "Honeycomb," a software package that includes a new data capture process announced in May. StorageTek Titanium archive platforms are also part of the lineup.

Thanks to its $4.1 billion acquisition last year of StorageTek-long established as a digital video storage market leader in both tape and disk drives-Sun inherited some major-league TV and video customers, including The Walt Disney Co., which owns ABC, ESPN and several other networks; HBO and its vast domain; several Public Broadcasting Service stations; and a number of smaller, privately owned video production companies.

Most large video storage companies use servers and software from several vendors in different areas of their data centers. A recent visit to ABC's KGO-TV network affiliate in San Francisco turned up a data center that included no fewer than 10 storage server makes, including Dell, HP, Thomson/Grass Valley, StorageTek, NetApp and EMC's Clariion.

KGO-TV, like most other television or cable stations and networks, dedicates each server to a specific task, such as scheduling, continuity, play-to-air broadcast, ingesting, asset managing, backup and archiving, as well as for regular in-house IT work.

"The biggest problem in coordinating all these babies is the fact that there is no common API for tying them all together in one nice package," said KGO-TV IT Director Dave Graham. "We have to spend way too much time, really, programming these things to work the way we want to use them."

Are HD-DVD and Blue-ray dead formats? Click here to read more.

Why aren't video server manufacturers giving up on their proprietary APIs and working toward standards? It's because they don't have to, Graham said. "Some vendors, especially ones like Thomson/Grass Valley [a privately held, specialized TV station/network server maker] and Pinnacle (now owned by NetApp), know they have high-quality products and haven't made much effort to contribute to standards organizations," he said.

That may be changing, thanks to the influx of new competitors coming into the fold and eyeing their own slice of that coming $65 billion market.

Several high-end video storage customers said handling an ever-increasing flow of data means testing juggling skills. Here's a look at how some managers handle the video onslaught.

Turner Entertainment Networks, Atlanta: Ron Tarasoff, vice president of broadcast technology and engineering, told eWEEK that his company has 25 video or audio feeds it creates and sends to virtually every corner of the globe on a 24/7 basis via GPS. The most well-known Turner network feeds are TNT, TBS, Turner Movie Classics, NBATV and Cartoon Network.

Next Page: "The days of videotape and film are long gone."

The days of videotape

and film are long gone">

"All the content we have is on servers, and everything is redundant," Tarasoff said. "The days of videotape and film are long gone. We have a one-button backup system. If the on-air server fails, our operator can push one button to enable the backup."

Turner uses a five-tier video storage system consisting of "edge" servers (mostly Pinnacle and OmniOn play-to-air servers); seven days' worth of programming backup (and that is also backed up); SATA (Serial ATA) drives with up to 30 days' worth of programming; an Asaca DVD jukebox server with 15 disk players that houses 1,200 DVDs; and StorageTek digital tape archives, which house 200,000 titles, including movies, television shows, commercials and promos.

"Altogether, we have about 26TB of video content stored right here in one place," Tarasoff said.

Tarasoff said the eight Quantum SDLT-600A tape drives that Turner uses "simplify our infrastructure so much because each drive has a GUI-visible file directory that goes with it. It includes all the associated data with each piece of video, so accessing and using each file is easy. As we continue to grow in the number of files we have, this simplification becomes even more important."

Lucasfilm: Lucasfilms ILM division invested in a Spinnaker video storage system in the late 1990s. Spinnaker was later acquired by NetApp, which, like Sun, is reaping the rewards of ingesting a highly respected, network-standard video storage and retrieval system with a number of prominent built-in customers.

"We have a 200TB NetApp storage system attached to our Spinnaker servers, with a 40GB Ethernet interface," said ILM's Thompson. "As you can imagine, we move humongous amounts of data from one place to another. NetApp provides the software stack for us; the load balancing and virtualization of all that data is handled very well."

Here is how the digitization process works: Artists in the studio, such as animators, modelers and renderers, create content at their workstations, and it is all saved overnight to the scheduler in the storage server (one of 20 Spinnakers in the ILM data center) that is handling their project.

All the files are marked with metadata tags for the editors. The editors then search through the file system directory to find different takes of the shots  they want for a particular scene.

Thompson, the storage administrator, sees the entire pooled-asset storage system as a huge "virtual disk" on his NetApp/Spinnaker console.

"When a rendering project starts to take more bandwidth than we originally allocated to it, the load can be distributed over multiple servers, as necessary, to get the job done in a timely fashion," said Thompson.

As shots are put together and updated, their status also is updated in the metadata database, so that any editor can check on the progress of each scene.

Once the "render farm" completes its work, editors put together a collection of shots to create a scene, which is sequenced into a full movie.

WGBH, Boston: Dave MacCarn, who has no particular title but calls himself "chief technologist" at the PBS station, said that he's getting about 50 years' worth of film and videotape ingested into digital archives and that he may never finish the job.

"It's a mountain of content," MacCarn said. "We have more than 300,000 hours of physical video to save. Film and tape just won't last."

IT managers should keep an eye on new display tech. Click here to read more.

WGBH uses StorageTek servers for archiving and a number of other servers for play-to-air, backup and scheduling. The station also has been a leader in trying to get a standard television video storage API and reference implementation established, MacCarn said.

HBO, New York: Ken Chin, vice president of broadcast engineering for the world's largest cable television franchise, told eWeek that his operation has to juggle 10 round-the-clock networks and that he's got about 200 days' worth of content stored in 50TB of StorageTek equipment.

"Ninety-six hours of content fills up about 1TB of storage," Chin said. "It's all stored on various tiers for immediate, occasional and archival uses."

HBO uses play-to-air servers from Thomson/Grass Valley as the front end of its system, and Chin said he's been impressed with their performance.

Next Page: Coming to digital.

Coming to Digital

"These things just go day after day after day, and they take a lot of beating. But we've been very happy with the result," he said.

Coming to Digital

There is no question that all video on film or videotape for broadcast use eventually will be ingested into digital form, several sources told eWeek.

But the process is slow, and many of the old analog film and video systems now in place still work well enough despite their aging components. Broadcast executives are cognizant of making sure their networks and stations get their money's worth before upgrading systems, which are both costly and time-consuming to install.

"Surprisingly, more than half the news operations in television stations in the U.S., at least, are still using videotape on a day-to-day basis," said Sun StorageTek media and entertainment solutions manager Tom Inglefield in Louisville, Colo.

"They are slow to make the changeover to digital. Budgets have their effect, of course, but the writing's on the wall. Videotape and film will disintegrate over time, and content will be lost."

In years gone by, much historic film footage from the early days of Hollywood and from newsreels was lost due to degradation of old film, poor storage and neglect.

Many classic movies have been restored by such archives as the UCLA Film Archive, the Smithsonian Institution, New York University, Carnegie-Mellon University and others, but there is much more to be done, Inglefield said.

"With these storage servers, nothing will ever be lost-or need to be restored-again," Inglefield said.

Storage by the numbers

50 to 60 percent: Rate at which data is accumulating for storage in the average business per year, according to Gartner Group

$20 billion: Estimated size of worldwide external disk storage market in 2006, according to IDC

$65 billion: Estimated size of worldwide external disk storage market in 2010, according to IDC

96 hours: Approximate amount of digital video that fills 1TB of storage, according to eWEEK reporting

25MB: Approximate amount of digital space that 1 second of high-definition video takes, according to eWEEK reporting

Check out eWEEK.coms for the latest news, reviews and analysis on enterprise and small business storage hardware and software.

Rocket Fuel