Imagine a company spending millions of dollars to buy workstations, software, servers, networking and storage equipment for everyone working on one large project. Then imagine recycling those systems and buying everything new for the next project.
This may seem like IT overkill, but it’s a way of life at Glendale, Calif.-based DreamWorks Animation, creator of such highly successful movies as the Shrek and Madagascar series, Monsters and Aliens and the Kung Fu Panda series. It’s also what has helped keep the company on the cutting edge of technology animation.
At DreamWorks, a new movie automatically means fresh Hewlett-Packard z800 workstations, new software, additional storage and other equipment for about 500 highly trained artists and supervisors. An animated movie-whether or not it’s 3D-takes four to five years to produce, and the equipment stays with the production from start to finish, so the studio gets its money’s worth from its investments.
Due to the history and reputation for quality 12-year-old DreamWorks has earned, its movies are virtually guaranteed high box-office returns. The films are also a major driver in the entertainment data storage sector, one of the hottest sub-sectors within the burgeoning international data storage market. In fact, this sector has its own professional conference: Coughlin Associates’ 2011 Creative Storage Conference, set for June 28 in Culver City, Calif.
This market category takes in all professional creative media, including feature films, documentaries, corporate videos, television shows and music videos. The escalating use of high-quality video-such as high-definition and super-high definition, which take up more than double the space of regular video-is a major cause of this jump.
Opportunities for new profits in this sector are substantial, to say the least. A report published last month by Coughlin projects the media storage market to virtually double in the next five years-from $3.8 billion to $6.4 billion in revenue, and from 11 exabytes to 62 exabytes in capacity. That’s right, exabytes: a million trillion bytes.
“Digital storage requirements are exploding due to use of higher resolution and stereoscopic content in the media and entertainment industry,” lead researcher Tom Coughlin wrote in the report.
Central characters from DreamWorks Animation’s “Kung Fu Panda 2” are (L-R) Tigress, Po (the Kung Fu Panda), and Monkey.
Ramping Up 3D Films
DreamWorks’ 2011 movie is the 3D-enhanced Kung Fu Panda 2, which opened May 26 and amassed $332 million in worldwide box-office receipts in the first 17 days. Since the movie cost about $150 million to make (not counting marketing and distribution costs), the producers are already in the black.
Because video quality has improved so greatly in the last decade, 3D feature films tend to be more successful at the box office than standard films and have been ramping up in sheer numbers.
“Once you’ve seen good 3D, you get spoiled, and it’s hard to go back,” DreamWorks CTO Ed Leonard (pictured) told eWEEK. “The expectations go up with each new release.
“I’ve been in this business for longer than I care to admit, and, in that time, I’ve observed that every film tries to outdo the last film. The roots are still in great characters and in great storytelling, of course. But we want to bring that to life in a way that you’re mesmerized. … You’re watching something that is taking you to a different place, and you’re forgetting about all the worries of life for two hours.”
DreamWorks outdoes itself on a regular basis, as do its equally well-regarded rivals, Pixar Animation Studios and LucasFilms Limited.
The company churns out about three films during a two-year span, so an ambitious schedule is always in the works. Each movie has its own animation staff and HP workstations, which are constantly being updated as new, faster Intel processors become available.
“Because every movie is new, we change the tools and technology [based on] what we’re trying to achieve,” Leonard said. “Literally, everything is constantly reinvented.
“For Kung Fu Panda 2, this is the first time we’re going to see this world in [true] 3D. All the things that we’ve learned in our past few 3D movies [such as 2010’s How to Train Your Dragon and other films] have led us to enhance the toolset and our creative skill set. You see this come to life in a really big way in Kung Fu Panda 2.“
DreamWorks obtains all new workstations about every six months, as new movies get started, Leonard said, with the hardware and software improving incrementally with each new purchase. The IT shop also spends a lot of time on performance-optimizing the software.
A typical four-year DreamWorks movie project coincides with current improvement cycles in IT, Senior Technologist Skottie Miller told eWEEK. “About every four years, there’s a quantum leap in computing power and I/O speed,” he said. “We find that it’s best to buy new hardware because it will pay for itself in about one year.”
“That’s what’s nice about working at a place like DreamWorks-you get to use all the new stuff,” Leonard said. “During the time of this production, we went from four-core Westmeres [Intel processors] to six-cores. Every workstation has 12 cores working, so there’s a tremendous amount of power [for each artist]. And each of those cores has gotten a lot faster.
“It’s kind of Moore’s Law on steroids. That’s what we’re chasing.”
Quality Is on the Screen
All this new equipment equates to one major thing: The power and efficiency of ever-improving hardware and software platforms enable animation artists to do more in less time, so that more iterations of scenes and characters can be created. This results in more choices of art for the directors and producers.
“On a broad scale, visual richness and interesting literal depth of what you see in the film is what we’re going for,” Leonard explained.
“At the end of the movie, there’s a climactic epic battle scene that takes place in the canals and harbors in the city, with lots of incredible water, fighting and other actions. These things are really hard to do in CG [computer-generated imagery], but you see them happening here without restraint. It’s pretty magnificent.”
Leonard said the most important change in the production software for Kung Fu Panda 2 was probably in the character toolset, “which gave us the opportunity to rerig our characters [based on the original Kung Fu Panda of 2008] and redefine some of the algorithms that create the motion and key performance enacting,” he said.
“Of course, all the tools we used on the original movie needed to be upgraded to be able to handle [true] 3D. So when our [scene] -lighters’ are working, they can now see their work in 3D. That’s new on this film.”
The sheer amount of detail in this film has gone way up from the 2008 movie. “The first Panda movie took about 50TB of data generated over the life of the film, and we used a little over 20 million render hours,” Leonard said. “With this new film, we stored over 100TB of data and used over 55 million render hours.”
That level of detail has a tremendous impact on the quality of the animation. “Look at the amount of richness, detail and expressiveness and all the things that go into making you suspend disbelief that Po [the panda] is emoting real character. That is a big part of what the technology does for the creation [of this movie].”
Cloud Rendering Is in the Mix
Ninety-minute-long 3D films made in 2011 require north of 100TB of storage capacity in the studio’s data center farm, Leonard said. Because of these new requirements for Kung Fu Panda 2, DreamWorks added a cloud services center to the mix for added storage and more agile operations.
“We did a lot of cloud rendering on this movie,” Leonard said. “We were very aggressive in moving that massive amount of compute that we need to finish these films. It used to be that we had to have all this stuff within our walls and under our control, but with this movie, we pushed a lot of it out to the cloud.
“I would say that more than 10 million render hours were rendered in the cloud, which is about 20 percent or more of the film-and that is a pretty big deal.”
During the next year, Leonard said, the company will be moving more of its rendering to its cloud storage system (run by both DreamWorks and HP). Eventually, most of the tedious rendering duties will be done outside its walls.
“This gives us the opportunity to lower our production costs, but, more importantly, it gives us the ability to -flex up’ when we need it,” he pointed out. “We can add or subtract some things quickly from a screening, or add a little more of something else, and we don’t have to worry about buying more servers or building out our data center. We just make the call and we have more compute.”
DreamWorks uses HP’s data center in Las Vegas and Cerelink’s in Corrales, N.M. “If we build a data center somewhere and render there, it’s not really the cloud,” Leonard said. “We prefer to work with our partners on that.”
Leonard said that despite all the additional work and technology that goes into a high-quality, two-camera, 3D rendition movie like Kung Fu Panda 2, the overall costs for these movies are actually going down at DreamWorks. “That’s because the process is so much faster and more efficient, [which enables] our artists to collaborate more often and get things done faster,” he said.
“If you can give artists 10 iterations instead of five, they can try 10 things instead of five things, and, in time, you’re going to get a better film. That’s what we’ve been doing with tools and technologies that allow us to deal with massive amounts of data and ever-increasing complexity, but do it in a way that makes the experience of the artists better.”
Of course, all this requires a huge storage capacity. DreamWorks, which uses mostly NetApp and HP for its data storage needs, may have actually lost count of exactly how much storage it owns. When asked how often he needs to purchase new storage, Miller told eWEEK: “Storage isn’t a buying decision anymore-it’s a way of life.”
“We have to store an awful lot of video,” he explained. “For example, there are 129,600 video frames in one 90-minute movie. Most scenes are 5 minutes or less in length. But there’s so much more [artwork] created that goes into the process before it’s finally cut down and melded together into a cohesive movie. Basically, we’re just doing file-based supercomputing every day at work.”
Churning Out Artwork
DreamWorks is the first movie studio to use multicore-powered workstations. Most of its CG stations have 12-core machines capable of 24-thread computing.
Where it once took 8 hours to render a single frame of a 3D movie, it now takes only several minutes. That’s huge in terms of time and cost savings.
The HP Z-series workstations offer about 50 percent more cache than previous versions (up to 24GB), which helps speed processing in a big way. Intel Xeon 5600 Westmere processors inside the workstations also run cooler, require less power and cooling energy, and have improved security features.
When CG artists put together all the attributes that constitute a character, several iterations must be developed: It starts with modeling and moves to rigging, hair and fur, clothing and surfacing. This involves very detailed-and often tedious-CG work. It takes a certain type of personality to excel at this, and DreamWorks has found hundreds of folks who fit this bill.
Artists can take 12 to 18 months to do the original drawings for a character. Since Kung Fu Panda’s characters were created seven to eight years ago, however, most of the basic work had already been done. Still, the original film could not be reused in this latest film because of the new 3D presentation.
Once the drawings are completed, CG modeling comes next. From here on out, DreamWorks uses Adobe Creative Studio for everything that goes on screen.
That’s where mathematics, science, HPC (high-performance computing), artistry and the professionalism of the artists all come together. Every square inch of a moving character is plotted out: how it moves, how it interacts with other characters, how it is affected by wind and water, and how hair moves and falls.
There are hundreds of factors and variables involved here, and they all have to be orchestrated perfectly to show the audience “reality.” In fact, DreamWorks has a separate CG unit that concentrates strictly on hair, fur and clothing because so much of it has to be created and rendered.
Surfacing is extremely important, especially in 3D presentations. Lighting, shadows, perspective and other visual factors come into play, and no detail is left untouched.
Much Has Changed
Miller, who’s been with DreamWorks for 12 years-after spending seven years at Disney’s animation studio-attested to the huge change he’s seen in HPC tools for rendering streaming video.
“In 1999, when I arrived here, we had a total of 140 cores [processors], and now we have more than 10,000-in fact, I’ve lost count,” Miller said. “We used to have to physically share the most powerful computers by putting them in boxes and shipping them from one studio to another to get a movie done. Sometimes we’d have fistfights over the RAM because memory used to cost $4,000 to $5,000 per megabyte back then.
“We just didn’t have the bandwidth, tools or computing power in 1999 that we have now. Today, we’ve got multicore processors, dynamic allocation of computing power and incredible amounts of storage. It’s been awesome to witness all these advancements.”