Case Study: Officials scrap tape replays and adopt a video synchronization system.NASCAR officials are monitoring a race at the Talladega Superspeedway in Talladega, Ala. The white flag has dropped, signaling the last lap of the race, but before the winner can cross the finish line, theres an accident on turn one.
A yellow flag has dropped. The entire field is frozen. According to NASCAR rules, because its the last lap, the position of every car at the time of the caution determines the winner and finishing order of all other cars in the race, explained David Hoots, managing event director for NASCAR, of Daytona Beach, Fla.
Up until last season, race officials relied on the television networks to provide them with tape-based video replays of many camera angles to determine the outcome of events, such as a finish under caution, said Steve Stum, director of field operations for NASCAR Images, the Charlotte, N.C., production company that supports NASCAR.
Acting as the technical interface between NASCAR and the networks, Stum witnessed the producers frustration with the officials constant interruptions.
Officials didnt enjoy being dependent on the networks, either. Nor did they like having to wait for video.
"NASCAR was delayed in being able to review something because that tape [operator] would be needed for air first. So typically the network would run the video on air first, and during a commercial break they would play it back for NASCAR," said Scott Rinehart, NASCAR Images director of operations and licensing.
According to Hoots, a finish under caution could take up to a half-hour to determine the position of every car in the field as officials constantly called up multiple tapes trying to match car positions with the caution lights.
No one was happy. So Stum and Rinehart started talking about a better solution—a massive DVR (digital video recorder) that would give officials the ability to control their own instant replays.
Their first thought was that this was going to be an overwhelming storage undertaking. A NASCAR race involves dozens of cameras running continuously for 4 hours. How many cameras could they realistically record onto disk at MPEG-1 quality?
In July 2004, Rinehart called VideoBank, in Northvale, N.J., which provided NASCAR Images with its DAM (digital asset management) system for race archives in 2003.
VideoBanks video storage solutions capture video content, convert it to a digital format, and then allow users to simply manage, store and distribute it by reducing the number of man-hours and the expertise needed, said Lou Siracusano, president of VideoBank.
At the time of the call, Rinehart discussed NASCARs need for a race replay system that could synchronize and record up to 20 cameras. When an event happened, officials could call up and watch a number of camera angle replays simultaneously.
Rinehart turned first to VideoBank and didnt consider any other video solutions provider.
"First of all, we liked the people at VideoBank. And then we liked the way they organized their data," Rinehart said, referring to VideoBanks DAM system, which can handle NASCARs complicated metadata—43 drivers with 43 different sponsors, car manufacturers and paint schemes, with driver and car number combinations that can change on a per-week basis.
The DAM system had race car color-coded icons to simplify cataloging of information. According to Siracusano, the theory behind this method is that simplified data entry translates to a better archive. "An archive is only as good as the data thats put into it," he said.
On that initial call to VideoBank, Rinehart asked, "How much storage would it take to record 18 feeds for 4 hours at MPEG-1?" Rinehart said.
VideoBank thought he was crazy, to which reaction Rinehart responded, "Probably, but lets start."
Click here to read more about NASCARs IT usage.
By August 2004, NASCAR had greenlighted a test for that seasons race at the Talladega Superspeedway. That gave NASCAR Images and VideoBank one month to create a six-camera prototype test solution.
The prototype was successful, and the response was strong, but it wasnt until December that they received the go-ahead to roll out all 18 channels for the first race of the 2005 season, the Budweiser Shootout in February, Rinehart said.
Siracusano considered all storage- and transmission-related issues when building NASCARs system.
First, NASCAR needed plenty of bidirectional bandwidth for recording and replaying video given that each camera was streaming 2M bps of MPEG-1 video. For simultaneous encoding and playback, the race replay system would also need fast processing and fast hard drives. On top of it all, VideoBank required ample disk space for the 8GB of disk space each camera consumed per race.
In the end, one race would take up approximately 150GB, Siracusano said. After the race, the data is moved to a portable FireWire drive, where its coded with basic tagging information, such as camera angles and timing of yellow flags. That drive is then shipped to Charlotte to be imported into NASCAR Images DAM system.
With the creation of the prototype and further development, Rineharts team—along with the officials—spelled out their demands for NASCARs Race Replay System.
They began to build a massive DVR that recorded 18 synchronized cameras on three RAID Level 5 servers. Each server handled six channels and could accommodate recording and playback, thanks to 15,000-rpm SCSI drives and load balancing over the three servers.
Each server also was consistently fed 12M bps of video over a 1000BaseT (or Gigabit Ethernet) network. A duplicate Gigabit Ethernet pipe also went back up to race control, allowing for video playback, explained Siracusano.
The most important aspect of the Race Replay System was the ability to simultaneous view, frame by frame using MPEG-1 video, four perfectly synchronized camera angles.
"With a sport that moves at nearly 200 mph, a half a second represents quite a distance," Rinehart said.
An additional challenge for VideoBank was the officials need for dynamic selection of camera angles. Officials needed to be able to call up any four angles for any incident, to see any camera angle at full screen and to be able to review any historical moment in the race, Rinehart said. The key to success was making sure that all the video was in sync.
"We worked closely with the manufacturer, which is Optibase, on writing that timing information into the header file of the video file," Siracusano said. "So its not an offset or data variable that stores the timing information; its embedded in the video file."
Ironically, the worst-case finish scenario did happen at the first race of the Busch Series in 2005—The Hersheys Kissables 300, at the Daytona International Speedway. On the last lap, an 11-car incident caused a caution and the reliance on the NASCAR Race Replay System to determine the finishing order of all the cars.
"[The NASCAR Race Replay System] was exactly what [race officials] wanted. It made their life easier. They had control of their own replays now so they didnt have to bother the truck to help them out," Stum said.
"Its removed any ambiguity that existed before the system was invented," said Dan Patin, director of broadcast operations for NASCAR. "It gave us one more method of proving a call right."
David Spark is a freelance writer in San Francisco. Contact him at david@davidspark.com.
Check out eWEEK.coms for the latest news, reviews and analysis on enterprise and small business storage hardware and software.
Data Storage, Data Backup and Storage Virtualization 
