Digital crisis: Motion pictures may fade to black

February 8, 2008 (Computerworld) Current storage technologies may have a reputation for being plentiful and cheap, but not necessarily in Hollywood, where a recent study warns that the annual cost of archiving a digital film is 11 times that of storing celluloid film.

According to "The Digital Dilemma," a report recently released by the Academy of Motion Picture Arts and Sciences, digital film storage costs $12,510 per year, compared with $1,059 for celluloid. More dramatically, source materials -- those outtakes and audio recordings that often make up bonus content for special edition products -- cost 429 times as much to store, a whopping $208,500 per year for digital materials vs. $486 for film.

The report's authors state the data explosion could turn into digital movie extinction, unless the studios push the development of storage standards and data management practices that will guarantee long-term access of their content.

As the report points out, even if a 100-year black box were invented that "read data reliably without introducing any errors, required no maintenance and offered sufficient bit density at an affordable price," there would be nobody alive capable of repairing it if that box were to fail at 99 years. In the real world of data management, digital assets are stored on media with longevities much less than 100 years, vulnerable to temperature changes, humidity and static electricity. It can be misidentified, inadequately indexed and difficult to track.

Also, whereas a well-preserved 35mm negative has traditionally contained enough information to fulfill any requirement for ancillary markets, there's a question in the minds of some industry observers about whether the quality of masters archived in digital formats will be sufficient for quality duplication. In an age when home movie systems can often provide a better experience than some commercial theaters, that's not an unimportant concern.

"This is a clarion call to our people," says Milt Shefter, lead on the academy's Digital Motion Picture Archival Project and president of Miljoy Enterprises Inc., a media asset protection and preservation consultancy. "While there are great benefits to this technology, if you embrace it today, you are giving up guaranteed long-term access which you have with analog film."

According to Shefter, the price tag is of secondary concern to the studios. "The real cost is in not being able to guarantee that you'll have the material."

In digital we trust ...

When Paramount Pictures, under president Sherry Lansing, decided to create a 50th anniversary edition of Sunset Boulevard, one of the studio's numerous movie classics, it faced a dilemma. As described on the Paramount Film Preservation site, the original movie was shot on cellulose nitrate film. The studio had been making prints off of reprinted movies for years.

The problem was, movie duplicates had deteriorated. The prints had become grainy, and contrast had built up to the point where the darker scenes were almost black and indiscernible. The copy that had survived in the best condition actually included "small but very visible digs" in every frame. On top of that, through 50 years of studio history, every original piece of the film footage by director Billy Wilder had gone missing, including the original negative of Sunset Boulevard.

So rather than releasing yet another degraded version of the film with added material that would allow the studio to call the resulting DVD a "special collector's edition," Paramount called on its archivists to remove the scratches and dirt and restore the movie to as close to its former glory as possible.

In 2002, the preservation team hired the services of Lowry Digital (now named DTS), which applied a proprietary process to the movie, in which all visible damage was removed from a digitally scanned version of the negative. That high-resolution digital file was scanned back to film to create a new negative and output to a master tape from which a DVD was created. The collector's edition of the classic was finally released six months later in that same year.

That potential for losing irreplaceable content is supposed to be reduced by digital moviemaking, right? Directors shoot their movies on digital cameras and perform post-production on computers; the studios distribute the films to theaters via hard drives, tape drives or satellite; and then cinemas show the films using digital projectors. The ones and zeros behind the process never degrade.

The -- gulp! -- cost of storing a movie

But those digital bits add up -- to 2PB of data for a single movie, according to the academy. When 35mm film is used, studios can easily save everything: the original negative, the original audio, the still photos taken on set, notated scripts and other material. One case study cited by the report states that a two-hour feature film would take up about 129 cartons or cans, which are normally stored in vaults often located in underground salt mines. "Nobody paid any attention to what the budget was because it wasn't significant," says Shefter. The report estimates an annual cost of $1,170 to store the original production footage of a particular movie.

Compare that to the expense associated with storing that production footage in a digital format on tape in a fully managed storage facility. The cost is about $1.6 million.

While a director using 35mm film might shoot 15 or 18 minutes of film for every minute used in the final movie, "that ratio goes up tremendously when you go to digital," says Shefter. "It encourages more use." For instance, because film doesn't need to be loaded into the camera, the cameras just keep shooting -- even as the director steps out from behind the camera to talk with the cast.

Adding to the amount of data created in the making of a typical movie are the files generated during the post-production process, when the footage is turned into a sellable product. Directors believe they have better control when the movie goes to digital. "You can do so much more in the post-production process in digital than [you] were ever able to do in film," says Shefter.

The bottom line is that movie studios are in a position of having to maintain hundreds of terabytes of data for the material associated with any single motion picture, content that's barely or rarely cataloged or indexed.

Choices in storage

According to Brian Babineau, an analyst at Enterprise Strategy Group Inc. (ESG), the two basic choices for digital movie archive media -- tape or disk -- differ in advantages and disadvantages. Tape, the less expensive media, is highly portable, but the backup is usually compressed in a proprietary format. Also, searching for data on tape involves looking for files within files -- akin to looking through Zip files. "Then if you want to read data back, you have to restore tape through backup software into its native format," says Babineau.

The pricier disk-based archives store the content in their native format. It's more flexible than tape, says Babineau. "With disk, you can search the index, and then through any software package typically recall it. If you need fast access, it's much more feasible to do with disk."

According to analysis by ESG, maximum tape and disk capacities are about the same in commercial and government sectors. This year, disk storage capacities are expected to pull ahead of tape. By 2012, the amount of data stored on external disk will surpass that of magnetic tape archive storage. According to ESG, disk storage will account for 34.3 exabytes of data in 2012 compared to 31.5 exabytes on tape storage media. Optical storage capacity, which is also tracked by ESG, makes a nominal showing, about 2.4 petabytes by 2012.

The 100-year goal

The significance of studio movie collections, which first surfaced with the showing of old movies during the introduction of TV in the 1950s, has only grown in value with the advent of videos, cable, DVDs and video-on-demand delivery services. According to data published by The Hollywood Reporter, only about 19% of total revenue for the six largest movie companies came from theater showings. The remaining 81% represents revenue generated through DVDs, TV, pay TV and VHS.

As a studio's revenue potential grows from post-theater releases of movies, the drive to digital platforms has sped up because of the need to be able to access content quickly. "We're seeing a lot of people pick and choose what types of content they keep on more accessible media, in order to generate incremental revenue," says Babineau.

Yet Hollywood's history of archiving has never been standardized. Many early titles, produced on flammable film, have simply been burned in warehouse fires or turned to vinegar in uncontrolled storage environments. Fewer than half of all feature films made before 1950 have survived.

Now acetate-based films and their related materials are more likely to be archived in climate-controlled facilities with fire suppression systems. The goal: a film master that lasts 100 years.

According to Shefter, digital tapes and disks that have replaced acid-free cartons and steel metal cans used for film "have not proved to be a significant successful method of preserving this information." Some users reported to the Academy of Motion Picture Arts and Sciences that the materials on the drives couldn't be accessed after only 18 months.

The question of the long-term reliability of disk storage was the topic of a study performed by a group of university and vendor researchers in 2006. Their conclusion: While manufacturers of CD-Rs claim media "lifetimes up to 100 years ... actual lifetimes may be only two to five years." Media degradation happens not only as a result of "bit rot," a gradual accumulation of irrecoverable bit errors, making the data inaccessible, but also because media become obsolete. The scientists pointed to the once-ubiquitous floppy as proof.

The need for new processes

The major challenge that Hollywood faces in moving to digital has less to do with the actual limitations of the technology than with the reconfiguration of its processes, according to Steve Canepa, vice president of global media entertainment at IBM. "I talk about the factory notion," he says. "Today, you have literally dozens of ways that a piece of content can be viewed, depending upon the licensing rules, from theater to mobile to broadband to any number of devices in different formats. The methods and processes by which they go from higher resolution to lower resolution have to become increasingly more efficient."

That points to a major topic of the academy's report: the lack of standards. Studios are increasingly squeezing the window in which to monetize their movies, which means they frequently rely on archival technologies that are relatively immature. As media come and go, studios are forced to migrate to ever-newer platforms to be able to access their movie content. But that can leave content on older formats inaccessible. For example, LTO4, the current standard for tape drives in the movie business, which became available in 2007, is unable to read the contents of tapes written in the LTO1 format, the standard in 2000.

"We're seeing a lot of zero-day projects," says Babineau. "Whatever [companies] have right now in one format, they just keep it. But going forward they use a net-new type." The problem with that, he points out, is that the older content becomes unfindable unless equipment is kept around and maintained to read the older formats or there's a blending of indexes among formats.

"If you're dealing with a technology where you have to make a decision about what to do with it somewhere within a four- or five-year period, you have to know you're going to migrate it [or] get rid of it," explains Shefter. The studios' solution: to generate the majority of the revenue in that period before they have to make migration decisions.

Making way for what's to come

Migration projects shouldn't be a big deal to the movie industry, says Babineau. Having multiple copies of digital assets is simply a form of insurance against the loss of any particular master. "If you have something important -- The Godfather series, for example -- why not have a copy on film, but also digitize it and save a gold copy on disk somewhere? You can always go from disk back to film. We do it every day."

Canepa says IBM has customers in the film and broadcast industry that it helps migrate film to digital formats. Most of the time data is quickly migrated onto magnetic tape, which he says, is much denser and costs less. With compression, up to 1.6TB of data can be stored on a single tape cartridge today.

In fact, a similar migration project was announced in 2006 by IBM for Fox Broadcasting Co. -- not specifically for movies, but for its high-definition broadcasts such as NASCAR races, football games and shows such as American Idol. The broadcasts are beamed through satellites to an IBM xSeries server running Linux and then archived on magnetic tape.

Contrast that with ESPN, which, according to the academy's report, also runs a huge server farm. In the past, after a week's collection of broadcasts came in from professional and college sports, somebody -- usually an intern -- would go in and erase much of the data to make room for the next week's broadcast content. The process is described in the report as "triage on the fly." "That's a microcosm of what is going to happen in the industry," says Shefter.

The digital archivist

The job of the studio archivist -- the individual who, among other duties, decides what analog records need to be kept and maintained -- will become the responsibility of someone who is "more IT trained," predicts Shefter. "They will be in charge of distribution. They will decide what records they need to keep." One aspect of the archivist's work that will be lost in that transition, he says, is the ability to handle restoration. "It's sad, but I think that's the way it's heading."

Not everybody predicts the demise of future masterpieces with the advent of digital processes. In fact, the movement may portend even greater numbers of potential classics. Tom Streich, head of GripToyz, a movie production support company in Utah, recalls when digital began to take hold 10 years ago. "It was like, 'Oh, everything is going to change. We won't need lights. We'll save so much money on not having to process film.'"

But, he says, the movie digitalization is not saving anybody money in production. What digital has done, Streich believes, is opened the industry to myriad new, low-budget film makers. "They can do it on their PCs and put out usable products for television. And once digital projection comes out big, then it's not bad for theaters either."