Creating A Feature Film DCP Using OpenDCP

This article has been updated with new information gained from doing several DCP’s over the last year and with additional input from OpenDCP creator Terrence Meiczinger.

 
Hover over links like this for technical explanations

In December 2011 I wrote about using a MacBook Pro as a playout source for the cast and crew screening of the feature film McLean’s Money.  At that time it was the most practical and affordable way for us to get decent picture and sound quality in a modern cinema.

Since then a lot has changed and that is mainly because of a wonderful open source project called OpenDCP by Boston based developer Terrence Meiczinger.

What Is DCP?

A Digital Cinema Package is a specially designed collection file types defined by the Digital Cinema Initiatives organisation which was created by the major Hollywood Studios to standardise the distribution and playout formats for digital cinemas around the world.

Like a lot of people, I guess I initially expected these standards to be a variety of familiar formats with some special restrictions on the variants allowed.  The first time I looked at the specifications it, I saw hardly anything that I recognised – had they gone mad and decided not only to reinvent the wheel but make it a hexagon as well?

I have come to appreciate the soundness of DCI’s logic in defining the standards but I’ll admit that this hasn’t happened quickly!  If you want to see the DCI specifications themselves you can download the PDF from DCI.

It contains great lines like:

The DCDM gamma-encoded X’, Y’ and Z’ color channels are represented by
12-bit unsigned integer code values. These 12 bits are placed into the most
significant bits of 16-bit words, with the remaining 4 bits filled with zeroes.

Yes, a little light reading for a rainy day :)

A Simple Overview Of The DCP

  • MXF File(s) containing the pictures in JPEG 2000 Codec & XYZ Colorspace.
  • MXF File(s) containing the sound channels.
  • A number of XML files identifying the elements of the film and how they should be played.

An example of a DCP’s file list.

Some of the things that confused me initially about DCP are – having separate picture & sound files, the choice of the MXF format and the choice to have separate XML files holding the whole thing together.

It does actually all make sense though.  Having separate picture and sound files is very practical for films, for example it means that different language versions can be produced without having to re-encode all of the pictures.  In fact, the DCP can also have a separate subtitle file as well.

 

MXF Wrapper Format

As a long time Final Cut Pro user (since Version 1) I was very unfamiliar with the MXF format apart from know it as the format that Sony uses in it’s XDCAM cameras that has to be “re-wrapped” as Quicktime before use in FCP.  This concept of wrapper formats is an important one in understanding the value of MXF.

You can think of it in terms of a book.  You could have a paperback copy, a hardcover or even an e-book.  All three can have the same content,the same words, but the wrapper is different and makes those words more or less useful in different contexts.

In media these wrapper formats work in very much the same way.  DV video can exist as Quicktime, AVI or DV Stream files and the content the “words” of the DV encoded video can be exactly the same.  Or take a piece of h.264 encoded video.  It could be wrapped as an MP4, DivX or AVHD.  Same content, but in the DivX form it might be hard to watch on an iPhone for example, because the video player doesn’t recognise that wrapper.

So why use a wrapper at all?  Well to put it in very simple terms, if you think about a feature film having around 150,000 individual frames in it, then transporting and playing 150,000 individual files can be a real pain (but more on that later).  So putting them in an appropriate wrapper format is really useful.

One of the big advantages of the MXF (Material Exchange Format) is that it isn’t tied to any one manufacturer the way that something like Quicktime is tied Apple (as an easy example).  Why is this important for DCP?  The DCP specifications are the underpinning for the huge worldwide investment in digital delivery and projection technology.

While there’s lot’s of formats capable of doing the job, MXF is one that’s been developed by the industry and won’t suddenly leave the industry with licensing problems in a few years time.

It’s also pretty agnostic about about platform and codecs, so it’s very flexible and it was designed from the beginning to be very good at handling the additional information that goes with a lot of content these days – Metadata.

 

JPEG 2000

Within the Picture MXF the images are encoded with intra-frame compression in the JPEG 2000 codec.  This is very different to the everyday  JPEG format that everyone uses with digital cameras an a vast proportion of the images in the web.

Visible DCT Blocking.

Garden variety JPEG uses a type of  DCT  compression which breaks the image down into blocks of pixels near each other that can be grouped together and colour values averaged.  This works on the principle that pixels near each other are often similar colours.

In simple terms, the bigger the blocks, the bigger the reduction in the amount of data used.  JPEG & DCT are great at getting good looking images down to file sizes that are a fraction of the original.  The mathematics behind the process, therefore the processing power and time needed to do the job are relatively modest.

The downside is that when the compression does become visible it is in the form of those blocks.  This is most noticeable in big areas of similar colours light the sky or a white wall, where one block averages one way and the adjacent ones go a different way.  Most people would now be familiar with how this looks even if they have no idea why.  The problem with this is that when it does become visible it is quite obviously an artificial effect.

JPEG 2000 is a whole different animal.  It uses Wavelet compression which is much more complicated both as a concept and to implement.  The upsides are that the quality is much better and on the rare occasions when it does degrade the image, the effects look much more natural and are therefore much less noticeable to the audience.

DCT after 6 generations

DCT Compression after 6 generations

Wavelet compression works on a similar principle to DCT in that it assumes that parts of the image that are close to each other are likely to be similar and can therefore be grouped to reduce the amount of information with a minimal effect on the image quality.  The clever thing about Wavelet Compression is that it does this by breaking down the image by the amount of detail or wavelet frequencies.

Wavelet Compression after 6 generations

Wavelet Compression after 6 generations

So when Wavelet Compression does introduce artefacts into an image it does it by softening, and not even softening the whole image, just a particular band of detail frequencies.  So a highly compressed image using a Wavelet codec doesn’t show rectangular blocks and can still resolve fine detail.  On a 60 foot wide cinema screen, there is a whole lot of value in those two factors.

 

XYZ Colorspace

I’ve talked a bit about RGB Colorspace in my Secret World Of Colour Correction article and how it differs from the CMYK colorspace of the print world and how a weird mish-mash of the two is what we’re taught from early childhood is how colours work.  XYZ colorspace essentially takes RGB space and then refines it dramatically to match the way the various Rod & Cone “pixels” in our eyes actually combine to perceive colour.  So while RGB is a very electronic friendly way of capturing, controlling and manipulating colours, XYZ space is a very efficient way of presenting them so that they best match the human visual system.

XYZ Colorspace as it appears on an RGB screen.

 

XML

A DCP Asset Map XML

The eXtensible Markup Language has much in common with MXF in that it is an open, industry wide standard that is also highly flexible in being able to be adapted to different applications.

As markup language XML serves as a framework for adding more information into a document than the core text by marking it differently.  One of it’s most high profile implementations in the production world has been as the backbone of Final Cut Pro (pre FCP-X).

Like it’s brother  HTML is a simple way of recording and communicating information in forms that can be accurately interpreted by both the human eye.

So XML is a great choice for the files that identify the picture, sound & subtitle files and tell the projection system how to interpret them.

 

 

Making A DCP

So we know that a DCP is the gateway to the whole world of digital cinema, we know what the elements of one are and why they were chosen to work that way.  The elements are all readily and freely available, you can even export JPEG 200 straight out of Final Cut Pro using Quicktime conversion.  XML can be created in any text program.  So it should be pretty simple to manually create a DCP, right?

That’s what I thought.  Each of the many elements is in themselves pretty simple.

But getting each of them right, working together and then formatted correctly is to looking for a needle in a haystack what  designing a mission to Mars is to a game of chess.

I’ve been told that producers allow upwards of $20,000 for the creation of a single DCP.  And the complexity of making it all work does make sense of that price tag.

 

Enter OpenDCP

While new entrants have reduced the price of entry into DCP creation systems over the last few years from five or six digits down to the lower four digits, Terrance Meiczinger has been developing the OpenDCP software.  I’d been watching it for a while as it was a command line system, ie. you type in a series of text commands and then it goes off and does what you told it to (and hopefully you told it right!) the relatively recent addition of  GUI versions for Mac, Linux & Windows made me really sit up and take notice.  Now I’m not afraid of punching in a bit of code (after all I’ve written a successful package of colour grading plugins)  But the addition of the GUI versions took this a really practical level.

Open Source

There’s some genuinely amazing stuff coming out of the Open Source movement at the moment.  From the WordPress platform that powers this site to the Android mobile operating system and the Linux operating systems that power the actual digital cinema servers – Open Source is delivering some impressive software, often free and/or with completely new economic models.  OpenDCP could easily be the poster-child for this movement right now.  It has taken something that was essential for film makers in the 21st century but was prohibitively complex and expensive for most independent productions and made it relatively simple, cross-platform, high quality and there’s only one digit in the price tag… a zero.

 

The Process

The process of creating a DCP with OpenDCP is a small series of steps:

  1. Create an Image Sequence from your digital master. This can be in the TIFF, BMP or DPX formats.
  2. Create separate WAV files of each channel of the soundtrack.
  3. Convert the image sequence to J2C (JPEG 2000) files in XYZ Colorspace in OpenDCP.
  4. Wrap the J2C sequence into an MXF file in OpenDCP.
  5. Wrap the WAV files into an MXF file in OpenDCP.
  6. Create the DCP collection of files in OpenDCP.
  7. Copy the files to a Linux EXT formatted portable hard drive.
  8. Hand the drive to the projectionist.
  9. Buy some popcorn. (optional)

The important thing to remember is that the scale of each of these steps is proportional to the length and resolution of your project.  By far, the most processor intensive and time consuming part or the whole thing is encoding to the wavelet based JPEG 2000 codec.  On McLean’s Money we tried this with an iMac and the two hour film was going to take nearly 4 days for this one part of the process!!  On a 12 Core Mac Pro this came down to an easily manageable 9 hrs.  Out of interest, we did try this on The Film Bakery’s one (admittedly low spec) PC and it was going to take just over 18 days.  So this is an area where raw processing grunt makes a very big difference.

 

Our Current DCP Workflow

We’ve now converted 4 commercials, a feature documentary, a feature film and it’s trailer to DCP’s using this workflow.  The DCP for  McLean’s Money has now been commercially released in mainstream cinemas here in Australia.

Here’s that workflow step-by-step.

1. Create the Image Sequence

This is another of those things where at first I thought, why can’t it just convert from a Quicktime file?  Towards the end of creating the DCP for the preview screening of McLean’s Money we discovered there was a problem with the rendering of the colour grade on one shot near the middle of the film.  With less than a day before the screening this could have been a disaster but because OpenDCP forced us to work with Image Sequences we simply re-rendered that one shot, re-exported only those frames as TIFF’s and then converted those to XYZ J2C’s and dropped them into the 159,000 frame J2C sequence, replacing the original version on those frames.  Total downtime?  About 10 minutes.  I’m now a huge fan of using image sequences in this context!

There are a number of ways of creating an appropriate Image Sequence but for us Compressor proved to be the best solution.

The main reason for this is that it enabled us to export directly from the FCP timeline with frame numbers that could easily be related back to the FCP timeline.  This meant that I could stop the export of the TIFF files, take note of the final frame number and then go to that frame in FCP.  Right-clicking or Control-clicking on the timeline’s timecode readout brings up a number of options – one of which is “Frames”.

With this option selected and exporting using Compressor, the frame numbers of my TIFF and J2C files matched perfectly to the frame numbers in the timeline’s TC window.

Because we were exporting from FCP on one machine and encoding in OpenDCP on another in order to meet our deadline, being able to export 10 or 20 thousand frames and then move those across to the encoding machine meant that we were able to keep both machines churning along efficiently and problems like the incorrectly rendered shot could be dealt with as they occurred.

The first batch might be from frame 000001 to 021578.  To start the next batch I only needed to type 21579 into the FCP timeline TC window, add an in-point on that frame and then send to Compressor to continue.  The same for re-doing the one shot.  I just set the in & out points at the start & end of the shot and then sent it to Compressor which out put those frames, each with the correct frame number.

In Compressor I simply created a new  “Image Sequence” Custom Setting and selected “TIFF” as the Image Type, set our frame rate and made sure “Add Leading Zeros To Frame Numbers” was checked because it is essential for OpenDCP to have the same number of digits in the every frame number.

Because the DCP specifications for resolution are a container type spec, we were able to successfully use the 1920×1080 HD resolution to create a legitimate 2k DCP which had virtually imperceptible pillar box black bars down the sides on projection.  For me, the advantages in time and quality of maintaining the source resolution far outweighed the small advantage of having the digital image reach the final inches of the big screen in the darkened cinema and having seen the results with and without audiences in the room I would certainly be inclined to stick with this approach in the future.

The other big thing to remember with the TIFF Sequence is that it is going to take up a lot of disk space.  For McLean’s Money it was nearly 500 GB.

2. Create Separate WAV files for each Channel of the soundtrack.

I did this straight out of Soundtrack Pro by selecting the “Multiple Mono Files” option in the Export Window.  It is essential that the WAV files are 24 bit and 48 kHz as this is the only format that can be used to create a DCP.  Soundtrack Pro then creates files labelled “L” & “R” for Stereo or labelled for each of the 6 channels in a 5.1 mix.  Although the terminology might be slightly different, this is possible in just about any DAW, certainly in Logic Pro & Pro Tools.

3. Convert the Image Sequence to J2C & XYZ Colorspace in OpenDCP.

OpenDCP’s JPEG2000 Window

The design of the OpenDCP GUI is that most difficult of things –  elegantly simple.  The options have been well thought from the point of view of a serious user, the defaults are a solid starting point.

The first section of the JPEG2000 pane is dedicated to the encoding preferences.  OpenJPEG is currently the only option for encoder, your project is either Cinema 2k or 4k  Profile, it’s either Stereoscopic or it’s not.

The big variable here is data rate or Bandwidth and it’s relationship to Frame Rate. Although you set the actual playback frame rate later, the Frame Rate option is important here because it affects how much OpenDCP compresses each frame.  For example, if you set the frame rate to 24 fps in the JPEG2000 window but actually intended to produce a HFR 48 fps DCP, then the data rate could easily exceed the capacity of the cinema servers and the DCI spec.

Conversely, if you set the frame rate in the compression settings to 48 fps and actually had 24 fps material, you would be getting half the data rate that you intended.

The default setting of 125 mb/s is great quality but probably heavy handed for most purposes.

The DCI maximum data rate of 250 mb/s is actually designed to allow for 4k 3D, so 2k in 2D can actually achieve the same quality at much lower data rates.

For McLean’s Money I did tests at 125 mb/s and then at 75 mb/s because I’d heard of some studio pictures using that data rate for 2k.

For the version that was released to the cinemas we dropped the data rate  to 55 mb/s and this allowed us to deliver the film on a 64 BG USB stick and the quality was fantastic.

Meiczinger has reported successfully using data rates as low as 20 mb/s for 2k 2D, so 55 feels like a comfortable margin to me.

A DCP USB of McLean's Money

A DCP USB of McLean’s Money

As well as storage space, the other reason for wanting to keep a handle on Bandwidth is the transfer times both to the delivery drive and for each cinema ingesting the film.

We have been using Paragon Software’s ExtFS for Mac.  One thing we’ve discovered is that when the drive is formatted as EXT-2 the transfer times are dramatically longer.  At 75 mb/s the 60 GB DCP took nearly 6 hrs to transfer to an EXT-2 drive!  By formatting as EXT-3 or EXT-4 that drops to about an hour.

We have  also tried setting up a dual boot machine to run Ubuntu Linux that works very well for formatting and copying the files.

The Image Parameters section has some very important options.  Source Color can be set to either sRGB or Rec. 709 depending on the calibration of your monitor and output.  There is a significant difference between the two so if you’re not sure then definitely do some tests to find out.

I used the XYZ Transform within OpenDCP and was so pleased with the accuracy of the conversion that I feel no need to explore any other options, although there are many other ways of doing this at the Image Sequence stage.

If you’re working from DPX Log format images (the successor to the Cineon digital film format) then that box should be checked and it is by default.  If you’re not working from DPX files in Log format then forgetting to un-check this will produce images that are almost unrecognisable and because of the XYZ colorspace it might be difficult to tell until you’re in the cinema.

DCI Resize is something I haven’t had any need to try yet.

Input and Output Directories are pretty self explanatory.

When you hit “Convert” you’re starting the biggest part of the DCP creation process so it’s worth double checking all of those options.

4. Wrap the J2C sequence into an MXF file.

In the MXF Paramaters section you can choose a Type of either JPEG2000, MPEG2 or WAV.  The MPEG2 option is really only relevant if you’re dealing with a cinema that has a very early DCP Projection system that requires that format.  For broad compatibility you want JPEG200 and then you’ll use the WAV option to do your sound MXF.

Similarly with the Label the new SMPTE specifications for DCP are the global standards now and unless you have a specific need for the old MXF Interop spec then it’s best to leave it alone.

The big thing here is frame rate, because this is where you set the actual playback frame rate for all those many thousands of frames. This has less to do with any picture frame rate you’ve dealt with before than with the frame rate that you’re audio is at.  For example, on one of the commercials we’ve done it was shot and edited at 25p for Australian television & digital cinema but we had a need to produce a 24p DCP for the international market.  So from the same J2C’s that we used for the 25p DCP we then created a 24p MXF and changed the playback of the sound to 24p in Cinema Tools before producing the separate mono WAV files.  When the two were married up the result was seamless.

Again in the Picture Parameters Stereoscopic, Slideshow and Slide Duration do exactly what they sound like they’ll do and if you don’t think you need them then you definitely don’t.

The Picture Input is the folder where you put the Output of the J2C conversions.  Output Files is where the MXF is going to end up and it’s not yet critical where that is.

It’s worth noting here that because for a feature, you’re dealing with folders that have many tens of thousands of files in them, just selecting and navigating the input and output folders can make it look like OpenDCP has crashed.  Even on the 12 Core MacPro I had this experience, but if you give it time to figure out how many files you’re looking at it does come good.  Setting the Mac’s Finder to display in List view rather than Column view helps enormously with this issue.

5.  Wrap the WAV files into an MXF.

Again, in this stage of the process, specifying the correct Frame Rate so that the sound and pictures will match is crucial.

You then need to select whether you’re doing 2.0 Stereo Sound or 5.1 Surround.  The new option for the new 7.1 Surround format is currently greyed out but tantalisingly sitting there nonetheless.

Then it’s a matter of identifying the WAV file for each of the 2 or 6 channels you have.

One thing I would add here is don’t be afraid of using Stereo if that’s what you have.  There has been some criticism of this on the net and I accept the theory but in practice I’ve now watched the Stereo DCP of McLean’s Money with numerous paying audiences and I’m confident that stereo is quite acceptable for the right kinds of films.

If you’re doing an action film then sure, you probably want surround.  But for small drama/comedy, stereo is ok.  In the analogue sound days of film, there was a huge difference in quality between stereo optical tracks and Dolby Surround and that issue is not there at all with DCP.  The technical quality of the stereo channels is the same as the quality of each surround channel.

One of the factors of the DCP format is that the digital stereo is the same quality as the surround.  Yes, you won’t be able to envelop the audience with sounds coming from all around the theatre but the actual quality will still be very good.

For the output, I’ve just been putting the sound MXF in the same temporary folder as the picture one, since they’re moved during the creation of the DCP anyway.

The next pane is for subtitles but since I haven’t had any need to experiment with that yet I can’t speak with any authority on it.

From there we move onto the DCP Pane and that I can talk about.

6.  Create the DCP

The first thing you see in the Composition Parameters section is the Title text box.  The SPMTE standards specify a relatively large amount of information to be conveyed in the title and luckily OpenDCP has a button to the right of the text box to access a Title Generator.

When you’ve entered all of the relevant information into the Title Generator it formats this information into a technical title line that follows the SMPTE specifications.  I’ve found this to work really well and helped the projectionists I’ve seen importing the DCP’s.

There’s an option to add an Annotation which can be the actual title of the film and there’s another box to identify the Issuer of the package, specify the Rating and the Kind of project it is.

The only limitation of the public beta release of the GUI version of OpenDCP is that it is only capable of handling one “Reel” so the entire film has to in one Picture & one Sound MXF.

The use of 20 minute reels in the photochemical film world is a physical necessity.  However, like many other people, I’ve found that splitting a digital feature into reels makes a lot of sense in terms of handling files and making changes.  To be able to complete the DCP on a features other five reels while still making tweaks to reel 2, for example, will be a great feature when it comes to the GUI.

I understand that it is fully functional in the Command Line version, so I’m assuming that it will turn up in a future GUI release.

Having said all that, once you’ve got you’re whole film in the Picture & Sound MXF files it’s a very simple  process to create the DCP.  The first time I hit the “Create DCP” button I thought something had gone wrong because it just asked me where to save the files and then said it was successful.  Nothing seemed to have happened.  But the process really was that quick.  It’s another good example of how well thought out both the DCP format and OpenDCP’s implementation of it are.  The bulk of the work & processing is early in the process when it’s easy to catch and repair.

This final stage is just a matter of creating the XML files which as a group are less than 20 kilobytes in size and then moving the MXF files into the same folder.  If they have been stored on the same drive this will be virtually instant.  If they’re on another drive it will just be the time to normally copy them across.

There is also the option to copy rather than move the MXF files.  Unless you’re doing many different versions I would recommend leaving this as the default option of moving the files.  Unlike the process of wrapping the JPEG’s and WAVE’s into the MXF files where you can’t see the originals in there, creating the DCP still leaves the MXF files visible and accessible in the DCP folder.  Another upshot of this is the if you have a new sound mix, for example, it is possible to just replace the sound MXF file.  It is probably safer to recreate the DCP but I did do this successfully before the preview of McLean’s Money when we had a last minute tweak od the sound and I didn’t want to go through the 6 hr process of copying the new DCP across to the delivery drive.  Instead I just copied the new Sound MXF with the same filename as the original to the delivery drive and it all worked beautifully.

7.  Copy the DCP to a Linux EXT drive.

On some systems you might have success with a Mac or PC formatted drive but I’ve been told by some projectionists that they can be problematic.  The DCP servers are mostly Linux based and seem to be most comfortable with their own drives and that’s what the ISDCF has defined as best practice.

On McLean’s Money we have had the feature and the trailer on the same USB hard drive in their own partitions and this worked very smoothly.

8. Hand the drive to the projectionist.

In the projection booth the drive was plugged into the server which then recognised it as two drives, each with their own DCP and it offered to import both.  Again, it’s interesting to note how the DCP format puts all of the emphasis on making the backend of the process as easy and seamless as possible and complicating the front end as much as necessary to achieve that.

For our tests and the preview screening we had some very good projectionists at the Hoyts screening room adjacent to the Fox Studio lot here in Sydney.  But around the world I’m sure that there are thousands of teenagers with a part-time job that now includes loading DCP files into digital cinema servers and the DCI group seems to have taken this into account in designing the process!

Our files were loaded into a Doremi projection system and the process was so straightforward that I have only bothered to watch it happening once.

9.  Buy some popcorn.

As I said, this is optional.  At our screenings I didn’t choose this option and I think with the benefit of hindsight that it might have been the better choice.  ;)

Seriously though, the end result was genuinely thrilling.  The picture and sound quality was a exact replication of what we had in our final grade & mix (every flaw & compromise perfectly reproduced!).  It’s just that it was all BIGGER.

The digital revolution over the last decade has been a wild ride and I count myself very lucky to have had a front seat through a lot of it.  In many ways the accessible digital delivery to cinemas has been the last key piece of the puzzle to fall into place.  The long wait for cinemas to be given a compelling reason to make the switch to digital projection finally came to an end with Avatar.  In fact 3D might have made it’s greatest contribution to the industry simply by providing this reason to change.

But then the complexity of the DCP format and the rigidness of the DCI compliant servers (both for reasons of reliability and security) meant that it was still very difficult for indies or even more so for one-off screenings like our previews to get to the now commonplace digital projector.

OpenDCP has completely changed that and provided the crucial, direct and practical link between the desktop post production system and the digital cinema.

It is now possible to take an entire feature length movie from what comes out of the camera to what gets sent to the cinema all on a desktop Mac.

I’m confident saying that because I’ve done it.

Will this lead to a new golden age of independent cinema like what happened when Arri introduced the first 35mm camera that could shoot sync-sound whilst handheld in the early 1970′s or when Kodak launched their T-Grain high speed filmstocks at the start of the 1990′s?

I hope so …the technology is certainly ready.

Comments are closed.