Avid is Self-Aware – Fast Import

Avid is Self-Aware – Fast Import

Avid is self-aware.

And while not in the Skynet sort of way, Avid is aware of it’s own.  Commonly this is found in a shared user environment (Unity, ISIS).  However, it can also be found in terms of raw media.

A little known feature within Avid is the concept of a Fast Import.  This complicated term does exactly what is says – it imports media into your Avid faster than a traditional import.  However, lately I’ve been frequently asked, “well, why not use AMA?  No importing is needed!” Let’s address this before we jump into Fast Importing.

AMA in Media Composer 5 (or as I called it: AMA v 2.0) and above introduced a new feature to the previous incarnation of AMA: The ability to use QuickTime files in your Avid project WITHOUT having to transcode during import.  While this is fantastic for instant gratification, it can cause problems during creative edit.  Newer, highly compressed codecs (See: RED, 5D / 7D, etc) commonly cannot play in real time inside Avid.  This can also lead to problems later – exporting and Digital Cuts are somewhat limited by AMA files.

Best practice is to convert all media into an Avid codec – for HD, it’s a flavor of DNxHD – during import.  Avid is then dealing with their own codec.  Aside from the excellent quality of their codecs, they also do not break down during post due to renders.  So, for several reasons, working in an Avid codec – natively – is an excellent choice.

Back to smelling one’s own.

The straight import process into Avid can be a long one.  Not only does the conversion of the file into an Avid codec take a while, it also effectively kills your use of the Avid.  The Avid is unusable as the progress bar creeps across the screen.  But, if we create a file that is ALREADY in the Avid DNxHD codec, Avid will recognize it as such and skip the transcode process.

Avid will still need to wrap the file into the proper MXF shell to be used effectively – however, this is usually 70% – 80% faster than a traditional straight import.

Geek Sidebar: Avid has always had excellent media management.  Maintaining this requires Avid to separate a QuickTime file’s audio and video into separate files, wrapping them appropriately, and placing them inside the ‘Avid MediaFiles’ folder.  This process takes time, hence the ‘only’ 80% savings in time.
/end Geek Sidebar

Wunderbar!  How do I get my grubby paws on this technique?

Very simple!

  1. Download the Avid codec onto the machine which will do the encode (or as I call it, pre-encode).  It’s free!  http://www.avid.com/dnxhd

    Codec Details Window (Click to Enlarge)

  2. On the same machine, open up your encoding software du jour.  I am a fan boy of Telestream’s Episode Family (Episode Engine, using multiple computers – Nice!)  Other solutions may be Apple’s Compressor, Sorenson Squeeze, etc.
  3. Choose to encode into a QuickTime Movie (.mov) Typically there is an ‘Options’ or ‘Advanced’ button, which will allow you to dig into the particular settings of the Encoder.  You want to choose “Avid DNxHD Codec” from the list.
  4. You can force FPS, but I typically leave it at ‘Best’.  If you want better quality you can select “Multi-Pass” (if your software allows the choice).  “Single-Pass” works fine for most applications, and is faster.
  5. As far as Compressor Depth: 9 times out of 10, this should be at “Millions of Colors”, NOT “Millions of Colors +”.  Why?  The + allows for an alpha channel, which the Avid DNxHD codec now supports.  Unless you desire an Avid DNxHD file which has an alpha channel, disregard this.  The Alpha channel will cause a longer encode and a larger file size and is typically not used on media that is camera generated.  This also goes hand-in-hand with the option below “Alpha”.  Select “None”.
  6. This one is so important, it gets it’s own number in my list.  Color Levels: 709 not RGB.  Why?
    Avid DNxHD Codec Chart

    Avid DNxHD Codec Chart (Click to Enlrage)

    Rec. 601 and Rec. 709 are color spaces for SD and HD video, respectively.  These values translate into Avid in an Y’CbCr color space, which Avid plays nicely with.

  7. Resolutions:  This is where things get hairy.  Avid, in an attempt to not confuse people, created DNxHD Codec ‘families” to aid in ease of encoding. Unfortunately, it causes confusion when you get past the topical discussion about it.

    Geek Sidecar:

    Avid has 3 DNxHD Codec Families: DNxHD 36, DNxHD 145, and DNxHD 220.  These numbers correspond to the data rates of the family (Mb/sec).  However, what may not be obvious is that these data rates vary according to frame rate (23.976, 29.97, etc), however the family remains the same.  Thus, 23.976 fps in DNxHD 36 at is indeed 36Mb/s, but 29.97 fps at DNxHD 36 is 45Mb/s.
    Huh?  What?

    Yeah.

    Avid has gotten better at this, and as you look at the chart below, you’ll see the new numbers reflect the correct data rate.  Read for yourself: http://www.avid.com/static/resources/es/documents/dnxhd.pdf.
    Here are some (of my) good rules of thumb for each family:

    • DNxHD36: Offline quality, not fit for broadcast.
    • DNxHD145: Threshold of broadcast quality.
    • DNxHD220 or 220x: best quality possible

      Avid’s Format Tab (Click to Enlarge)

    /end Geek Sidecar

  8. Start the encode!
  9. Next, we must start an Avid project (or change the project) to be in the correct format to accurately use the imported files.  Click your “Format” Tab  and ensure your “Color Space” is set to YCbCr709.  It already is?  You’re a quick one.
  10. Import Settings – Image Tab (click to Enlarge)

    File – Import – Options.  Click the Image tab.  Make sure that “File Pixel to Video Mapping’ reflects the aforementioned “601 SD or 709 HD” Setting.

  11. Select the “OMFI / AAF” tab.  Make sure the OMFI / AAF options are set to “use the source file’s resolution’. This will ensure Avid doesn’t attempt to cross convert your file!    Also, choose what drive the media will be copied and stored to.
  12. Select your file and Go!

You’re waiting less now, I hope!

Import Settings – OMFI / AAF Tab (Click to Enlarge)

I find the speed benefit to be about 80% faster (trials done on a Dual Quad Core 2.93 GHz ‘Nehalem’ MacPro, a Mid 2007 Intel Core Duo 2.4 GHz  ‘Merom’ MacBook Pro, and a HP Z800 Dual Quad Core 2.93 GHz PC)

I do want to make one point clear:  This speeds up the import into Avid.  You still have to wait for the initial encode to be done.  Depending on your encoding software, it could be faster (or slower!), but it DOES free up your Avid to work while the file is encoded elsewhere.  Also, the concept of “pay now or pay later” comes into play.  Would you rather wait at the front end of a project for the import (less stress), or wait until the END of the project (more stress) – with deadlines – and deal with the transcode / render / mixdown then?  Also, think about how the creative edit will be impacted when things don’t play in real-time.

As always, feedback is welcome.

29 Comments
  • Evan
    Posted at 13:48h, 16 November Reply

    This really is a time-saver. I always try to ask for DNxHD-encoded Quicktimes whenever I’m on a VFX-heavy film, but unfortunately most of the time it’s not possible. I guess since most render farms are linux based, and since the Avid codecs don’t exist for linux, you can’t ask the farm to encode with the Avid codec. But if you’re not rendering on a linux-based farm then this is definitely the way to go.

    • Michael Kammes
      Posted at 14:31h, 16 November Reply

      Hi Evan-

      Thanks for the feedback.

      There actually IS a way to READ DNxHD in Linux.

      FFmpeg has had the ability for over 2 years know. Via CLI with a -F switch, you can force MFFmpeg to play the file. It works great here (ya just have to force it). Since FFmpeg is based off of the libavcodec library, the ability is there.

      An associate wrote a batch file (in Windows), so whenever I drop a DNxHD (MXF wrapped) file on it, it opens up FFmpeg with a -f switch and whammo, instant viewing of DNxHD – and in it’s Avid MXF wrapper to boot. Obviously, this would require a bit of know-how to go from the decode to encode, but the framework is there. However, I do believe a DNxHD .mov should be fine even without a switch. Since FFmpeg has the abilities on each platform…..

      Aside from that, lower-cost Apple Compressor farms- or even the really good Telestream Episode Engine Farms – this works great.

      I thought Autodesk products (Linux based) Smoke, Lustre, Flame, etc. all understood DNxHD Media.

      Per Autodesk (v. 2011+)

      Avid MXF Limitations
      Keep in mind the following limitations when working with Avid MXF files.

      ■ Support for Avid MXF files is currently limited to DNxHD encoded files. IMX, DV and uncompressed formats are not supported at this time.
      ■ Conforming of Avid AAF -> Avid MXF files is limited to material generated from an Avid system that does not reference P2 Spanned clips.
      ■ Conforming of Avid audio MXF files imported from stereo source material and split into single channel MXF files will only relink to one channel.
      ■ Problems may occur when MXF files greater than 2GB are accessed through NFS, due to a limitation of the NFS protocol. This may cause the application to crash.

      If they can then there has to be yet another solution!

  • Evan
    Posted at 15:11h, 16 November Reply

    Sounds promising that there might be a way to encode new DNX quicktimes, but I think unless it’s easy most VFX houses won’t care enough to put the time in to figure it out. From their perspective a Photo JPEG QT works just fine (and can use less space), but yeah, then we in Editorial have to spend time converting it on import.

    I also think they probably don’t get a lot of requests for Avid-ready QTs, since a lot of people don’t know about the Fast Import option and how much time it can save them (hence the article!)

  • Scott Simmons
    Posted at 06:30h, 17 November Reply

    Great article. Very good read for those that don’t understand DNxHD.

    What would really help out us all even more is if there was a good compression utility that could transcode files right to Avid flavor mxf formats. Redcine-X can do that. Even better if was affordable, like Magic Bullet Grinder.

    • Michael Kammes
      Posted at 07:41h, 17 November Reply

      Thanks Scott.

      There are a number of compression utilities that do it.

      However, this is a feature most often found in higher end products (i.e. PAID!). Root6’s Content Agent can do it, for example. However it;s $10,000 – starting. Telestream’s Flip Factory can do it, although the price is even higher. Metafuze can do it….but is painfully slow.

      It’s funny – Avid’s DNxHD codec is a SMPTE standard AND Avid has their Avid Media Toolkit, so all the material needed to do it is out there. Just need it to be financially beneficial for someone/thing to find the R&D time and be able to profit from i!

  • Scott Simmons
    Posted at 09:21h, 17 November Reply

    That’s what i don’t get. It’s all the expensive products that can create Avid .mxf but not the affordable ones. I guess they haven’t seen a market or call for that yet. Let’s hope they do as MC5 gets into more people’s hands. We know it can be done (Redcine-X) as this little tool can do it even though it’s not a dedicated transcoding utility:

    http://www.videotoolshed.com/product/15/offloader/2

    • Michael Kammes
      Posted at 10:45h, 17 November Reply

      I’ve never heard of the app; I’d love to hear feedback on it.

  • Pingback:NLESystems Blog » Avid is Self-Aware – Fast Import
    Posted at 12:57h, 30 November Reply

    […] effectively – however, this is usually 70% – 80% faster than a traditional straight import. read more… Tags: AMA, Avid, DNxHD, […]

  • Eric Wise
    Posted at 12:08h, 17 February Reply

    Great tip Michael! I don’t think enough editors know about pre-encoding or “pre-rendering” as you call it. This should be a SOP if it isn’t already.

  • Chris Foley
    Posted at 11:56h, 11 April Reply

    Great article… Is there any benefit to using an outside program to make it DNX other than saving Avid-time? In other words, if I can just transcode overnight, wouldn’t that be the best way?

    Seems like one downside to pre-converting would be an extra copy of the media… in effect, you’d have: 1.Cam Originals, 2.Dnx conversion, 3.Avid file

    • Michael Kammes
      Posted at 09:05h, 14 April Reply

      Hi Chris, sorry for the delay in responding, I’ve been MIA at NAB!

      There are several advantages of using a 3rd party app to accomplish this.

      1. As we’ve discussed, you don’t tie up your editing bay.

      2. Processor usage. Media Composer isn’t really efficient as it could be when importing – thus (as you’ve probably seen) it can be slow. 3rd party apps Like Telestream’s Episode can saturate the pipe, so to speak, and generate usable files quicker.

      3. During Import, if Media Composer does not recognize a file, or runs into an issue, you’re stuck. Media Composer waits for you to Abort, Continue, etc. Leaving the app to chug overnight importing is good in theory, but without a pair of eyes, you have to *hope* the import went 100%. I believe this is one of the reasons why Avid added in the ability in 5.x for emails to be sent when tasks are completed.

      Yes, this certainly eats up more space. I think it’s the lesser of 2 evils: Do you want to have a horrible editorial experience with non RT playback of media and effects and very long export times (but save space) or have a good (dare I say pleasant) editorial experience but take up extra space? Pick your poison, I suppose. You do have the ability to delete the DNxHD.mov files AFTER you fast import them into Media Composer – remember, Media Composer is copying them and wrapping them, not using the file as it sits.

      ~Michael

      • Chris Foley
        Posted at 12:09h, 14 April Reply

        That makes sense… (I have definitely had those “Hold my breath and see what happened overnight in the edit bay” moments.) Thanks for the wisdom, Michael!

  • riaz khan
    Posted at 16:43h, 11 June Reply

    Dear mic, we been runng offshore production in london, issue is our filming happens in Europe, offline footage goes to india for editing, we are not typical avid guys, transforming ourself coz of DS, right now we capture HDV footage through media composer dv25411 mxf, transcoded to 15.1 mxf for the offline edit, we send it to india for the edit, we receive back 15.1 attached to the media composer relink actual footage, this is what we been doing now, its really look rediculous, but its our self made way, i will be happy to hear is there any effective way of doing this? any posbltiy to use third party software for the transcoding,
    thanking you in advance
    riaz khan
    CEO
    sasyinternational

  • Zach
    Posted at 17:17h, 22 June Reply

    Fast import is awesome and a great time saver once you get the settings right. Pair it with Avid MetaFuze – Avid’s new utility that will convert your HD quicktimes (of any flavor, but its really fast if they are Avid DNxHD already) to Avid’s native MXF media and an ALE, which you can copy into your MediaFiles folder and use immediately. Import the ALE into a bin and there are all your clips, complete with original metadata. It’s kickass! No tying up your Avid waiting for files to import!

    • Michael Kammes
      Posted at 17:24h, 22 June Reply

      Hey Zach:

      The problem is that Metafuze (almost 3 years old at this point) is painfully slow – slower than a traditional import into Avid AND it’s PC only (natively). Thus, we gain no time benefit by using Metafuze, only the ability to NOT tie up the edit bay. I, for one, would spend money on a fast transcoder than wait on Metafuze.

      That being said, Metafuze DOES handle some oddball formats other transcoders don’t (it’s based off of Avid DS code), so I guess the user has to pick their poison depending on codecs and budget.

      ~Michael

  • Febriano Jody
    Posted at 10:35h, 26 June Reply

    Dear Michael,

    Many thanks for this great articles.. 😀
    but i have few questions.. are this setting can be used in PAL world? because i’ve been using your setting except for that resolution setting. i’m using DNXHD120, because i’m living in PAL country. The project setting i’m using is 1080/25. But when i’m trying to import the file that have been pre encode on sorenson squeeze, it wouldn’t do the fast import. Avid still re-encode the files. I’ve change the color space both in avid and squeeze to 701 but it still won’t do fast import. did i made a mistake? or the fast import just won’t do on PAL?

    Regards
    Jody

    • Michael Kammes
      Posted at 11:15h, 26 June Reply

      HI Jody:

      PAL or NTSC or SD or HD – it makes no difference to Avid for Fast Import. I’m suspecting the file wasn’t encoded right, if your Avid settings match what I have above, albeit with different frame rates.

      ~Michael

  • Jim Mahan
    Posted at 10:50h, 10 November Reply

    Is there a way to determine the details of a DNxHD file. (ie which codec was used to create it)

    • Michael Kammes
      Posted at 14:44h, 10 November Reply

      I don’t quite follow. We can examine the codec (i.e. how the video is encoded) – but as far as which *program* created it – those are 2 different things.

      I use “Media Info” quite often, and Google will show variants for Mac and PC. The program which created the file – that’s up to the software to leave a marker embedded in the file, which I rarely see.

  • Vladimir Pavlovski
    Posted at 14:46h, 21 November Reply

    Rhozet Carbon Coder on Windows transcodes almost any media to Avid DV, DNxHD MXF media with AAF. Then just drag and drop. No need for fast import.

    • Michael Kammes
      Posted at 20:46h, 21 November Reply

      Yes, it does.

      Sadly, it’s is very expensive ($20,000+ for a turnkey system at last check) and is tricky with all metadata being passed. It does work well in enterprise installations, however.

      Thanks!

  • aleigh
    Posted at 18:25h, 23 January Reply

    HI. I am a confused first time avid user! I am an expert FCP user but trying to learn Avid while editing a short film. I am working with all 5D footage which I converted to prores. Is it necessary though to convert it to DNxHD?? I am using the free trial download of 5.5 and then planning on using the free trial of 6.

    If using the prores footage, should I set the color space to YCbCr 709 or RGB?

    Thanks!

    • Michael Kammes
      Posted at 13:17h, 27 January Reply

      5D footage is H.264. H.264 is an acquisition codec – not editorial. While Avid can use this natively (v5 and v6), the editing experience may not be “pleasurable”. H.264 can be difficult for computers to play, and thus you may experience playback issues like stuttering, etc. when playing multiple streams or on an older computer. Converting the footage to DNxHD ensures that you are using a codec that not only plays easier, but holds up better after subsequent FX, renders, and export.

      In v6, you can do this same fast import trick with ProRes footage which is excellent.. The same holds true – 709 is the correct color space.

      Good Luck!

  • Andi Loor
    Posted at 09:24h, 28 January Reply

    For your information: Red Cine X is a utility to handle Red Raw files and can be found at http://www.red.com

    • Michael Kammes
      Posted at 09:54h, 28 January Reply

      Thanks Andi!

      I was referring to the Video Offloader product Scott had mentioned – Red Cine-X has been in my tool box for some time 🙂

      Thanks!

  • Pingback:Diary of An Avid Switcher - Part 2: Understanding The Basics | Jonny Elwyn - Film Editor
    Posted at 09:10h, 02 March Reply

    […] camera files, the DNxHD .movs and the MXF re-wrapped files now in Avid Media Files. Check out the comments section from Michael Kammes great post which was part of Jason’s inspiration for this workflow. Michael suggestion that while 3 […]

  • Nuvagenic Reviews
    Posted at 18:01h, 19 July Reply

    Wow that was unusual. I just wrote an incredibly long comment but after
    I clicked submit my comment didn’t appear. Grrrr… well I’m not writing all that over again.
    Anyways, just wanted to say excellent blog!

  • Relmar Boys
    Posted at 08:56h, 22 August Reply

    Buy this guy a beer. Or a deer.

    Great solution – that shoddy import bar was one of my main drawbacks from using Avid.

  • Angry Mike
    Posted at 10:48h, 06 November Reply

    I will buy you a case of that stuff. Break-neck solutions, thank you awesomely much for that advice.

Post A Comment