My viewing experience is tainted by the “cleansing” of perceived foul language and/or content by network and cable censors.

Issues to consider:

1. No station is going to simulcast 2 streams of video and/or audio just for language. This will eat up satellite bandwidth and cost millions.
2. You don’t fuck with the FCC.
   

Thus, we are left to find a way to selectively filter out content using existing technology and infrastructure.

Here is my spitball concept:

During Editorial – within Avid or Final Cut Pro – a marker could be placed in the timeline at the start of the offensive word / and or phrase. These markers, when exported, could easily become metadata within that exported file. As that file gets transcoded, moved, and subsequently broadcast, this metadata “flag” could trigger a process within the end user cable box to replace the offensive audio with a bleep. This keeps the censorship on a per household and per TV set basis – i.e. the adults make the call on what’s acceptable; not the station. As an added bonus, there could be parameters for types of markers: language, violence, religion, etc…think of a V Chip on steroids.

A modification could also be made to this workflow to have the “flag” be automatic.

How?

Closed Captioning requires transcription of the spoken word. This is the encoded into the video signal and timed intervals. A filter added to the broadcast center gear would allow any closed captioning word or phrase that is equal to a set of parameters, to effectively be censored. Closed Captioning also has other “hidden” metadata, that could also be used to trigger the censor, if the text is not deemed accurate enough. (Remember, closed captioning text is not always in sync with the audio – and closed captioning can always be turned off by the user.)

What problems this does present is:

1. What if the system ever fails? Broadcast outlets could face steep FCC fines if someone tripped a power cord and Pulp Fiction was broadcast sans bleeps.
2. What if people are using over-the-air HD sets? No cable box would force the brodcasters to have the security in place – not the end user.

Penny for your thoughts?

Matt Stratton turned me on to Clicky Stats awhile ago, and through Clicky, I’m able to see what web searches come up with my blog as a result. I also see when these searches lead them to my blog – but then they leave because they didn’t find the answer. I thought since the search engines think I already have it on my site, perhaps I should.   Below is a sampling of the search queries that my website supposedly already has the answers for.  So, I present to you:

It came from the Searches, Volume 1.

dnx-36 firewire 400 and bandwidth requirements DNX36
Yes, DNxHD36  (aka DNx36) can certainly work through a firewire400 pipe. DNxHD36 runs at a 36mbps, while firewire400 has a theoretical max of 400mbps. This makes firewire400 a fine choice as a transport for DNx36 offline workflows.

29.97 and dnx36
This is a source of confusion for many Avid users, and I’m in the process of authoring a blog on this. DNxHD36 is a “family” according to Avid. While the “36″ label refers to the data rate for 23.976 material, the DNxHD36 “family” codec can be applied to 29.97 material. This would be technically known as DNxHD45 (45mbps as opposed to 36mbps).

“episode engine” cpu restriction
This is kind of vague. Within Episode, you can assign priority of encodes and cpu usage. Quicktime already is pretty poor with processor usage to begin with, however.

Metafuze Lossy?
Yes. After all, you’re transcoding from one codec to another – there will always be quality loss. In addition, DNxHD is a lossy codec – so, yes.

edit room day rate
This varies wildly. Depends what gear is in the room, what the room is used for (editorial, audio, finishing, color, etc.) your experience, and geographical location – and hell, what your rent is! Normally a reputable rental rate at a good facility would be a couple hundred bucks per hour or even much more.   As a side note / soap box,  don’t sell out the rest of the industry by undercutting other facilities.  Talent and experience are worth the money.  That’s a majority of what the client is paying for – not the gear.  By undercutting everyone else, the industry is forced to adopt your lower rate as a standard, then everyone has to drop their prices.  And the cycle continues. You’re shooting yourself in the foot.

Does clipster support facilis
Clipster runs on Windows XP 64bit, which Facilis has a client for. Fibre is the best choice (Ethernet won’t deliver the bandwidth needed), and it should be a recent version of Facilis software to enable file locking – volume locking will cause organization issues, but can be done.

convert red footage cluster setup
I wish I could do it better than these guys have:
http://www.kenstone.net/fcp_homepage/compressor_multi_cores_stitzer.html
http://www.fcsoutlet.com/home/Studio_Outlet/Entries/2008/9/1_Virtual_Clusters_-_Compressor.html

fcserver script
Good luck. Most everything is manual at this point. Try CATDV as a more GUI based alternative.

QIO-PCIE sonnet tech
The QIO comes with either a PCIE host card or a E34 (Express slot) host card. Unveiled at NAB 2009, and finally shipping now! Mac only support for now. List Price is about $1000.

FTP, KiPro
Well, once the KiPro encodes to ProRes, you can certainly FTP the file – although it’s gonna be huge. Your best bet is to offload it to a portable drive for transport or use a transfer solution like StorageDNA.

share msm database avid
I can only assume you’d like two or more machines to access the same Avid Database at the same time, in order to share media. Nope – the functionality you are attempting to do is achieved through Avid Unity, ISIS, and Interplay.

final cut server frontend
Final Cut Server (FCSvr) clients have a front-end based in Java; it looks the same on Mac or PC – and the administrator can restrict what functions a client can have. A popular alternative is creating a web page interface which ties into the database and media of FCSvr. The functions on thew web page are usually limited and typically only used for “review and approval” functionality. Several companies out there can create a custom web based interface – although it is far from cheap.

episode pro pro res 422
Yes, If Episode is running on a Mac, you can encode and decode into any of the flavors of “ProRes” (1 word). Decoding is only available (currently) on the Windows version.

Of course I’m (initially) referring to the almighty DVD, which all of you assistant editors need to burn daily – all, naturally, with different watermarks and circle takes or various cuts.   Plus, that Blu-Ray for that Director with the fancy car – you know who you are.

However, more and more above your paygrade individuals who foot the bill for your paycheck are using non-traditional devices to view media.  The web has undoubtedly become the new standard, with petaflops of media files being FTP’d daily. Flash, Windows Media, and heavily Compressed Quicktimes are all popular options – and many times, are all needed.  New fangled video enabled mobile devices – like the iPhone, Blackberry, and Droid – have increasing become the way to view video while on the road between business meetings and Starbucks pit stops.  Those facilities that have been wise to adopt Digital Asset Management (DAM) internally will also need a version for tracking, as well.

So, yes, dear assistant editor, you now have several DVDs, web versions, and mobile versions to create.  Plus, there is always – always – that oddball version you need to create.  It’s inevitable.  All in time for the next edit session tomorrow.

This is when you need encoding.  And a hellova lot of it.

I can only hope that all of you, dear readers, are familiar with the concept of a Quicktime (QT) Reference.  No?  Then Unca Michael will enlighten. 

A QT Reference is simply a pointer file.

Quicktime file vs Quicktime Reference file (Click to Enlarge)

It’s a small media file that appears as a full movie file to most media players, but during playback, it points back to the initial media used in the generation of said QT reference file.  Think of it like that bookmark in your web browser.  That bookmark isn’t the entire website, but it pulls up the website when you click on it. It’s a link.

QT references are your best friend.  Why?  Most NLE’s can generate a QT Reference much faster than exporting a whole “complete” file.  Thus, less time watching a progress bar.  (Although, ironically, having the wait time does give you time to get coffee to stay awake and watch the progress bar longer.  Interesting.) This speed boost is accomplished by having the NLE only create new (render) files for media that has some sort of effect on it in the timeline.  All other media is untouched, and thus, the QT Reference can point to it.  Brilliant!

So, I’ve already saved you oddles of time.  Waiting.

Now, I need to earn that bonus, and you need to take this QT reference goodness and utilize the strength of it’s pointers to create the compressed files needed for your deliverables.

At this point, it may be a good time to check out this post, as I explore choosing the right encoding solution.  In a nutshell, you need an encoding solution that can accommodate all of the formats you need to deliver.  It’s mighty beneficial to have a solution that can encode into multiple formats at once.  Again, less thumb twiddling time.

Here is where I drop the “But…” bomb.   I must tell you something vital when dealing with QT references:  Your encoding solution MUST sit on the same network as your edit system and media storage.  Huh?  The QT reference is aware of where the original media is in relation to itself when it is created.  If a user were to say, I don’t know, email the file, or even move the QT reference to another folder (even on the same system), the link to the original media is lost.  Yeah, big gotcha.  That’s why I recommend having your encoding weapon of choice on the same SAN/NAS as your edit bay…or have a firewire drive on standby to copy the QT reference and media (retaining the same hierarchy) to move from the edit bay to the encoding solution.

Moving a Quicktime Reference out of it's natural habitat (Click to Enlarge)

Solutions like Root6’s Content Agent (see my review here) are actually built explicitly for this purpose: not only can Content Agent sit on any Windows compliant SAN, but the system can handle concurrent encodes, auto FTP files, email status updates, and automate DVD burning with watermarks.  And while I’m sure burning and labeling those DVDs fulfills your life’s purpose, there is even a more efficient way to streamline that.  Look into Rimage:  Automated network DVD authoring and label printing, scalable from small runs to large runs.

While I am sure this will not keep you from many late nights in the edit bay, it very well may give you a few more hours to crash on that old couch in the back of the room.

Encoding solutions are much like interns: you have no idea what they are fully capable of, you have no idea how well they can multi-task, and there sure as hell there are a lot of ‘em.

Thus, we need to develop some baselines with which to judge perspective candidates.  There are several yardsticks with which to measure these by.  But which ones?  Features?  Speed?  Cost?

Let’s talk Lowest Common Denominator for today’s economic times: the almighty dollar.  When applied to encoding technologies, this yields three encoding solution categories:

1. Software Encoders
2. Hardware Encoders
3. Hardware with Software Encoders (complete bundled solutions).

Most software solutions – that is, no dedicated video hardware appliance or card – will run under $10,000.  They range from free to several thousand dollars.  Sorenson Squeeze and Apple Compressor, for example, come free with Avid Media Composer and Final Cut Studio, respectively.  They typically lack in advanced features, such as: robust format in/out support (codecs and wrappers), as well as concurrent & efficient encoding.  Hey, ya get what you pay for.

Higher end software solutions, such as those offered by Telestream (Episode family, Flip Factory family) cost $500+, and add varying degrees of features, including concurrent encodes, image adjustments, and the like.  The last spec sheet I saw from Telestream listed half a dozen pages of formats per product.  (I recommend you check out their website for specifics).  As you’d expect, the more money = the more features.  This usually manifests in broadcast and VOD feature sets and automation (watch folders, notifications, etc.)

This is where I present you, Teeming Millions, with good news and bad news.

Good News:  You can (most likely) use your existing computer to run the software!

Bad News:  Your encode times will suffer if your existing machine is sucking wind.  (My modded Dual 1.8GHz Quicksilver, for example, encodes slower than my Droid phone)

Thus, buying a new CPU – and the best one possible – will slash those encode & transcode times.  However, an industry standard Mac Pro or HP Z800 will cost several thousand dollars.

Lastly, this solution obviously leaves you high and dry when it comes to capturing from or laying back to tape.  Short of a splattering of solutions which handle firewire based solutions, those aging analog decks will have to remain very large paperweights.

The next level is Hardware Encoders.

It’s long been known that hardware encoders give better looking encodes.  It’s also known that they are usually faster than their software counterparts, due to the main fact they’ve been built for one purpose, and one purpose only: to crank through that file, smile, and ask for more.

This comes at a premium, however.  Flexible and robust hardware encoders are expensive – no matter which of the two popular methods you choose:

1. Purchase a hardware card for your computer (Digital Rapids, Inlet, Content Agent, to name a few).  This allows you to utilize the power of the card for encoding.
2. Purchase a standalone hardware appliance (Telestream’s Pipeline family, for example).

No matter which method you choose, you will still need a CPU as the host.  One benefit of choosing a hardware solution is that the software that accompanies the hardware is a bit more robust than the standalone software solutions.  A little somethin’ for the effort, if you will.  This can include increased advanced format support, concurrent and/or distributed encoding, logic based encodes, and higher levels of automation.

A majority of the cards and appliances also have the added benefit of audio and/or video spigots on board to interact with video decks.  If anything in your workflow requires the use of a deck, then a hardware based encoding solution is your only viable option.  In some cases, these solutions can handle some of the work your expensive edit bay does, such as batch capturing from an EDL (or ALE).

I’d be remiss if I didn’t mention Hardware Accelerator solutions – that is, computer cards that have no I/O, but serve as a way to encode faster by offloading the encoding process from the CPU to the internal computer hardware accelerator card.  These, too, usually cost several thousand dollars and often only accelerate the encode of a small number of codecs.  Beneficial for some specific workflows, but not very flexible when multi format deliverables are needed.

Lastly, we have the highest echelon: Hardware with Software Encoders.

In most cases, it’s a qualified, bulletproof solution that has a tricked out CPU, video I/O card, a dedicated hardware encoder, and possibly storage, tuned perfectly by the manufacturer to be as efficient as humanly (err, mechanically) possible.  You get the whole machine from the manufacturer.

The system typically leverages a combination of hardware encoding and software encoding to produce the best quality image in the shortest amount of time.   These solutions also tend to be ala carte – you pay for a feature or codec you may need.  The base software is powerful, but specialty codecs and or abilities are a pleasant add-on to the bottom line.  Omitting these is great when you need to save money, but a headache at 3am when you suddenly need that feature.  These integrated solutions offer the highest level of automation, quality, and flexibility – but usually run over $25,000.

Whenever I need to determine the right solution for a client, I have three main points I touch upon:

1) Input:

Is the client capturing baseband video?
Is the client simply transcoding existing files (or, using NLE generated files)?
Does the client have a large volume of media to encode in a short period of time?

These questions determine if the client needs video I/O ($10,000+), and if the client needs a robust software solution that can do multiple encodes fast (encode farms)

2) Output:

Is the client outputting baseband video?
Is the client simply creating digital deliverables?
Is the resulting encoded media going to a broadcast facility, NLE, web, or film out?
What formats does the client need to encode into? (The more detailed the better!)

These questions determine if the client needs video I/O ($10,000+), and if the client needs a more expandable (read: expensive) solution which can handle professional deliverables.

3) Budget

This, of course, dictates what solution you can provide in the pricerange – and when to break it to the client that they need to re-think the situation.

Good Rules of Thumb:

Encoding from a baseband source – using a professional, non kluge Rube Goldberg solution, will cost close to – if not more – than $10,000 – without the CPU.

Most encoding solutions still require the purchase of at least 1 CPU.  This is obviously not true at the highest echelon – but for most solutions – the user will need a powerful CPU.

Professional formats, (broadcast and sometimes Flash) require the purchase (licensing) of a 3rd party codec – usually from the encoding manufacturer.  See: ala carte.  This is why knowing the desired input and output format of the files is paramount.

And lastly, this is the most common question I am asked – and the one I still have the most problems answering:  How long will the encode take?

Encode times vary wildly, and depend on CPU, hardware, source format, output format, encoding settings, storage, and length of clips – to name just a few variables.  It is nearly impossible to give a concrete answer.  It’s akin to choosing a camera: you need to test it, and see for yourself.  Your workflow will be different than everyone else’s.

I’ve assembled a comparison of popular encoding solutions, based on cost (MSRP). I’ve included base configurations, as well as “loaded” configurations – with all of the bells and whistles.

Comparison of popular encoding solutions (click to enlarge)

So, we’ve already built the logic which determines the parameters for the encode(s) via the web browser interface, and delivered that to the encoder.  But now we need to get the files to be transcoded to the encoder.  If you’re on a LAN (internal network) then it’s time to call the I.T. nerd, and have them map the path from your encoding solution to your computer.  This creates a link to the encoding solution and your computer which houses the media.  Both systems now “see” one another, and given the proper permissions, they can read and write each others files.  A user can now move the files to be transcoded (sometimes as easy as drag and drop) into the appropriate folders on the encoder.  At this point, the encoder utilizes the settings from the web interface, applies them to the files, which the user has just moved and WHAMMO – encoding begins.

Further automation is very possible, including setting up watch folders on the encoding solution to make copying of files easier for the end user.  Instead of having to navigate to a specific and oddly named folder, a user cpuld drop the file on an easy to read and locate network folder.  Since the encoding solution is constantly watching this folder, it’s as if the files were deposited directly on the encoder.  Many solutions can also send emails on the status of the encodes, and can even do their own housekeeping – moving and deleting files to maintain free space and organization.

To streamline the process further, solutions like StorageDNA have the ability to copy files over your network faster than a traditional drag and drop.  Add that feature to delta synchronization, encryption, and a host of other features, and now you have a pretty efficient solution.

This solution is perfect for several users needing to access the encoder on a private, local network.  And guess what?  Short of LAN acceleration, some solutions actually already do all of this.  Which is why this workflow doesn’t stop here.

Let’s say you’re remote.  Or – better yet – you want to start a business for encoding for the teeming millions out there?

That presents a bit of a problem.  How do we get the media to the Encoder?  How to we marry the logic based encode parameters to the media?  And most importantly, how do we bill for it?

With DAM.  Digital Asset Management.

DAM solutions are best suited for their own tech column post (forthcoming) but essentially, it’s a piece of software that organizes all types of data – media or otherwise.  It catalogs every conceivable piece of data about the file (metadata), and enables a user to search across all of the fields – including custom ones.  By depositing the web based encode parameters into the DAM, we now have a record of the user, their encode parameters, and a place for the uploaded media to reside where it can be referenced by the DAM.  All of this data can then be dished off to the encoder, where the machine chugs away.  The encoded media is then returned to the DAM, where information like “encoding time” is cataloged and organized.  A report can then be generated for the client.  Client’s can then be billed for the encode time.  Some DAMs, like CatDV and Final Cut Server are extensive enough to allow for backend coding to allow for a review and approval process.  (See my last entry on FCServer.) This would allow a client to, say, watch several versions of their encoded file (with a watermark, natch), pick which file best suits them, and then download.  Oh, and pay.

Storage DNA also falls into this nicely.  It offers the same benefits on a LAN as it does a WAN.  Faster transfer times, resuming broken connections, and even a web interface to track the files.

I know that’s quite a bunch of tasks and features for these puzzle pieces to “automatically” do.  While the basic framework (“hooks”) for this workflow is built into many DAM solutions, don’t kid yourself – this will require some forethought and engineering time.  But it is possible.

The last and final step is to optimize your encoding solution.  For high volume batches, consider using software that scales.  That is, software that can distribute the encoding work to other machines to get the trancoding done quicker.  Telestream’s Episode Engine family is excellent at this.  Apple’s Compressor can also accomplish this – albeit with limited format support.

Here is our finished workflow.  Pat yourself on the back.

Completed workflow. (Click to enlarge)

Venn Diagram of Encoding Ability

The post industry lives and dies around the concept of deliverables.  What specifications have to met to appease the viewer, server, or engineer on the other end.  Many times, just getting the deliverable out is a chore in itself.  The last encoding format sheet I read from a leading encoding manufacturer had 5 pages of supported input / output formats. Being able to decipher these often cryptic encoding acronyms and numeric values appears to need a degree in engineering.

This is Science.

Translating from one medium to another has it’s own inherent problems.  Does the image blur or breakup when the camera pans?  Is the audio intelligible after it’s compressed?  Do the lettering or graphics remain legible when the image is resized?  Often it takes a slow hand (and a smooth touch) to know when to finesse parameters to convey the artistic vision when transcoding.

This is Art.

Depending on the deliverable, one may have options as to what path is the easiest to walk down.  Is a certain codec really the right choice?  Will it play nice in all browsers or devices?  Will it cause the media to load slowly?  These kinds of decisions are based on thorough knowledge of the medium, and the gotchas associated with the platform.

This is Experience.

There was a time when all three of these traits of these were the core component of a now fading job: The Compressionist.  Like a Colorist, their job was highly specialized, and had a job to ensure the best quality and visually appealing image once the media left the edit bay.  Our attention span has now waned, and our deadlines have been cut short, and one click presets are rapidly becoming commonplace when outputting.

There has got to be a better way.

Replacing the job of a Compressionist – the human eye, the human attitude, the human heart – cannot be replaced.  However, we can begin to bridge the gap with what I am calling “Conditional Encoding”, or for you coding geeks, “Boolean Encoding” (patent pending).  Of course, both my employer and myself would prefer KCM: Kammes Compression Methodology.

There are baselines that can be assembled for specific criteria.  This, coupled with easy to understand non-techie based concepts yield a easier path for users to navigate: An example:

Does the source media involve fast motion?

User: YES

Encoding System:  increase data rate.  Change motion estimation to a higher level.  De-interlace if the file is interlaced.  Perform a 2-pass encode.

User: NO

Encoding System: maintain average data rate, reduce motion estimate factor, perform 1 pass

Does the media include large amounts of dialogue?

User: YES

Encoding System: Use less compression on audio, apply equalization favorable to vocal frequences, reduce volume on non vocal frequencies.

User: NO

Encoding System: Use more compression, apply broadband equalization.

Logic Based Workflow

So, we can essentially build rules based on the human perception of the source media.  By utilizing these within a Boolean conditional set  (IF, THEN, ELSE logic), we can very quickly create a set of encoding parameters “closer” to what a Compressionist would choose.  While they will not replace the role, they will certainly get the dart closer to the bull’s-eye.

Think of Choose Your Own Adventure books.  Each subsequent decision is based on a prior decision.  By incorporating this methodology, this logic could create the desire encode.   Once applied to each deliverable, the process becomes streamlined.

Another key to this is presenting users with options and descriptions which translate between technical and laymen.  Bitrates could instead be replaced by “need more space”, or “higher quality”, for example.  Logic would have to be configured and scripted in to the parameters to balance the users options when answers conflict.

To further efficiency, this could be incorporated into a familiar interface which users could also readily utilize and understand, like a web page.  This means an easy to use GUI for the user, the possibility for remote access, and the ability to be ported over to many platforms.

The webpage could then conceivably generate a data file that the encoding solution could parse and understand.  Formats like XML have long been a web standard, and several encoding solutions already utilize this format:  Root6’s Content Agent, Digital Rapids Stream software, and several of Telestream’s product offerings already have this functionality (albeit limited) built in.

Several consumer software solutions have “wizard” functionality, which incorporates a slimmed down version of this concept.  Apple’s Compressor, for example, has once-click device encodes (DVD, iPod, etc.), or time-based encodes (how much material do you need to fit on a given medium).  Each of these selections loads up a preset of encoding parameters.  This Boolean Encoding brings this concept to the next level.

Compressor Format 'Wizard'

Compressor's DVD Encoding Presets

I can see it now: The Microsoft Paper Clip A.I. turned encoding vehicle.  I think I just died a little inside.

Next week, I’ll explore a much larger scale workflow which not only incorporates this concept, but utilizing it as a possible side business – or even a current business enabler.

Pre-rendering.

Wha? Huh? Isn’t that an Oxymoron?

Hang with me here for a minute.

Q. What necessitates media needing to be rendered?
A. Well, the fact that I put 10 different effects on one clip, including this groovy flying toaster effect..

Q. OK, slow down, Orson. Do you know why you have to render those?
A. Well, because my computer is too slow.

That may not be the case. Codecs – what your media is encoded and decoded for playback with – are designed to use as little storage as possible when encoding a media clip. Often times, the trade off for a small storage footprint is a media file so compressed that your computer simply cannot play those effects in real time. Yes, that $10,000 computer may be able to play the raw clip, but it simply cannot decode the media – and put your flying toasters and star wipe on it, along with the other tracks of video and star wipes – without dropping frames. Hey, it’s not the computers fault. You’re trying to make it do something it was never intended to do. Thus, you need to render. And render. And wait.

So, why not give your computer a break?

Give the computer a file that it can more easily decode – and put effects on – in real time. Yes, you need more storage, but is a little bit more storage cost worth your wait time? Have you seen how cheap storage is now?

Some tapeless codecs, like XDCAM, etc. are what they call “Long GOP” (Group of Pictures). Unlike the concept of film, where each frame is a complete, full frame picture, Long GOP based codecs only track the change from frame 1 to frame 2. This makes the information in frame 2 much smaller in size than a full frame of information. By only having a full frame of information every, say 15 frames, you save a plethora of space. This means longer recording times on those flash memory cards, and happier consumers. Until they want to edit it. And put flying toasters on it.

Other video compression technologies, like RED, AVCHD, or AVC-Intra, may not utilize Long GOP structures, but instead they compress the file even more – which makes your computer work even harder – just to play the file. Many of these compressed codec’s utilize encoding schemes, which can employ several other techniques to save space. Two of the most popular are:

1. The camera not shooting a full size frame of information, but shooting a smaller scale version, and letting the computer scale it up to full HD frame size while it plays back.
2. Shrinking the color space to save storage space, but yielding a file that degrades further in post and making it poor for chroma keying and VFX work.

Here is a good example to put things into perspective: HDV.  HDV is very common for consumers who want inexpensive HD on a small tape. HDV file size is over 70 times SMALLER than a clean, pristine, uncompressed signal. Your computer has to work much, much harder to play that file back and add effects – in real time. The same reason your car has issues pulling an additional 14,000 lbs.

So, you’ve now been sworn off of editing Long GOP and other uber compressed formats. What do you do with that groovy (pseudo) HD camera you just picked up?

This is where pre-rendering (finally) comes in. Take that compressed “camera original” media and “flip it” to a more robust codec that is NOT Long GOP based, or compressed to the nth degree.   There are dozens of software and/or hardware solutions out there, from your basic Apple Compressor, to your running-of-the-gamut solutions from Telestream.  This encode then yields a file that your NLE will love you for, and I guarantee you that your time waiting for that progress bar will decrease sharply. For you Johnny 5 “MORE INPUT!” data hogs out there, many encoding solutions out there have the ability to pass along the metadata inherent in those camera original files and incorporate them into the new files.  This means that within your NLE, you can sort, sift and organize, relying on the metadata introduced by the camera during recording.

Codec’s like Avid’s DNxHD or Apple’s ProRes have quality (bitrate) levels which are considered “broadcast ready”. They are of a high enough quality that almost any network would accept, in addition, since these codec’s are optimized for their respective editing platforms, they can do more nifty things in real time.

And, we have an added bonus: These “edit friendly” codec’s, like DNxHD and ProRes, hold up better in post. What does this mean? This means that the 10 effects you do place on the clip will not degrade the clip as much as the 10 clips you placed on your camera original. This makes for better compositing, better color grading, and eventually – a better end product.

So, set those files up and let your computer chug overnight – flipping those files into something less stressful. And those flying toasters? Don’t.

In this room, aside from the usual barrage of CD and DVD authoring, download and uploading of files, temp graphics and label creation, they did basic encoding, usually by a lesser paid assistant. Certainly not glamorous, but essential. Definitely not the first notions of what a post facility does: Offline and Online suites. Finishing. Audio rooms or dub stages. The flagship rooms.

Why?

Well, typically your talent – the editors who have named clients – command more in terms of pay than the backroom assistants, and the talents’ workspaces also have a lion’s share of the gear with which to make them shine.

Yes, I am of course referring to the almighty R.O.I.

With the current race to zero, rates for the client focused suites are continually dwindling, definitely at odds with the cost of gear and talent operating within them. Possessing a ‘Money Room’ already begins with less overhead – both financially and technologically.

So, what can you do in this money room to earn some of dat cabbage?

First and foremost: Encoding. Every website nowadays is content rich, from youtube to Facebook. Everyone has multimedia on their phones. All of these media riddled avenues necessitate a *special* and unique format. Those have to be created somewhere. Why not from you?

Utilizing your offline / online bays to chomp through 100 different formats for deliverables – when you could be billing for editorial or finishing in the room – is simply poor planning. A facility could conceivably upgrade to a new computer for the Online suite, and use the older machine as an encode station in the Money Room. This not only boosts the productivity (and marketability) of your Online suite, but also gives your Money Room a CPU to encode with.

So, I have a box with some processors. Now what?

Quicktime References. Have your offline / online bays and your encoding station see the same storage over a SAN. Whether it be via SMB re-share / Ethernet, Fibre, or simply cloned / portable firewire drives, these are a sure shot to not only increase productivity, but create a more efficient workflow. Have your editorial room export a QT Reference which links to the original media, then have the encode machine pick up that reference file, and let the number crunching commence. You’ve now significantly cut down the export time out of your edit bay (QT References are much faster to generate than a complete export) and also freed up the client bay for other activities. Hopefully more glamorous. Hopefully billable.

I recommend you create encodes for the following:

• H.264 or the like for web review and approval, or FTP uploads. Perhaps even iPod or iPhone versions.
• Flash versions for embedding in websites.
• MPEG formats for DVD or Blu-Ray dailies and/or screeners.
• Predefined proprietary formats for youtube, Facebook, myspace, Hulu, etc. Each site has its own requirements for submissions. Perhaps you can charge per location’s format?

Advanced encoding software packages allow for multiple simultaneous encodes on one machine, and some allow for distributed encoding over many machines. Others still utilize watch folders that are always looking for a QT Reference to begin encoding from, and even sets of parameters for multiple groups of encodes. What a value add it would be to tell a client that you could not only do the editorial, but give them deliverables for any destination they could desire – same day.

So, you’re not billing out your editorial room enough to justify something like this. I get it. As an example, this is where the promises of highly compressed formats (such as RED) being quicker can actually backfire and allow other revenue streams.

These abnormally large sized and compressed files are still a very, very intensive process for editorial machines. It takes a great deal to chomp through a 4K file – especially when 99%, if not more, of this material will never be viewed at 4k, or even on a playback system that would do it justice. More often than not, you will see it at less than 50% of its original quality – HD – or even less on broadcast TV or on the web. Given this truth, one can make an argument for simply downrezing the raw 4K footage to a manageable frame size and codec; like DNxHD for Avid or ProRes422 for Apple. This previously difficult to manage format in now in a much more edit machine friendly format for use in the editorial process. These formats can exceed broadcast quality standards – very appealing.

So you’re a purist – I get it. You want to have a 4Koutput. That doesn’t mean you can’t do a pre-encode (in this case, transcode) to an offline format for ease in editorial. Despite being suitable for broadcast, DNxHD and ProRes 422 – as well as DVCProHD – make create offline codecs, too. Provided the computerized tool (or the assistant!) does things right, your facility can matchback from the offline material to the 4K when onlining. Sound familiar? This is what telecine houses have been doing for years to DVCam –and charging you a ton for it.

I’m amazed at just how under-utilized this concept is: not only as a pure way to tighten ones belt, but to simply be more efficient. As an example, I happened to be at one of the studios here in Hollywood that gets the editorial output – rough cut and fine cuts – in one, and only one, format. A format that is antiquated and popular almost 10 years ago. Each department downwind of that facility spent hours encoding into a format they could use with their systems, meeting their visual and technological specifications. Imagine the amount of money spent in manhours working around this issue.

The film? Catch it this summer in theaters. Budget is $200 million.

**v 1.1 – added RedRushes for DNx36 & 220 & Pro Res 422 HQ batch encoding

I’ve decided to expand my testing after inquiries regarding other encoding solutions…and it developd into benchmarking single & batch encode times into various codecs with various encoding solutions.

Specs, standards and universal notes:

2.93 GHz MacPro, 6GB RAM.
10.5.6 / QT 7.6
Avid Codecs 2.0 (shipped with Media Composer family 3.5+)
All media local on OS Drive.

R3D Proxy _H quality was used for all tests.
Builds tested: 16, 17, 18.
10 clips ranged from 00:26 seconds to 04:31.  Median was 02:19.  Since every editor’s batch will be different, this was a ballpark for an average shoot.
All clips were resized to full frame HD frame sizes during encoding.  As a side note, the frame resizing from the native 2048 x 1024 to HD frame sizes was not a significant factor in the delta for encode times.
No LUTs or image adjustments (aside from resizing) were used.

Pro Res 422 HQ -The highest quality compressed HD codec that Apple offers.  Exceeds Broadcast standards.
DNx36 is 1080i/29.97 8bit. The lowest resolution of HD Avid offers.  Used for offline editorial.
DNx220x is 720p/59.94 10bit – The highest quality compressed HD codec that Avid offers.  Exceeds Broadcast standards.

REDCODE RAW Quicktime Codec: 3.5.0
FCP: v6.05 (FCStudio 2)
RED Final Cut Studio installer 1.0
RedRushes: v3.60
Compressor: v3.05
Compressor Local Virtual Cluster: 16 instances, all local.
Episode Pro (Desktop): v5.1
Episode Engine (16 Processor License): 5.1.2.  Split and Stitch disabled, as there seems to be a bug in the stitch process.

Final Cut Pro L&T:  Batch not applicable; Log & Transfer only processes 1 file at a time.  DNxHD codecs are not traditionally used within FCP.
Red Rushes:  Batch not applicable, only 1 file processed at a time.  Quarter Res Debayer Quality.
Compressor: Batch not applicable, only 1 file processed at a time.
Episode Pro: Batch not applicable, only 1 file processed at a time.

Findings:

Amazingly, those of you who use Final Cut Pro as your editor will find you have the seemingly fastest encoder out of the bunch – and free.  It does require some basic setup to get the cluster working – and is known to be flakey, but seemed to be a rockstar during my testing.

It should be noted that the free RED codec for Mac OS – REDCODE – is *still* a Quicktime component.  That means no matter what encoder you use, the QT component will be the bottleneck.  In addition, whatever bottleneck Redcode with QT causes, it’s only part of the equation:  The codec (in this case, ProRes and DNxHD) you are encoding to must be written to be able to take advantage of multi threading.

RedRushes utilizes REDline as their encoder, and seems to be the best at utilizing available CPU horsepower.  It averaged 15-20% more processor usage at any given moment then any other non batch encoder (FCP L&T, non VC Compressor, and Episode Pro).  That being said, this was usually only around 50-55% at best.  Batch encoders seemed to be able to take advantage of the remaining processor cycles, although Compressor with a VC seemed to be average 95-100%, whereas Episode Engine lagged behind between 80-85%.  Unfortunately, the Stitch function of Telestream’s Split & Stitch technology seemed create a playable but greenscreen media file after stitching, so that feature had to be omitted.  This feature may yield better results.

Pro Res 422 HQ, across the board, yielded slower encode times.  Avid DNx220 would be the Avid equivalent to ProRes 422 HQ (although, technically, it should be vice versa) and was always done quicker.  This is by no means a visual quality test, this was raw speed.

Although I cannot prove it (aside from my results here) it seems some encoders just “play well” with some codecs and data rates (i.e. high compression/low data rate DNx36 vs. lower compression / high data rate  DNx220 & ProRes 422 HQ).   This contrasts with the Episode Family, whose encode times were pretty similar across codecs.

All testing was done local (internal OS drive), as the differences in mass high speed storage varies from user to user and therefore difficult to baseline.  I define mass high speed storage as RAID sets with Firewire eSATA, Fibre, or SCSI connection.  While I expect times to be similar when Firewire/USB drives are used as the source drive (as most batch encoders write locally to a cache for processing, then write back out to the destination drive), I certainly expect encode times to decrease when mass high speed storage is used, as larger files require more time to write after the encode is done and the cache has to copy out to the destination drive.  I do not expect this to be drastic, but it may save a few minutes each hour.

I attribute the increased times with batch encoding with Compressor with VC to this.  (I know there is a Cluster option setting for this, however altering it seems to break the Virtual Cluster)  I could have decreased the encode time by up to 20% if the application did not have to write out locally, as the merging of the distributed quicktime segments took almost as long as the length of the clips (RT) themselves.

When batch encoding with Compressor, it’s important to remember that the application is splitting the transcode up to the available processors.  This is great if a batch of 2 clips are the same length, but if one clip is, lets say, 1 minute longer than the other, then the longer clip will no longer benefit from the distributed encoding when the first is finished.  Normally this is only an issue at the tail end of a batch encode  (as once one encode finished, another will start).  For long encodes, this bares mentioning.

Across the board, encode times are cut in 1/2 to 3/4 from the last gen of Mac Pro (Harpertown, 3.2Ghz 8 core)..making the new Mac Pro, in the RED realm, a great investment for high volume encoding.

It should also be noted that even though some of the more expensive encoders (Episode Family) are not the fastest, the increased encoding options and variables, codec support, templates, watch folders, and bells and whistles they contain may be worth the investment.
http://www.telestream.net/pdfs/datasheets/EpisodeSeries_Format_Support.pdf

As of this writing,Telestream’s Desktop Products: Episode & Episode Pro will run you $495 / $995, based on options and their Enterprise line Episode Engine & Episode Engine Pro runs $3950 / $8450.

Final Cut Studio 2 (with compressor) is $1299.

RedRushes is a free download from red.com.

Questions?  Comments?

I have only tried with DNxHD and on Unity (so everyone can see the same media), and verified all qt ref export settings with Avid…i.e. use network media resources, etc. I can recreate the issue as well.

My thoughts are that Avid Codecs changed in 3.5 from earlier revs (maybe for stereoscopic) and are not backwards compatible AND/OR the latest Avid downloadable codecs (1.10, from 1.9) do not have the same codecs as 3.5.

UPDATE #1: Examining the installer package from 3.5.1 shows the Avid PE and LE codec package as being 2.0.0, which is NOT available on Avids download site (avid.com/dnxhd).

SUMMARY: Avid apparently tweaked the codecs from 3.1.x to 3.5, and no one seems to have noticed yet. Theoretically, one should be able to take the codec installer (2.0.0) from the 3.5.1 editor install and use that on any system that needs to view media exported from a 3.5 editor machine. That does an end user no good who doesn’t have a copy of 3.5.1.

Update #2: The plot thickens. Avid has posted a link to the 2.0 codecs included within Media Composer 3.5.x (link here) however it is still *not* linked at the http://www.avid.com/dnxhd microsite. Bigger still is the fact that Avid has uncovered this is also a Unity issue. This has been escalated to top priority within Avid Engineering, and I’ve provided them with some sample footage. Until this is resolved, it appears QT Refs with 3.5.x are kaput. In addition both of the 3.5.1 releases (first rev of .1 AND 2nd rev of .1) do not address this issue.

Update #3: Avid Engineering informs me that revving to the as-yet-unsupported QT 7.6 should fix the issue, as wella s downloading the 2.0 codecs, both which appear to work.  Both the OS and my encoding solution(s) see the file and can play it / manipulate.

*whew*