Cheap software that promises the world is nice. Software like Final Cut Pro has made our business possible. We are proud to have been an important part of posting the first major feature film ("Cold Mountain") and primetime network series ("Scrubs") to use FCP.
But as you start to move higher up the production food chain, things expand really fast - even if the building blocks are lots of cheap workstations, storage, and IO. It becomes apparent very quickly that no real production facility, no matter how much it leverages the current crop of great software, can survive without serious attention to hardware.
When you hear the word "facility," you might think that we are bigger than we are, especially when you hear that we are currently posting three TV series, as well as a number of shorts, at the same time. There are in fact nine of us.
Yet, we have been able to triple our workload in the last two years, keeping roughly the same (relatively) sane hours - thanks to carefully selecting hardware to help with the heavy lifting.
DESIGNING A FACILITY
In 2007, we designed a new facility to meet the opportunities we saw coming. We had already been working with Sony HDCAM SR and the Genesis Viper, and saw the potential of other emerging, high data rate cameras, so we made a simple assumption: 4K.
The 4K data rate is insanely high (1.35GB of data per second, over 80GB/min.) Playback of such high res data is fraught with peril, even with local storage. I will go into more detail about this later, but shared storage was a critical part of our vision - and sharing 4K is impossible with less than a fibre network. Our entire facility is wired with fibre for this purpose, with switches capable of scaling from 2Gb all the way up to 8Gb data rates.
Rather than a bunch of suites, we wanted to design an editing facility. All of our Mac Pros are fully tricked out with Final Cut Pro or Avid Media Composer - and all of them are stored in a central machine room. All we have in each edit room is monitors, keyboards and mice, with a single Cat-6 Ethernet cable linking each computer via a KVM (keyboard, video, mouse) network.
KVM products are simple and inexpensive, and work great with even just a couple of computers. The KVM system has created a huge ripple effect for us. Any computer can be called up from any bay with a couple of keystrokes. No one bay is more powerful or more sought after than any other.
We also now have the ability to work on multiple edit stations at the same time: Our editors or post operators often click to one computer, start captures, switch to another, start renders on that one, switch to a third one and continue to QC the show, while the previous two computers capture or render away.
When our operators need to leave one edit location, and say, move to the machine room to lay off the program, they can call up the computer they had just been working on from the new location.
And if we need another bay, all we need to do is connect an Ethernet cable, a keyboard, a mouse and a monitor. We could add one in a closet if we needed to.
A FINISHING HOUSE
When Final Cut Pro and its related hardware allowed anyone to create "online" quality video, being an "online facility" became meaningless. Instead, we positioned ourselves as a "finishing house," handling all of the broadcast and technical requirements needed to finalize and deliver a show.
While it is true that inexpensive cameras and laptops can do great things, they are not suited to heavy lifting for delivering air masters for broadcast. The broadcast specs from PBS, for example, run over 80 pages. There are details about which line should the ancillary data be encoded to, the precise specs for Closed Captioning, and much more.
We set up a facility-wide Tektronix-based scope system that allows us to output and monitor broadcast quality color and luminance. The same system also performs automated quality control (QC) analysis of all outputs, and creates detailed logs of any errors. We also created a sophisticated hardware and workflow system to output numerous deliverables for our clients, each of them properly captioned.
Which brings us to the key ingredient for a successful tapeless workflow:
Shooting and post workflows are becoming increasingly tapeless, but a tremendous amount of work is still being shot on tape.
We found this ourselves as we worked on the film, "The Forbidden Kingdom," starring Jet Li and Jackie Chan. The production spent a great deal of time on location in China, far from any cities. They were way out in the jungle, and in mountains and deserts. Even the primary soundstage was in a rural village, a five-hour drive from Beijing.
We used the Panavision Genesis, which has proven to be an extremely reliable and rugged camera. Even though integrated solid state storage is available for it, the production relied on tape. Computer problems are far more likely than deck problems, and if something had broken down this far away from anywhere, the production would have been completely screwed.
Shooting to tape was also more efficient: no need to offload data before the next round of shooting, no concerns for storage other than a box to keep the tapes in.
The format of choice for digital cameras recording to tape is Sony HDCAM SR, one of the few formats that can handle the high data rates associated with 4:4:4 RGB recording.
Even if recording is to disk or solid state media, especially when you get to broadcast, delivery is exclusively on tape. Nobody that we work with in either Los Angeles or New York is going to air from digital masters on disk. Broadcast facilities simply can't reconfigure to disk-based infrastructures overnight - and from what we have seen so far with the ones who are starting the transitions, the last part of the system to change will be playout to air.
This is because every part of the process leading to air is tape-based, starting with QC systems. These are automated so that nobody has to watch every second to ensure standards compliance, and they are built for tape.
Our workhorse tape formats are Sony HDCAM SR and Panasonic D5. We get plenty of both, on a regular basis.
We also have just about every other format represented, including HDCAM, HDV, DV, DVCAM, Digital Betacam, Betacam SP, all the way down to ¾" U-matic (You'd be surprised how much work those ¾" U-matic decks get to this very day.)
Not that digital workflows aren't increasing. ABC has added a requirement for IMX dailies, MXF files which they use for their in-house asset database. Any of their executives can log on with a web browser and review the production in process.
The idea is that this prejudice against tape in this "modern, digital age" may be quite misguided.
One of the best uses of tape is actually for backing up data recorded on drives and solid state! To this very day I am hard pressed to find a more efficient, lowercost way of creating a digital backup for a one hour show, than by simply outputting it via a digital path to a robust digital tape format. Blank tape is ridiculously cheap, the "backup" happens in real time, and tape is a proven and reliable way of storing media.
Backing up digital material to robust tape formats is actually the recommendation of the Science and Technology committee of the Academy of Motion Picture Arts and Sciences, as presented in their report, "The Digital Dilemma."
While the paper notes that film has the longest reliability by far, the research they present convincingly points to a much longer life cycle for tape than for computer media.
In any case, even when shooting RAW files to disk, and an entirely tapeless post workflow, everything for broadcast ends up on tape, for both master layoff and air master delivery.
Even though we have always been known for our work with Apple software and hardware, we were not willing to let platform allegiance stand in the way of our ability to succeed.
That did not just mean adding Windows to our environment. That meant adding everything: Linux, IRIX, the whole shebang. Each of them runs unique applications that are essential to our business.
We knew that we needed to run all of these applications, and work with all of the media associated with them, on a single, shared-storage network, also known as a SAN.
We had been SAN-savvy from the very get-go, which now seems another lifetime ago. Our network design for "Cold Mountain" for example, was based on OS 9.
Because we had been working with Apple for so long, we looked closely at Apple Xsan. We learned that the foundation code for Xsan is licensed from a product called StorNext, now part of Quantum. Xsan was simply a subset of StorNext, with a sexy Apple GUI (Graphic User Interface) on top.
One of the advantages that we found in going directly to StorNext was the ability to create a single network shared by all of our platforms.
I have already mentioned our Video Village, and the tape formats we work with. Here is how the rest of our facility lays out:
Xsan Metadata Controllers require their own, dedicated Ethernet network to send all of their commands, while the fibre pipeline is devoted to data transfer only. We also have a backup server so that we can provide uninterrupted service if the other fails - a key idea in any SAN environment.
We've amassed a few of the 128 port, 4Gbps Qlogic 9000 Fibre switches, where all of the fibre connections reside. We have found the Qlogic switches to be one of those miracle pieces of electronics that just keep on ticking. Not a single issue we've ever had to resolve in our SAN fabric has ever been traced to the fibre switch. The Qlogic switch is a perfect piece of machinery, if there is such a thing.
Our Xsan storage is striped as RAID 5 for maximum redundancy and protection. If a drive module fails (and once in a while it does), we can replace the module and rebuild the RAID, without any loss of data or time.
We have more storage based on the HP Storage- Works Scalable File Share for Linux.
Our combined storage space varies, and with typically anywhere from 50 TB to 100 TB on line. It can easily go higher.
Mac OS X is one of the many platforms we run in our facility. We have a boatload of Mac Pros, all running Final Cut Pro and using either AJA Kona 3 or Blackmagic Design Decklink video cards for capture and output. Other key pieces of software we run on OS X are Apple Color and Motion, Adobe After Effects, Maya, Avid Media Composer, and many others.
We use Windows XP-based render nodes, and Linux-based tracking software.
We have several SGI CPUs running the IRIX platform, for running our Autodesk Flint and Flame applications.
Many people thought that After Effects would kill off hardware-based solutions such as Flint and Flame. Almost all daily effects work today at DigitalFilm Tree involves AE. However, when you need heavy lifting, there is no substitute for heavy iron.
For example, we did almost all effects work in Flint and Flame for the film "Crank," starring Jason Statham. There is simply no substitute for the quality and speed those hardware-based solutions offer. They were also handling 2K and 4K files long before apps like AE and FCP could come close. When it comes to high-end special effects work done under the gun, very little can match them.
Before we were even halfway through our facility upgrade, we started to realize the fruits of our labor.
Our editorial, effects, color and titling could all proceed simultaneously, as opposed to in a linear order. The colorist could start roughing out the show before the editors or the effects artists were through. Editors could make changes and the rest of the staff working on the show could catch up without any lengthy emails or details going around. We realized huge savings in storage space, because we were not doubling up media.
Right away, we were able to take on a second TV show without any additional staff. And then a third one, with the same number of people. We let the hardware take care of the heavy lifting.
One of the very first shows to take advantage of this with us was "Everybody Hates Chris." One of their producers, Erik Whitmeyer, has said that their show realized a savings of $11,000 per week, due to this new way of working. He also noted that other benefits included working sane hours and less time spent making changes.
Now underway: work on three series ("Cougartown," "The Sarah Silverman Show," and "NCIS Los Angeles"), while also working on other projects, including coloring a film for Apple, featuring Walter Murch's digital workflows on the set of Francis Ford Coppola's next picture.
HOW TO BEGIN
Buying all of this was expensive - but we didn't buy it all at once and sit in our shop waiting for work to materialize. When we began, we survived by being an Apple Authorized Training Center. We trained people to edit in FCP and FilmLogic by day. At night, when there were no classes, we used the training computers to do our work.
Then we bought our first DV deck. That was a monumental event!
We got friendly with ebay and very slowly bought the basics. We scoured the internet for post places going out of business and bought wiring and other essentials. Sometimes we found those essentials in a large cardboard box that they had put in a dumpster.
It is so very important to say that we were not rich, nor did we have any bank loans or investors. We survived by paying ourselves next to nothing, and putting the money back into our company. Some of us slept in the office to save on rent.
Once in a while, DigitalFilm Tree founder Ramy Katrib's Mom or brother would drop by with a basket of food that would last us a few weeks. It was about as home-spun an operation as one can imagine.
We definitely never overstepped our bounds. We never purchased a piece of equipment until we had the job firmly in hand, often with some of the money paid up front, so that we could run out and buy the equipment to help us do the job. We were very open and transparent with our clients about this.
As you consider getting started in your own business, and then prepare to grow, watch and control your payroll. That is key. By the time health insurance and all the Federal and State requirements are satisfied, payroll expenses become massive. Measure the increase in business that you may gain from adding another member to the team, against how much it will cost to pay them.
We do not have any specific secrets to share beyond these, Our view is that, in times like these, you just have to try harder. It takes more effort to find and get the work. But it can be done. TV shows are still airing, movies are still being made, and commercials are still being played. All of it needs posting.
Even if everybody needs to take a pay cut, now is not the time to stop investing in your future. While we certainly wish everyone success, we have seen in the past that failure in one place can lead to opportunity in another.
WHAT COMES NEXT
I do not mean to suggest that big investments are the only ingredient for big success. We have also built our business on embracing disruptive, low-cost technologies.
We see the emergence of these technologies going through the five phases of grief: denial, anger, argument, depression and acceptance.
First, denial. "This can't be happening!" Anger comes next: "This is hurting my business and my way of life."
Then comes the argument phase, where people turn academic on you. "Well, I offer THIS, and you only offer THAT."
Next, depression, often accompanied by frustration and bitterness. And finally, for the survivors, comes acceptance.
For example, even though we have been working successfully with it in mainstream, large-scale Hollywood productions since 2002, we think that FCP is finally arriving at the acceptance stage.
Over and over again, we have see this cycle repeat. What is interesting is just how Darwinian the marketplace can be. If it works for people, they will go for it. No amount of Jesuit distinctions or "?Good enough' is not good enough" logic can stop the marketplace from adopting what works.
The first step is to watch for coming changes. Instead of fighting to hold them off, embrace them quickly. After you have settled into the mindset of adaptability, the next step is to wisely invest in the hardware that will ensure your differentiation from the cheapest options, and ultimately, your success.
One thing is for sure: the future of post technology has never been brighter. There have been hard times before, and there will be hard times again. Business opportunity abounds for those who look forward.
Find more great Creative COW Magazine articles by signing up for the complementary Creative COW Magazine.