Legend of the Guardians: The Owls of Ga'hoole
COW Library : Cinematography : Alex Timbs : Legend of the Guardians: The Owls of Ga'hoole
Editor's Note: "Happy Feet" was Australia's first digital animated feature, and won the 2006 Academy Award(R) for Best Animated Feature. At the helm of production and visual effects? Melbourne's Animal Logic. Founded in 1991, they have been part of some of the most groundbreaking, visually rich pictures and television of our time, including "The Matrix," "Babe," "Farscape," "Moulin Rouge!" "The Thin Red Line," "House of Flying Daggers," and many more.
Just a few months after "Happy Feet" was released came another film that Animal Logic worked on, the stunning "300," directed by Zack Snyder, whose work also includes "Watchmen." On one level, it might have seemed odd for someone whose last couple of features were rated R for graphic violence, sex and language to take on a movie based on a popular series of children's books, "The Guardians of Ga'hoole." On closer examination, it was a remarkable fit: stories of an alliance to protect a kingdom, a group of "Pure Ones" whose leader wears a mask to conceal his battle scars, kidnapping, slavery, and the rise to honor in the midst of exceptionally bloody battles.
Oh yeah, and the Guardians of Ga'Hoole are owls. Now you know at least some of attracted Zack Snyder to what turns out to be not exactly a children's movie, and some of why he naturally turned to Animal Logic again, this time to make an all-CGI feature with the intensity and visual impact he required, released as "Legends of the Guardian: The Owls of Ga'hoole."
When we had a chance to speak with Alex Timbs, Animal Logic's head of IT, we suspected that Zack had very strong feelings about how to execute a vision that was certainly rooted in the books, but every bit as certainly, a world with Zack's stamp on it. Because a project of this scale typically takes three years from start to finish, we also suspected that Zack began working with Animal Logic very early on. Here's some of what Alex told us about the project, starting, it turns out, before the beginning.
At Animal Logic, we come in very early in the process, even before a project gets greenlit, at the point the project is being quoted. We start by determining the cost to deliver a single seat -- the facilities to support that artist, the food, environmental control, desks, telephones, etc. There were about 400 people working at the peak of production for "Legends of the Guardian," with nearly a third of those people hired specifically for this project.
As things get greenlit, we begin working with productions to come up with very rough render and storage estimates, to at least get a stake in the ground. We then review what sort of infrastructure we have available at the time, a) whether you've got the capacity, b) whether you've got the processing power, and based on whatever you already have available, what you're going to need to add. Even whether you've got the right models or brands of products to address that specific production.
It turns out that there is significant infrastructure churn to keep running at the capacity and performance required to get the images out the door. Over the course of a three-year production, you're refreshing your entire render farm, or a large portion of it, and potentially replacing your entire storage after each project as you ramp up for the next one. Each project almost carries its own storage and render budget.
You know the old rule about doubling processing power every 18 months? We're going through a period where we've seen big improvements in individual machine processing power, much MORE than double every 18 months. Because of multi-core technology, and being able to pack more, faster cores in a given box, processing power is going up exponentially. Of course, we're seeing production requirements go up exponentially too, especially on "Legends of the Guardian." Even from very early on in the piece, we could see that we were going to have to advance technology trying to meet the demands of the graphics in some of these Zack Snyder signature pieces -- not only in effects rendering, but also just for editorial.
Director Zack Snyder, (photo by Simon Cardwell).
For example, he has these amazing speed-ups, where he might speed up the action 1000 times, or slow down 1000 times, while he's wrapping the camera around a character. That was putting unreasonable demands on the disk system that we had in place at that time -- which was reasonably new. We bought it for this production, a SAN designed to support up to 10 editorial seats at once, but we found that a single editor could cripple the entire network because of the enormous demands being placed on it. We used a combination of workflow changes and configuration optimization to the SAN for this particular problem. Things like, "Okay, if the director wants to see this shot a thousand times bigger, you guys are going to have to render it first before we can look at it," because they were trying to look at it immediately, and you can imagine what this did to the disk system. Rendering was constant of course, just about any codec you can think of, just about any resolution. We somewhat jokingly call ourselves the world's largest proxy farm! We generated 30 or 40 different codecs in many different sizes, depending on the part of the workflow we're talking about. In editorial, we were mostly working in 720p, but sizes might be as small as 320 for proxies in the asset management system, all the way up to 2K uncompressed DPX files.
The goal was to optimize data so that we weren't sending huge uncompressed data where it was not needed.
(L-R) Kludd, voiced by Ryan Kwanten and Nyra, voiced by Helen Mirren.
(L-R) Soren, voiced by Jim Sturgess and Gylfie, voiced by Emily Barclay.
"Happy Feet" was our previous baseline as a company for render requirements: we used about 4,000 cores at the peak.For Guardians, at the peak it was 11,336 cores! That should give you some indication of the changing requirements. And we're talking about cores with many, many times the memory access speed.
We saw the same kind of growth with blades. For "Happy Feet," we used 2-4 gigs in each blade, 4 gigs at the peak. This time, the blades had 24 gigs of RAM that was many, many times faster than the previous blade tech.
We used a range of HP blades, starting with the then-current, now older, G5 infrastructure, with 1152 ProLiant BL2x220c double dense blades. During production, we added 256 of the newer G6 blade, which uses the Nehalem architecture. And for some very specific water and effects simulation work, one ProLiant BL685 G6 Server, a 24 core blade with 96 gigs of memory -- that is, 24 cores, each with 96 gigs of memory -- and two HP DL385 standalone servers, again, 24 cores, each with 96 gigs of memory. We probably could have used a lot more memory than that.
A small portion of the data center used by Animal Logic for "Legend of the Guardians: The Owls of Ga'hoole." At its peak, the production had over 400 TB of storage online at any time, with over 11,000 cores handling the processing to add over 24 gigs of new footage every week.
"LEGEND OF THE GUARDIANS: THE OWLS OF GA'HOOLE"
REVIEW AND APPROVALWe were quite proud of the review system we were able to set up. Zack was at his base in the States for part of the production, so we set something up for him at the Warner Bros lot. [Ed. note: Along with Village Roadshow Pictures, Warner Bros. Pictures was the film's distributor.] Zack was also in Vancouver as he was shooting "Sucker Punch."
We have an in-house proprietary software management system, and one component of that is a delivery system that is triggered when someone checks in an asset. We set up a permanent VPN link which automatically trickled over any of the information that was queued for delivery to him. The data would arrive early afternoon his time, and while still quite early in the morning here in Australia, all of that material was online and ready to go.
For the US and Canadian locations, we had shipped over a number of Final Cut Pro seats and local storage, and some pre-calibrated HP DreamColor monitors. We also set up rooms with color-sensitive lighting so that we knew we were looking at the same thing. An interactive system allowed us to work through individual frames together. On top of that, we had an extensive HD videoconferencing set up at both locations. Zack is very expressive and passionate, so there's a lot of value to be gotten from his body language. We could have gotten enough from hearing him, but there was so much more that we were able to get because we could see him in HD.
Soren, as voiced by Jim Sturgess.
INFRASTRUCTURE CHURNYou can be more dynamic in the early stages of production than when things get hot and the pipeline is busy, so a lot of the changes that we have to make are done either before or in the early stages of production. That requires some strategic decisions around networking in particular, and a commitment to a flexible networking core. We started with 10-gig throughout, from storage to the render farm to core services. And with storage, we look for something that was scalable in a very granular fashion, because no matter how good at predicting we are, the production goes at its own rate. Sometimes it goes more quickly than expected, earlier than you had budgeted for, so you don't want to spend a whole heap of capital at the start of the project, especially on storage or render capacity. You know you're always going to need more, and you want the best that you can get at each step in the project's growth.
Just to give you a picture: at the peak of production, we had around 400 terabytes online at any given time, generating around 24 terabytes of unique data per week.
By looking at how we can easily integrate new blocks of storage, or more rendering, in very small increments as production grows, we can manage budgets more carefully. There's so much of it that's pretty well fixed early on in the budgeting process -- cooling, and desks, and networking -- so we need to be able to opt to refresh storage and rendering at just the right time. The key is in those choices you make before you even begin, to have all the connectivity we need for what we can see immediately, and for what we might need beyond that in the future.