Creating the 3D in Gravity
COW Library : TV & Movie Appreciation : Debra Kaufman : Creating the 3D in Gravity
Gravity – Official Main Trailer from Warner Bros.
Director Alfonso Cuarón has been at work on Gravity for five years. For almost as long, Prime Focus World has been working on the film's masterful 3D stereoscopic conversion. Whereas 3D can be an after-thought – quite literally when a classic movie is converted to 3D years after the fact for re-release – for Gravity, 3D was integral to the storytelling, nearly all of which took place floating in a very three-dimensional outer space.
Three years ago, says Prime Focus Senior Stereo Supervisor Richard Baker, Gravity Executive Producer Nikki Penny came knocking at the company's London facility. "She's a VFX producer in London so we knew her from previous projects," said Baker. "She approached us about doing a conversion test based on the native 3D test shoot they did that threw up quite a few issues for them."
The producers always knew Gravity would be a hybrid of CG and live action, but early on they hadn't yet decided whether the 3D live action would be shot in 3D or converted. The production had done a test shoot in 3D, which involved placing the cumbersome 3D rigs inside the cramped space capsule. When Penny brought the 3D live action footage to Prime Focus World, she asked that they test 3D conversion by utilizing a single eye from the same footage. The results convinced Cuarón and producer David Heyman that conversion would not only look as good as shooting in native 3D, and would be much easier and more efficient to accomplish.
Here is one final shot from the film. We'll take you through the conversion process of this frame in the article.
Prime Focus World ended up converting 27 minutes in Gravity: 85 shots including one shot that was comprised of 15,531 frames and lasted over 10 minutes 47 seconds. PFW's View-D Conversion & Production work was primarily done in London, with support from its Indian facility.
But, unlike the more typical post-production 3D conversion, Gravity involved PFW working on the 3D conversion during production. The work on 3D conversion began with six months of pre-production, during which the production's Stereo Supervisor Chris Parks worked closely with the PFW team to explain his carefully plotted depth map, which contrasted the vast emptiness of space with the tight, nearly claustrophobic quarters of the space capsules.
From the beginning, the challenge for Prime Focus World would be to make sure that the 3D converted footage integrated seamlessly with the extensive 3D CGI. Both the live action footage and CGI would be 3D, but 3D from two different worlds, conversion and visual effects – and they would both be active or "live" during the production process. At the same time, the PFW team wanted to retain the creative flexibility of its View-D conversion process, while integrating the stereo VFX universe. What made it trickier was that, unlike a completed film, Prime Focus would be working with work-in-progress shots and an open edit.
"Rather than convert at the end of production, we were running in parallel with Framestore, the main VFX provider," says Baker. "There were shots where Framestore required a converted plate to comp their stereo CG elements into the live action and shots provided by Framestore as 2D comps with VFX elements in. All of this required a level of production management not usually associated with a conversion show. We were integrated into the whole production and VFX process. We had many early meetings with Nikki Penny and VFX Supervisor Tim Webber to ascertain what assets were available, what assets we could share, and how we were going to approach certain technical challenges that we anticipated."
Prime Focus quickly got to work developing a pipeline and new tools that could meet the challenges. Key among the R&D was a mathematically accurate way for the conversion and VFX pipeline technology to interface. PFW Global Technical Supervisor Rajat Roy notes the difference between the 3D worlds of conversion and CGI. "An aesthetically pleasing conversion is not necessarily accurate," he says. "But that doesn't work in VFX terms where you have to be very accurate in terms of the distance of objects. To join these two worlds – one very linear and accurate and another one more flexible and aesthetic – meant we had to create a new algorithm. We had to calibrate both of them and work out a way for the computer to translate it."
"It was very unusual for everyone, and an entirely new way of working," says Baker. "Normally, we have a rotoscope-based pipeline, but for Gravity, we developed a pipeline with geometry as well as rotoscoping. Typically we'd get a scan from Framestore, something we'd normally rotoscope and then base our conversion on that. With this new pipeline, Framestore would start their VFX work on a shot and match-move it to get their tracking info and also have geometry for the space capsule. We'd create a very accurate depth map and combine that with the assets Framestore provided."
Eoin Greenham, PFW Global Head of View-D Pipeline, noted that PFW and Framestore had numerous meetings to come up with the best way to join their ecosystems. "We had to strive quite a lot to be able to describe the building blocks," he says. "We made plug-ins into our pipeline so we didn't have to change the two giant machines but keep everything moving and leave us the ability to swap information. Between the two of us, we've exchanging thousands and thousands of assets and all the tracking information."
Because the shots were live, Prime Focus frequently had to adjust the modeling. "On a live set, if you take a laser scan and create a CG model you can use, sets are there for days and days at a time, because shoot last for months," explains Roy. "Part of the sets move around, and then the LIDAR scan of the set no longer matches up. So the model never really completely matches. What we had to do is take the set model, add new bits of geometry and shift it around so the geometry actually matches the visual models in the shot."
"Basically, we designed our virtual stereo cameras to work in both our converted 3D world and the VFX environment," adds Roy. "This allowed Framestore to plug our 3D virtual cameras into their VFX universe, seamlessly integrating the two stereo approaches. Our conversion was in the middle of the VFX pipeline. Nothing was finished. We started work on something that was temporary and then they'd change it. Everything was live and we had to be very reactive."
Tracking the huge number of assets and versions was another challenge. "Any changes to a shot needed to be recognized and accounted for in the others' pipeline immediately," says Greenham. "Since the director wanted to keep decision-making open, that meant there could be many changes. On a whole, we were looking at a large amount of small alterations and adjustments. But even slight changes to a camera move, for example, resulted in massive amounts of new tracking information."
Part of the solution was to agree on naming conventions with Framestore. "There were also lots of scripts and plug-ins to talk between the different composting softwares," Greenham adds. "We use Nuke but also Fusion, Silhouette and other packages; Framestore uses Nuke." The assets also traveled geographically, to PFW's India facility, which handled a lot of the rotoscoping, and back to the London facility where the assets traveled to and from Framestore.
Although rotoscoping is a 2D process, it still played a role in Gravity's 3D version. "We were using it to define Z maps and disparity maps," Greenham says. "We'd make sure that all the shots were conforming to this space as a starting point. Then Richard and Chris were able to creatively move that into a space that was more aesthetically pleasing, with consistency and uniformity. If a shot had to be broken down into a team of artists, you absolutely do need everyone starting on this same visual page."
"This technical rigor helped to handle all these long seamless pieces," he says. "You don't see the cut points but there are quite a lot of them in these long complicated shots. We looked at points we could cut at depth, while Framestore was looking at where they could cut from a 2D image. They're staying in a 2D world trying to join 2D images together so it doesn't look jarring, so you don't see a pop. We did the same thing in 3D, trying to pick out subtle moments to make the cut."
Gravity was, without a doubt, the most detailed and sophisticated shoot Prime Focus World has completed to date, says Matthew Bristowe, Senior Vice President of Production/VFX & View-D in PFW's London office. "The length of the shots, the high level of detail required in the conversion, the long 3K sequences all made Gravity a game changer."
Anyone who's seen Gravity 3D will agree that the 3D was a game changer: a perfect meld of technique and content. The fact that 3D conversion was created in production, in conjunction with VFX, just hints at the possibilities of more innovative productions in the future. For the meantime, Gravity 3D is a great celebration of thinking outside the box.