LIBRARY: Tutorials Reviews Interviews Editorials Features Business Authors RSS Feed

Freeing Artistic Vision from 3D's Limitations

COW Library : Stereoscopic 3D : Neil Feldman : Freeing Artistic Vision from 3D's Limitations
From The Creative COW Magazine


Creative COW Magazine presents - Freeing Artistic Vision from 3D's Limitations


Neil FeldmanNeil Feldman
Westlake Village, California, USA

©2009 Neil Feldman and CreativeCOW.net. All rights reserved.


Article Focus:

In-Three's Stereoscopic 3D Solution begins with Single-Camera Footage.




In-Three originally developed its technology to revivify dormant film libraries. We call our process "Dimensionalization®," using patented software tools and techniques to create a second view from any two dimensional image.

With delays in the rollout of 3D theater screens and platforms for 3D video in the home, we have had only modest opportunities with legacy films. However, as our R&D has continued, we have seen a strategic opportunity to solve problems in the creation of live action 3D features. We believe that these problems, particularly around the hassle of using two cameras to get a 3D picture, are among the reasons why so few 3D features have been released.

Let me preface this by saying that I don't want to be perceived as dumping on shooting. Shooting has its natural domain, which includes anything related to broadcast - ballgames, operas, and other live events that are already being presented in 3D in theaters, to great success. That's natural. Those kinds of presentations don't need Dimensionalizing, and they don't need CG - which also has its natural domain of course.

To begin with the difficulties around multiple cameras for 3D production, one is lens differential. There are inevitably subtle physical differences in things like lens refraction. There are differences in perspectives for things like specular highlights and reflections that may be fully visible in one eye, and not present at all in the other.

Each of these is exaggerated by vertical disparities in the lenses. Our eyes obviously handle horizontal disparities just fine - this is how we see in 3D as we look at the world, by resolving horizontal differences in perspective. But resolving vertical disparities would require one eye going up as the other is going down!

Even if each of these errors is small, they accumulate over the length of a movie, and become disturbing. Your brain is telling your eyes to adjust to something that they can't quite sync. It wears you down.

The challenges become even greater when shots have to match. If two shots inside a room are converged on a different point in space, the room can look like it is a different size in each shot! It is shallow in one shot, deep in the other. Even if your eyes can figure out what is going on, it's not very believable, and it's certainly not very comfortable

There are some powerful warp engines and other sophisticated tools to make things a little better, but really, there is no way to fix them. Once you take the pictures, these things are locked in.


Dimensionalization
Footage shown above is undergoing Dimensionalization and is a scene from James Cameron's Titanic.


DIMENSIONALIZATION

We start with the footage from a single camera. We take that 2D footage, and we copy it to create a right eye, left eye view. Every pixel is mapped perfectly, with no discrepancies from lens effects, no misalignment. Now we can begin to add perspective, shape and depth.

When I say add depth, I mean to move one eye's view relative to the other eye's view. Each eye tracks independently.

We give shape first by rotoscoping, then by morphing. As we extract depth, we expose an area in the background footage that was previously occluded, and have to recreate a background that does not exist in the footage. We also create a kind of binocular disparity that your brain interprets as shape. But no vertical disparities, no light balance disparities, no pixel disparities. You eyes are tracking the way that they do in nature.

If we do what I just described well, we can achieve what I call "perfect 3D."

We are also not just limited to a single set of 3D characteristics for a given shot. Our system can isolate 20 elements in a scene, each with its own camera, so to speak. Which means that, for each of these individual elements, we have a unique set of those three controls - perspective, shape and depth. Instead of those three "knobs" on a single camera to cover an entire scene, we have 60 "knobs" to turn: perspective, shape and depth for each of 20 objects that we have isolated.

Our original software was called In3 Depth Builder, and we're just starting to deploy a new package called In3gue - pronounced "Intrigue." This allows artists to keyframe the individual elements that they have identified to have unique depth, shape and perspective. The software propagates those values, interpolates through the shots, makes it all practical, and our software spits out an occlusion map. Each of these elements can now be treated individually, apart from the shot as a whole.

Now, let's say we go to the next shot, and it's a different view. We can isolate elements that we DON'T want to change, so that the internal depth of a scene stays consistent from shot to shot. Think of it as "depth grading," analogous to color grading, in order to maintain depth continuity.

That is, even if the dailies look perfect by themselves, here come another 200 scenes to cut together. In one of them, the focal element might come halfway out of the screen, and in another only 10%. Your eyes starting ping-ponging back and forth, and in and out, as you're trying to follow a single object from one cut to the next.

The problem for you as a viewer is that it drives you crazy to watch. The problem for the director is that, once the footage is in the camera, those disparities are locked in. It's hard to get immediate feedback during shooting, and it's virtually impossible to match 3D composition across shots in the edit.

We solve that problem. We make sure that the depth grades smoothly from one scene to the next. Directors see this and are amazed. They tell us, "This is the first time I've been able to have this kind of control over depth. I know what I'm getting." As we work, all changes update in the blink of an eye. I have signs all over our facility that say, "Blink Time." Our engineers are driven to keep things moving just that quickly in depth grading sessions.

SEEING IS BELIEVING
[Ed. note: The following are quotes from a panel held at ShoWest in 2005, introducing In-Three's process to 2000 cinema exhibitors.]

George LucasGeorge Lucas
"When I first saw In-Three's Dimensionalization process I was truly amazed. The 3D was of a quality better than anything I had previously experiencedů.When you see some of this test footage, it's shockingly good, and you can see how people would want to go see it. It means we can repurpose a lot of old movies, and at the same time it really gives a whole new dimension to the movies we're making now."

James CameronJames Cameron
"Though I still love 3D original photography, the technical solution provided by In-Three was a welcome addition to our palette of stereo film-making tools [for "Aliens of The Deep"]. They were able to add real depth to the 2D video images captured by our robotic vehicles, and these shots blended beautifully with our 3D shots. I predict that their innovative techniques will continue to expand the possibilities for 3D content."

Randal Kleiser (director, "Grease")
"After shooting in 70mm for Disneyland's "Honey, I Shrunk the Audience," I know how complicated this format can be. With In-Three's technique, completely convincing depth is created from any two-dimensional film. George Lucas showed me In-Three's work at I.L.M. Seeing Star Wars in 3D was amazing enough, but when I saw John Travolta singing "Greased Lightning" in 3D, I was blown away. It was just like being back on the set. I know every fan in the world would come back to the theater again to see a complete 3D version. The possibilities are mind-boggling."



DIMENSIONALIZATION WORKFLOW

The specific placement of Dimensionalization in the production pipeline depends on our client and their schedule. We generally receive the .dpx image sequences in log space before they have been color graded, but on a single project we can work with files both before and after grading.

We prefer to work on locked shots with handles, but we have often had to work as the cut, and even the final image, is in flux.

We recently worked on the Jerry Bruckheimer film, "GForce," directed by Hoyt Yeatman, distributed by Disney. Our stereo work fell into three major categories: full live-action shots to be Dimensionalized, full CG shots to be rendered in stereo, and live-action shots with CG elements.

The full live-action shots were Dimensionalized by In- Three. The full CG shots were rendered completely by Sony Pictures Imageworks. Often, the third type of shot was handled through a combined effort by In- Three and Sony Imageworks.

Dimensionalization is ideal for this, because we store depth metadata and can use it to conform stereo CG with stereo live action in real-time.

For these live-action shots with 3D elements, Sony would create a temp stereoscopic render of the CG elements, and we would Dimensionalize the live-action plate to match. The reverse can also work, but this approach was the one predominantly used in "G-Force." Sony would then do a final render of the CG elements. Then, depending on the specific shot, either In-Three or Sony would final the composition, and touch up the shot as necessary.

In other words, we don't necessarily see Dimensionalization solely as an alternative to live action shooting or CG. As "G-Force" has shown, it can be an excellent complement to them.


DIMENSIONALIZATION DESIGN DETAILED
Every artist station is equipped for real-time Depth Grading with two 120hz 3D enabled Samsung LCDs (SyncMaster 2233RZ) or Samsung CRT monitors (SyncMaster 1100DF). Additionally, we have several artist QC stations with 60" 3D enabled Samsung DLP TVs.

In our screening rooms, we use an NEC NC800c 2K Digital Cinema projector with the XpanD AE110 infrared emitter, projected on a matte white screen.

In addition to using our own proprietary software for screening playback, we also use Iridas FrameCycler DI for EDL conforms and long form QC screening. The screening stations use an appropriate stereo-capable video card with SDI out, dual Intel Quad Core Processors, 8 GB DDR-S RAM, and an Adaptec 5085 SAS controller managing two external RAID 0 disk arrays at 16 TBs each, for real-time 2K playback.

Each artist is using wired MacNaughton shutter glasses with stereo driver (HSG02-dg1), Real D Crystal Eyes with Desktop emitter, or NVIDIA GeForce 3D Vision wireless glasses at their workstations. At our QC stations and in our screening rooms we use XpanD X101 Cinema Active glasses.



THE FUTURE OF 3D

There are a lot of smart people in the world, and a lot of them are working on shooting stereo. As a result, I think that we will get past these problems. In the short term, I'm hoping we can sort the problems out well enough to see some really, really good sports events, because that's what's going to drive 3D television in the home - and THAT's what's going to make 3D become the financial blockbuster we know it can be.

While I'm rooting for stereo shooting, I also believe that directors are always going to want to have the most possible control over their shots. I think that we as a company, and as a representative of conversion technologies, are going to be a key factor in day and date releases of live-action features going forward.

When all of that happens, people will truly get caught up in 3D as a new entertainment vehicle. I think that that is when Dimensionalizing legacy movies is going to bring new financial life to film libraries.

In the meantime, our goal is to continue to maintain the quality of Dimensionalization, while driving down costs through technological innovation.

Whether through shooting, CG, or post-production Dimensionalizing, we can all participate in constructing 3D content. By understanding what our strengths and weaknesses are, we can construct 3D content together, and work together to achieve the highest quality at the lowest cost, and the fastest pace.


Neil FeldmanNeil Feldman
Westlake Village, California USA

Neil Feldman is CEO and owner of In-Three. He is also the owner of Dallas-based Video Post and Transfer. Beginning in early 2000, he saw the potential for digital image-based 3D conversion technologies. Additionally, Video Post and Transfer is the leading film and video post production firm in the Southwest USA, and has built a reputation for innovation: the first independent post-production facility to offer 1" Type C editing, to build all digital on-line editing bays, and the first in the country to provide unified digital audio and video editing suites.


Find more great Creative COW Magazine articles by signing up for the complimentary Creative COW Magazine.
Comments

Cool!
by Chris Heuer
Hello! I wrote an article for the Stereoscopic issue of the Cow Magazine. I converted the film Nosferatu to 3D using a slightly cruder version of the same technique (I used Mocha for Roto and After Effects for the composite).

Your company has dug much deeper into the science behind the art. Nice work! I'd love to hear more detail about your technique. How automated is the element extraction (roto)? How do you create the final clean BG plate? Will your software eventually be available for purchase?

If you haven't read it, My article is here:

http://magazine.creativecow.net/article/creative-cow-magazine-nosferatu-bec...

I'd love to hear your thoughts. Let me know when you plan an East Coast operation!! ;-)

Chris Heuer
good read
by shonnan wibrow
Thanks for sharing, I was wondering how labouring it must be to hand roto each shots to extract depth. How precise would these rotos have to be ? very articulate/accurate or could you get away with more loosely rotos... ?
Also what program does In-Three use ?

cheers


Related Articles / Tutorials:
Stereoscopic 3D
NVIDIA's GPU Technology Conference M & E Roundup

NVIDIA's GPU Technology Conference M & E Roundup

There was truly something for everyone at NVIDIA GTC, and a surprisingly rich amount of thoughtful presentations for the Media & Entertainment crowd. GTC provided many opportunities to learn the basics and, for the truly nerdy, many opportunities to dig deep. As a mere semi-geek, Debra Kaufman learned a lot and met a lot of interesting people. Read her roundup of new M&E technology thanks to super-fast GPU computing.

Feature, People / Interview
Debra Kaufman
Stereoscopic 3D
Tim Squyres Edits Life of Pi

Tim Squyres Edits Life of Pi

Tim Squyres, who was nominated for an Academy Award for his work on Life of Pi, got hooked on film when he took an introductory film course at Cornell University in upstate New York. Squyres talks to Creative COW about the challenges of editing Ang Lee's first digitally shot feature film, which was also a stereoscopic 3D release.

Feature, People / Interview
Tim Squyres
Stereoscopic 3D
Debra Kaufman's Review of The Hobbit in 48 fps 3D

Debra Kaufman's Review of The Hobbit in 48 fps 3D

Debra Kaufman started her high school's Tolkien Club when she was a big fan of The Hobbit and Lord of the Rings. Though she hasn't celebrated Frodo's birthday in many years, she did get out to see The Hobbit in 48 fps. Here are her thoughts on The Hobbit in 48 fps and HFR Cinema in general.

Editorial
Debra Kaufman
Stereoscopic 3D
Setting Standards for High Frame Rate Digital Cinema PART 2

Setting Standards for High Frame Rate Digital Cinema PART 2

As part of Creative COW's ongoing look at high frame rate cinema production and exhibition, we spoke with Michael Karagosian, co-chair of SMPTE's HFR Study Group, about his thoughts on the ways that higher frame rates factor into mastering, distribution, and exhibition for cinema.

Editorial, Feature, People / Interview
Michael Karagosian
Stereoscopic 3D
The Making of Flight of the Butterflies in 3D on IMAX

The Making of Flight of the Butterflies in 3D on IMAX

SK Films has released Flight of the Butterflies in 3D, an IMAX/giant screen film that details the true story of the scientist who spent 40 years researching the incredible epic migration of the monarch butterfly. Using, among other technologies, a specialized 3D snorkel camera designed by Peter Parks for close-up photography, the feature film features spectacular real footage of the brilliant monarch butterfly sanctuaries.

Editorial, Feature
Debra Kaufman
Stereoscopic 3D
Stereo D Converts Abraham Lincoln: Vampire Hunter to 3D

Stereo D Converts Abraham Lincoln: Vampire Hunter to 3D

If you think that converting a motion picture from 2D to 3D is a post production process, think again. Stereo D just wrapped up the conversion of director Timur Bekmambetov's Abraham Lincoln: Vampire Hunter from 2D to 3D, and the job began before a single frame was shot.

Feature
Debra Kaufman
Stereoscopic 3D
3net's Best Practices for 3D TV

3net's Best Practices for 3D TV

With a clatter of possibilities for 3D production, 3net has issued a guide - sort of a lion's roar - to help TV producers and creatives learn how to be confident in their knowledge, capable of finding the additional information they need, and resourceful enough to make killer 3D TV programs on budget and on time, all while dealing specifically with all the technical issues required to produce a successful 3D stereoscopic TV show.

Feature
Debra Kaufman
Stereoscopic 3D
Fields of Valor: the Civil War in 3D

Fields of Valor: the Civil War in 3D

Inspired by the massive number of historic stereoscopic still photos from the Civil War, 3net has ventured to deliver the four-part miniseries, "Fields of Valor" in 3D, dubbed "the most ambitious 3D project ever produced for television." 3net will air a special encore presentation on Saturday, March 24, 2012.

Feature
Debra Kaufman
Stereoscopic 3D
The Dog Days of 3D

The Dog Days of 3D

We've seen complicated rigs, higher budgets and audience skepticism deflate the hype of new movie-making technology before. What can the past tell us about 3D's future?

Editorial, People / Interview
Angela Gyetvan
Stereoscopic 3D
LG Debuts 3D Ultra Definition TV

LG Debuts 3D Ultra Definition TV

LG's internet connected super high resolution 84 inch Ultra Definition 3D TV will offer resolutions of four times that of existing HD panels. The "world's largest 3D Ultra Definition TV" has 8 million pixels for a resolution of 3840 x 2160.

Feature
Debra Kaufman
MORE
© 2016 CreativeCOW.net All Rights Reserved
[TOP]