Perception and The Art of 3D Storytelling
CreativeCOW presents Perception and The Art of 3D Storytelling -- Stereoscopic 3D Tutorial

By working in the area between what you see, and what you can see, Brian Gardner is helping re-write the rules of 3D filmmaking - right before your very eyes. In this article, he tells how some of these new rules played out in his work on the amazing movie, Coraline.





Our human perceptual systems have been driving changes in our entertainment media, one sensor at a time.

Since we are built to see motion, pictures naturally evolved into motion pictures. Motion pictures evolved to incorporate sound, color, 3D sound, and 3D vision. We use all of these, all day long. There's no reason we would want to give any of them up when we watch a movie.

When sound first came out, a lot of people complained that it was a gimmick. "So, what're you gonna do, have every movie be a musical now?" They didn't think about the emotional impact of hearing just regular dialogue's nuances, nor of a full soundtrack. They were so used to working another way that it never occurred to them to ask, what becomes possible now that wasn't possible before?

That's where we are with 3D. It has been treated like a gimmick, but we are starting to think about how it can be used as a creative tool.

My own goal isn't to make 3D "real." I don't want to make it a simulation. I treat 3D the same way that people use a soundtrack. I use 3D to make the romantic moments more romantic, scary moments more scary. I use 3D to create an emotional undercurrent.

I start by making a huge graph, the depth score of the movie, to show where every major element is in space, relative to each other, and to the screen.

I also make notes about the size of the space. For example, when the 3D goes deep behind the screen, you get this large empty space, this feeling of the grandeur of God, the vastness of the possibilities. I generally like to put that right at the act one climax, as the character goes off to explore the new world, to convey that sense of adventure. That's an example of what I mean by using 3D depth to underscore emotional dynamics.

Depth Scoring


DEPTH SCORING, part 1.

(Click image above for larger.)
The thick Black horizontal line represents the cinema's movie screen. Everything above that line will be in front of the screen, toward the audience. Everything under that Black line will be further away, behind the movie screen.

The thin vertical dotted lines represent the cuts in the sequence, from one shot to another.

The solid Red line is the location in depth of the Point Of Attention - what the audience is looking at. Usually, the POA is the main character in the shot.

The dashed Orange line is whichever object is nearest to the audience. It could be a character, a prop, or a piece of scenery.

The dark solid Blue line reprents the object furthest away from the audience. Often this is a mountain, some trees, or the far wall of a room.

The dashed sky blue line is the sky (if visible). Since this is an outdoor sequence, the sky is in this score.

The amount of space between the two dashed lines tells you the total amount of depth in the scene. Their placement up/down on the score tells you the placement of the scene in depth, relative to the movie screen.

At a glance, you can see that the sequence starts rather calmly (the smooth parallel lines of the left side of the score), and gets dramatically more exciting as it goes oninto the climax of the scene - the jaggiest lines to the right side of the score.

The lines vertically squeeze together and spread apart, as the character's emotional state swings, driving his failures and successes at each moment of the scene's conflict.

The main turning point in the character's emotional crisis is the sharp peak at the middle of the score. This is a gripping moment for the audience.

You can see that I've staged the scene's depth in four movements, with markedly different line qualities: curvature, separation, beats, and parallelism.

This divides the scene into four distinct emotional stages, which I want the audience to experience during these story beats.

Basically, I try to artistically direct the 3D to take the audience on an emotional ride, so that they will feel the story more intensely.

Isn't that what movie making is all about?




ALL THE CRAYONS

When color came into movies, people started with technical questions - how do we chemically develop the film? But they also had to ask questions like, what does it mean to the story to choose a red dress rather than a yellow dress?

That was exactly the pattern when I started working on "Coraline." I began with the technical side. I showed the crew, the camera crew, and the effects department some basic rules about how to place things in depth, and hazards to avoid, like mismatched reflections and nearby objects straddling the edges of the image.

There were also some issues specifically related to stop motion, like planning 3D for forced perspective sets, shooting single camera 3D through rain and windshields, and resolving 3D strobing artifacts with motion blur in post.

 

3D Advice
Click image for larger.
Brian giving 3D advice to Christ Peterson (lighting cameraman), as they prepare to shoot a scene with the Coraline puppet (behind), sitting on a Bug Chair in the living room of the Other World.



Once they became familiar with these kinds of things, I found that they became more and more interested in my story notes, and how that applied to their 3D work. "Why do I want Coraline in front of the screen, versus behind the screen?" Once you understand the emotion you want to get out of the shot, you can think about how to use 3D to evoke it.

"Coraline" was really the first 3D movie to explore soft focus through shallow depth of field, which was considered a big no-no in 3D. It's why the other stereo experts who were interviewed for "Coraline" before me told the DP, Pete Kozachik, not to use it.

When I met Pete, he was upset. It looked like they were going to prevent him from using depth of field and rack focus blurs in the movie, and he felt that these were incredibly useful artistic tools for him. He didn't want to lose them, just because was doing a 3D movie.

The first thing I said when I got on is, "No, Pete needs this." Not only can you do depth of field blurs in a 3D movie, but you should do depth of field blurs in a 3D movie.

The general rules of using DOF in 3D are different than they are in 2D, of course, so Pete, John Ashley, Chris Peterson, and I then spent several lunches discussing how to approach it. I don't know what the final percentage was, but I imagine that over a third of "Coraline" has some sort of shallow focus. It worked very, very well for the storytelling, because of Pete's artistic vision as a DP.

I think that this emphasis on storytelling first, rather than 3D itself, is one of the two reasons why I got hired. By the time that line producer Harry Linden brought me in, the producers of "Coraline" had spoken to a lot of stereoscopic experts before me. There had been a focus on technical aspects of the production - not hurting people's eyeballs was almost like their paramount concern. To me, it's a given. I don't even put it on my list of issues.

Other people were trying to maximize the 3D impact. Some just approached it with the specific goal of getting more butts in the seats. I got the feeling that, by the time I came along, everyone they'd talked to had a different approach, but all with technical mindsets.

I guess I was the only one who talked about 3D as a storytelling medium. While there are technical issues with 3D, film has technical issues, cameras have technical issues, lighting and costume - everything has technical issues. But 3D isn't a technical problem to be solved. It's an artistic tool, alongside other artistic tools.

I think that there are probably a few people in the 3D community who don't like the fact that I'll advise people to use a shallow amount of depth for certain scenes, but I like to use all the colors in my crayon box.

That's why there's no such thing to me as an optimum amount of depth. That would be like having an optimum amount of light. Sure, there's a certain amount of light you need if you want to see everything perfectly, see all of the colors accurately - but I wouldn't use that kind of light in a horror movie. You want to pick what's appropriate for the story you're trying to tell.

The director of "Coraline," Henry Selick, is very much about the story. Hearing me talk about how to invoke emotions with 3D must have struck a harmonic in him. I could see his eyes twinkle. He then said a few kind words to me, and I received the job offer shortly afterwards.

CHICKEN LITTLE: A GENERATION'S WATERSHED

Ray Zone calls the release of Disney's "Chicken Little 3D" the watershed event in this generation of 3D exhibition - and says that it might never have happened without Brian Gardner.

"I stay after hours on a film that isn't 3D," Brian told us, "just to do sections of it in 3D and show it around, in hopes," he adds with a laugh, "that maybe I can convert people!"

Ray recalls that Brian was working as a visual effects doctor on "Chicken Little" in March of 2005, and just for his amusement, made a second eye view of it. Disney execs happened by as he was playing back an anaglyph view, and were so impressed that they decided to release the film in 3D - in November!

It was the first "wide" release for RealD (84 screens), and the rest, as they say, is history. Or depending on how you look at it, the future.





MATH AND PERCEPTION


I think that the other reason I was hired is that I can explain a lot of the math stuff in plain English.

People have a lot of false assumptions about the way that the human visual system works. I hear people saying that your eyes are 2 1/2 inches apart, and that if you look at something in front of you, your eyes kind of cross as you look at the thing, what's called convergence. And the further and further away the object is, the less and less your eyes cross, so that if you were to look at something at infinity, then both your eyes would be looking straight ahead.

Which means that, to them, if you were to put infinity on a cinema screen, then the furthest separation one object can have between the left and right eyes would be 2 1/2 inches. That was the mantra of 3D experts, that you can't go more than 2 1/2 inches of parallax behind the screen.

I told Henry, well, that's the English-y explanation of what they're trying to say with their math - and it's wrong.


CORALINE

I really appreciate having worked with Henry Selick and PK and the camera crew. You fight an awful uphill battle on many jobs, against people who just don't get it. This was the first show that I ever worked on where everybody got it. It was amazing. Someone would build a puppet, build a set, then the next person would come over and "plus" it, add something even cooler. Then the next person would come along and plus that. I think that if "Coraline" wasn't in 3D, you'd still see that amazing magic.

I'm hoping that the 3D adds to that without being noticeable, or distracting from it, and instead pumps up the emotional volume and pluses it the same way that everybody else's work plused it.



Everybody thinks that the eye is like a camera. We have this nice little evenly-spaced resolution of x and y, and the image goes straight through your pupil and centers on the back of the retina, and all the veins come up and feed it with oxygen from behind, and everything is in focus all the time.

None of those things are true - the reality is much stranger than that.

You actually only have about 3 degrees of high resolution, and about 11 degrees of medium resolution. Everything else is very low resolution. In fact, eyes are almost entirely motion sensors. They see color for the thing you're looking at, and about three degrees around it. Everything else fades into grayscale.

And the only thing that's really in focus is the thing that you're looking at, and maybe a couple of other things around it. Most of what you see is out of focus. You just don't know that, because every time you look at something, it's in focus!

Also, the veins don't come up from behind and feed the retina. They come through a hole into the eyeball, and spread out in FRONT of the retina. Your visual system actually has to erase the veins, and fill in the missing parts of the picture.

The eye sees THROUGH blood vessels

You don't have to converge on something in order to see it in 3D. You can look at something and still see the thing next to it in 3D - which means that all of the math about looking AT something is wrong.

So I told Henry, look, there's the science of the real world, which is the math these other guys are doing. And then there's the science of visual perception, what your EYES are capable of, and what your MIND is capable of perceiving from that.

Because eyes have to erase veins to reconstruct images, the mind is used to making things up. It sometimes creates a relationship between things that the real world may not have put there. For example, you see a chessboard next to a person, and even if they're not playing, you automatically assume that that person is smart - just because of what's sitting next to them. John Singleton Copley used to put fruit next to the women in his portraits, to imply that they are more fruitful people.

I use 3D to create similar perceptual associations. If I want to show that one person's life is deeper than someone else's, then I actually make the space around them deeper. You associate that person with depth. I can put another person in a shallower space, and you automatically think that that person has a shallower life.

I may not squeeze the space so much that people notice this and say, "Hey, this a deep space, and that one's shallow." But I play with the depth enough so that people feel it. They subconsciously pick it up the same way that they subconsciously associate somebody's intelligence with a nearby chessboard.

 

Coraline 3D Depth Script


DEPTH SCORING, part 2: Coraline

(Click image above for larger.)

Everything in 3D has a meaning. More than just a bag of tricks, there's a new visual language offered by 3D, which extends a much richer vocabulary and grammar than 2D did. It was not on my mind to use 3D for gimmicks, nor even for achieving realism. Coraline is a story full of fantasy and the magic of dream stuff. After all, this was a movie formed from the combined geniuses of Neil Gaiman and Henry Selick.

The 3D for Coraline had to compliment their story. It couldn't be allowed to distract from it, nor just make it more 'real'. It was important for the audience to be immersed into the fantasy of these story worlds -- so, my view of the 3D planning for Coraline was that the 3D had to be crafted to be emotionally immersive for the audience.

The image above shows two snippets from the actual Depth Script used on Coraline 3D. I've put these side by side in the graph, for discussion purposes.

The first snippet shows a transition from an indoor Other World scene to a Real World scene. The second snippet shows the transition from a Real Word Scene to an outdoor Other World scene.
Henry Selick first described his vision for Coraline 3D to me by using the metaphor of "The Wizard of Oz". In much the same that Dorothy's real world was in black & white, while Oz was in color -- Henry wanted a sense of boring flatness for Coraline's real world, while the fantastic Other World would be fuller and "better" in Coraline's eyes.

You can see the shallowness and lack of excitement in the Real World (middle of the score), which visually represents Coraline's state of boredom and how dull she sees the her Real World life as being...and her "shallow" life there.

Bracketing that (to the left and right sides of the score), you can see the deeper and more exciting Other World, which represents Coraline's intrigue and temptation by the larger-than-life Other World, which the Other Mother created to tempt Coraline into her trap.

Evolving beyond the initial metaphor of Oz, you can see that even the two Other World scenes also have a dramatically different line quality and spacing in the score, not just that they are both deeper than the Real World scenes. The fluctuation of the depth range in beats (on the left) is designed to subtly increase the audience heartbeat, so the audience can feel the fast excitement of that Other World scene. This can be independent of the cut timing.

The next Other World snippet (on right) spaces the depth range changes slower, fuller, and deeper to induce a feeling of grandeur, and lacks any regular beats, instead using depth events only to emphasize the character's specific emotional moments, to subliminally punch them up.



PERCEPTION, DEPTH

One of the keys to using depth as a creative tool is to understand how it works in relation to the frame. When you watch a 2D movie, you're aware that the screen is on a wall. In 3D, the screen acts more like a hole in the wall, a window into the world of the story, with action taking place in a physical space on either side of it.

If there is an actor who is only partially onscreen in 2D, you understand that the rest of the actor is "behind" the frame. But if you were to take that exact same shot in 3D, and the character was on the audience side of the screen, you'd see half the character floating in the air!

Part of your perceptual system says, "I can only see half of the person, because the other half of him is behind the window, so he's behind the window." Then your stereo system says, "The window frame is behind him, but half of him is way in FRONT of the window! That's not possible!"

We call that a "window violation," when something you see breaks your perceptual system. This conflict causes eyestrain and headaches. If there's a lot of motion, it can even induce vomiting. People are not used to having their perceptual systems in that much conflict for long.

SPLIT BY A TABLE

While today's digital technology has fixed the projection problems of the 50s, it introduces some new ones.

3D cameras capture left and right eyes at the same time, but single-projector solutions from companies like RealD and Dolby don't play them back at the same time. They use a time-alternating system that triple-flashes very quickly: left eye/ right eye, left eye/right eye, left eye/right eye, for each frame.

This creates a certain amount of false depth. Let's say that something moves from left to right. For some brief period of time, your left eye is going to be seeing the next frame, and the right eye is going to be seeing the previous frame.

When you're trying to fuse those images, your brain can't tell how much of that parallax is stereoscopic parallax, and how much of it is motion parallax.

If something goes up and down, then one eye will see it at one height, and the other eye will see it at another height. Your eye is not very forgiving of vertical misalignment, so if that jump is too far, your eye can't fuse the two images. The misalignment can also induce or amplify a strobing or stuttering look.

One of the ways that these issues manifest themselves is interpenetration. We had a character in "Meet The Robinsons" who walked closely behind a table and then stopped. The false depth from the projector made it look like the top half of him was in front of the table.

When he stopped, the top half snapped back, and the top and bottom were reunited behind the table!

So when you're planning 3D movies in this new digital age, you actually have to plan knowing that there will be certain artifacts from the projector. You have to make that character walk further behind the table than he might have otherwise

The audience will not notice the false depth, as long as the character onscreen is not being split by a table.



In 1953, Raymond and Nigel Spottiswoode created a short called "The Black Swan." They believed that the "movie" world should never come into the "audience" world. It was an issue of separatism. They rationalized that they could relocate this window into a fixed position, away from the theater wall, by adding a static black mask to their film, which was made slightly different for the left and right views.

In fact, they floated it halfway out into the theater to prevent any 3D object from accidentally coming into the "audience" world. There would never be a window violation, because everything would always be behind the window.

Audiences didn't like it at all. They could clearly see this static window floating in front of them, and it was distracting. Also annoying was that it created retinally rivalrous areas, large areas at the sides of the frame which could only be seen in one eye. It created more problems than it solved, and was never used again in 3D movies.

Now, when I was in college, I studied the stereograms of Béla Julesz, a perceptual psychologist. He used what looked like random dots to create stereo images on paper.

Random Dot Stereogram


RANDOM DOT STEREOGRAM

(Click image above for larger.)

This random dot autostereogram by John Hsu encodes a 3D scene of a (colorfully dotted) shark swimming before a (colorfully dotted) background.

Random Dot Shark

The 3Dd shark can be "seen" with proper "wall-eyed" viewing technique: the eyes have to be oriented as if focusing on an object behind the image, causing two neighboring repetition of the dot pattern to merge and the shark to appear.

Images and description courtesy of John Hsu, and published under the terms of the GNU Free Distribution License.

 

One of my homework assignments was to replicate Béla Julesz's random dot stereogram experiments.

And I messed it up!

But it was kind of an interesting mistake, so I started to explore it - which is the difference between a technologist and an artist. Technologists find bugs, and fix them. Artists find bugs, and explore them.

I started grappling with, "Why is the frame of the stereo image floating up off the page? Why don't I perceive the paper as falling backward, away?" Instead of random dot stereograms, I started applying it to animation and realized, "Hey, I've got this floating stereo window, and I can move it all over the place!"


I realized that I could lift it off the screen, or push it behind the screen. I could tilt the top toward or away from the audience, kind of a Dutch Tilt but in 3D space. I could rotate it, and even animate it - anything at all. By digitally manipulating the image borders, the window can be highly dynamic.

With the frame moving around so much, you'd think that it would be easy to detect. But when combined with the animation on the screen, you can't see it at all.

It turned out that the failure of the Spottiswoode brothers was that their floating window was static, which made it obtrusive. The new Dynamic Floating Windows, which were moving all over the place, were not even visible at all.

Their invisibility came because they are dynamic.

It was hard for me to convince other industry people that Dynamic Floating Windows would work, because the idea seemed so counter-intuitive to them. They thought that the window changing so much would be too distracting to the audience.

On the set of Coraline
Click for larger.
Brian Gardner giving stereoscopic advice on the set of Coraline.
To the right, you can see the (thin) Spink and Forcible puppets doing their trapeze act.



This is like the evolution of editing. It took a little while to figure how to make cuts invisible. Now, it's weird that a cut that changes camera position, maybe the position in time, sometimes even both - the audience doesn't notice it. People are fine with it. Yet, a Jump Cut, when the camera positions are more similar, is jarring. At first, invisible cuts seemed illogical.

The discovery of Dynamic Floating Windows suddenly opened this whole range of possibilities for 3D storytelling. It gave us the ability to get rid of all the window violations, and gave us a tool for dynamically controlling how we use depth in a scene.

The first time I used dynamic floating windows for storytelling was on "Meet The Robinsons," and it was a revolution that quickly spread. Almost every 3D commercial animated film since then has used them - "Beowulf," "Bolt," "Monsters vs. Aliens," Pixar's "Up," and even "Fly Me to The Moon" from nWave Pictures, a relatively small production company.

"Meet The Robinsons" wasn't just the first time that I got to use Dynamic Floating Windows in a movie. It was also the first time that I really started thinking of them as not just a fix for a technical 3D problem, but as a way to use the 3D window as an artistic tool and a cinematic element.

I used them with stop-motion in "Coraline," and I intend to use them for a live-action movie that I'm working on, still in the planning stages. This may very well be the first time that Dynamic Floating Windows will be used for live action.

I certainly hope so! Though, somebody may get there first, but it would be unfortunate if someone beat me to using my own tool for live action 3D!



Brian Gardner at Stereoscope Studio
Brian exploring stereo shooting options for an upcoming live-action film.



ART IN THE GAP

I like to call 2D movies "flatties," because I think that 3D is a lot like sound was, in the way it revolutionized the industry.

Back then, sound came out and suddenly those new movies were called "talkies." Some number of years later, they're not talkies anymore. They're just called "movies," and suddenly the other things are called "silent movies."

The same thing is going to happen with 3D. Right now, we call them 3D movies, but a few years from now, they'll just be "movies," and the old 2D movies really will be seen as "flatties." That's the way history goes.

I believe that art is that gap between what you experience in reality and what your perceptual system is capable of experiencing. If you can play in that gap and show things that the real world might not naturally present, but that you can present on a movie screen -that's where the art comes into it.

I see 3D undergoing a revolution. It has gone from a technical problem focused around cameras, into a cinematic tool, and now, into an artistic tool.

I can't talk about the specific movies, but I'm currently working on two shows almost like an art director, where I'm coming on fairly early, talking about everything - the set designs, and costume design, and how all of it fits into the gap between reality and perception, the space between what we see, and what we can see.

That's the space where people can perceive something emotionally, even if it wouldn't naturally present itself in the real world, even if they can't consciously see it and can't put their finger on it.

They FEEL it. I love playing with that.



Brian GardnerBrian Gardner
Los Angeles, California USA

Prior to being the Stereographic Advisor on "Coraline 3D", Brian Gardner was most known for having created the Dynamic Floating Windows first introduced in "Meet The Robinsons 3D", which are now a standard in 3D movies; and for having created Gartoons.

Gartoons renders computer animation to look like hand drawn animation of feature film quality. Complete with ink-and-painting, tone mattes, and all the artifacts of hand drawn animation, it can even "shoot on 2's" and do "trace-backs" successfully, which even more recent toon software apps can't do well. It was considered a break-through at the time. It was used in all 3 animated feature movies done at Warner Brothers. "Iron Giant" being the pinnacle for animation buffs.

His diverse experience includes everything from directing stage plays to being a technical director, 3D artist, or technology supervisor on visual effects blockbuster movies, like "X-Men", "The Matrix Reloaded, "The Matrix Revolutions", "Babe", and many others.

Brian holds a B.A. in Mathematics and an M.A. in Computer Science from Boston University, where he was also enrolled in an Interdisciplinary Ph.D. in the UNI program, majoring in Visual Perception & Computer Art. He began a second Ph.D. at M.I.T. in Media Technology, focusing on visual language, before being lured away into the movies.

"I don't usually get interviewed – I usually talk to and work with the people who get interviewed!" he tells us with a laugh. "I'm feeding them ideas, and I'm hoping that I can change the industry by sneaking around, and talking to people in private, one on one." We're happy to help him save a few steps as he reaches a lot more "ones."




Find more great Creative COW Magazine articles by signing up for the complimentary Creative COW Magazine.