Pixels: Going From 8-bits to Epic is No Game
COW Library : TV & Movie Appreciation : Tim Wilson : Pixels: Going From 8-bits to Epic is No Game
The temptation is easy. Take lo-res characters built on iconic video game characters like Pac-Man and Donkey Kong, make 'em big, put 'em in the movie, and leave it at that.
When it came to putting these characters and more into the motion picture Pixels, though, the project became considerably more difficult. As the story goes, space aliens misunderstand transmissions of classic arcade games as actual attacks. In return, they send big, destructive versions of the same game characters to our planet to wreak havoc on an epic scale. Hijinks ensue.
(Worth noting for those who find these things worth noting: while the US opening grosses for Pixels somewhat underperformed expectations, global box office is over $155 million after its fourth weekend. With another dozen-ish weeks to go in theaters, Pixels is doing fine, thanks for asking.)
Mårten Larsson, Visual Effects Supervisor for Digital Domain, is no stranger to creating epics. His previous work includes lead roles on Letters from Iwo Jima, Pirates of the Caribbean: At World's End, and 2012, for which he was nominated for a VES Award for his work as Effects Lead.
"I was on Pixels very early," says Mårten. "I did tests before the movie was greenlit, starting in October 2013. We started shooting in early summer of 2014, plates started rolling in in August-September 2014, and we delivered in July.
"The big thing we were doing is making these game characters come to life and terrorize the streets of New York. From the concept art we knew they would be emitting light, and that was pretty much it going in."
The composition of the characters as we first met them was as lo-res pixels, but this needed adaptation for their incorporation into the "real-world." First, they needed to exist in the same 3D space as New York and the rest of the world, hence their first conversion to "voxels," or "volume elements" – in practice, 3D boxes.
An additional problem immediately presented itself. Characters like Donkey Kong and Pac-Man strike recognizable poses and have distinctive movements in their original contexts, and we've been familiar with those for 35 years.
Donkey Kong, Pixels. Digital Domain has supplied us with beautiful big images, so please click for larger view.
"There was a lot of back and forth to make sure that Donkey Kong stayed looking like Donkey Kong, even when he was in angles that you'd never seen in the game," Mårten says. "Or when he moved between poses," he adds with a laugh. "In the game, that's a one-frame swap. Obviously we needed the in-between motion to make him look somewhat real too, but the tricky thing was to maintain that characteristic look even in non-characteristic angles and poses.
"In the end we tried to minimize that, and not use angles that were too far off from what the game is."
A problem that occurred to me was the challenge of adapting lo-res, 8-bit characters to 4K digital large-format screens. It turns out that the lo-res origins of the characters was actually helpful, Mårten says. "Looking at Donkey Kong for example, his eyes were exactly 4 voxels large in the film, because they were 4 pixels in the game.
"That meant that we had very simple models – but to fight that, and be able to add detail to fit them into the real world." Digital Domain also wanted to ramp up the idea that the characters emitted light – just as the games did of course.
"To play on the whole light-energy idea, we added patterns and light-emitting patterns to the voxels themselves, so the individual boxes had different textures on them. Those patterns would move across the individual voxels, and then across the character. We had edges glowing and light moving, and it gave a lot of detail that worked even if you get up close – but we still retained that lo-res throwback feeling. I like where we landed with that."
The pipeline had to take all of that into account: how the characters would look, how they would move, and how they would balance the need for integration into our world without losing their roots in the game world.
"We started modeling the characters more like a regular skinned character, starting with iconic angles and key poses. Those were rigged for animation, but not as voxelized versions, because that process was fairly heavy. They would animate in Maya, and a plug-in for the Houdini engine would allow them to turn on the voxels to check those key poses, to make sure feet weren't penetrating the ground, things like that. They then published that data, and the post process would turn it all to cubes that were sent to lighting.
"If something needed to happen to those cubes that was specific to the shot – something would break, or some energy had to run through it, then the cubed character was handed over to our effects department, and they would run simulations on top of that.
Along the route, around 160 people from DD touched the film at one point or another
Mårten notes that there have been 3D versions of Pac-Man before, which gave Digital Domain a pretty good start. "We knew how far the mouth needed to open for example, but we were still messing with the eyes quite a bit in terms of placement, size, and resolution that they needed to be for our purposes and still look like Pac-Man."
Previs had also given them some idea of Pac-Man's size, which turned out to be 16 feet. "We knew that he needed to be big enough to bite things – in the movie it was a fire truck, but in the previs it was a bus – so we knew it needed to be big enough to do that, but still fit in the street, so he wouldn't also be running through the sides of the buildings.
PIXELS: Official Pac-Man Clip – Happy 35th Birthday! ©Sony Pictures Entertainment.
"Motion blurring him was an interesting thing as well," says Mårten. "As he opens his mouth, his sphere is deforming, but because the voxels aren't actually moving – the voxels were just turning off and on – it tended to look a little weird. He almost got a little rubbery, so we played with velocities to see if we could make it blur differently."
Compositing was simple enough on the surface, but Digital Domain wanted to take the extra steps to make it really work. "We knew that Pacman would be a 16 foot glowing sphere, in a night scene in New York. To help with interactive lighting, we had a Mini Cooper with big yellow light panels with generators on top. That stood in for Pac-Man to create the emitted light.
"Sometimes it was in the shot, and we put Pacman right on top. We couldn't always shoot with that car, though. It depended on how hard it would be to remove the car from a specific shot, or if Pac-Man was doing something that we couldn't do physically with that car. In those cases, it was a reference pass.
"We also did Lidar scans of all the environments. Even if we shot a frame with the interactive light vehicle, the reflections would be wrong, so we modeled quite a bit of the street, the buildings and the parked cars, just to add interactive light specifically for Pac-Man."
VFX IS EASY, COMEDY IS HARD
Other shots presented additional challenges, of course. For example, with Centipede, there was just a lot of STUFF – mushrooms in the air, centipedes flying around and exploding into boxes as they were shot, and having to animate around the many soldiers in the scene.
Adam Sandler and Josh Gad taking on Centipede.
(Oh yeah, and the human characters, including star Adam Sandler with Josh Gad beside him. Nothing personal, guys. This is a VFX story.)
Overarching all of it, though, was the biggest challenge of all. "This was the first comedy I've worked on!" says Mårten. "The biggest revelation is that comedies are a different beast. There are a lot of edit changes – 'oh, this line is funnier than that line' – so things shifted around a little more in the edit than I'm used to."
A comedy cameo that by now is no longer a spoiler but very much of note: lo-res 80s icon Max Headroom. "We got the original actor, Matt Frewer, to do a facial capture, and we used that data with our Direct Drive technology to transfer it onto a model – very different than the rest of the movie, but still a lot of fun."
Lo-res 80s icon Max Headroom
When I asked for details on Direct Drive, Mårten says, "Direct Drive is a proprietary facial capture system that drives a modeled mesh without having to have a facial rig, so it's a very fast turnaround and very accurate.
"It's so accurate that we had to fight the system a little bit. Back in the day when Matt wore makeup as Max Headroom, there were a number of fixed prosthetics, so parts of his face didn't move as much. But we were getting a lot of movement in those parts, so we had to go in as a post process and try to make the capture a little less accurate!
FROM PIXELS TO VOXELS
You may have seen the most recent picture for which Sony Pictures Imageworks' Steve Nichols filled the role of Animation Supervisor, a Marvel Pictures joint called Guardians of the Galaxy. While there are obvious ways in which that one was more challenging than Pixels, there are ways in which Pixels is was the tough one.
"How do we take something that everybody knows and loves, with 8-bit games like Galaga, and then realize them as 3D characters in a real world environment, interacting with real world things?" Steve asks. "For me, that was a big challenge on the animation side, coming up with hybrid characters. Keeping the design of the characters, keeping the spark of what's in the video games, that simple style – then making it more.
"Then also the motion. There's a nod to what was done in the past, but you have to do something more with those motions to make them look like a character is flying, or attacking. It was getting those two things together and coming up with a cool, creative hybrid."
"There was a lot of trial and error in going from the pixels to the voxels to create a look, a lot of different ideas about how they emitted light, how they moved, even how their surfaces moved – did the surfaces have a fixed sort of feel or not.
"The Space Invaders very much kept the design of the original characters, only 3D-ified. They had to be tangible all of a sudden, they had to have volume, but it was all there for us.
"Other characters with more 'deforming' structures, that was where it became much more difficult. A LOT of trial and error," he laughs. "It came to the point where we were splitting up characters, so that some parts of their bodies would 'voxelate' and actually change voxels as the characters deformed. There were other parts where it was way too distracting, so we held those."
Not every reader will be familiar with Pixels the 2010 short by French filmmaker, which set in motion the events that led to the Adam Sandler feature...but then again, maybe you are. As soon as it went up on YouTube, it went viral, quickly racking up millions of views.
It's only 2:34, and in your scribe's opinion, is a giddy delight: essential viewing. We'll hold the rest of the article while you take a moment to watch.
PIXELS by Patrick JEAN (OFFICIAL BY ONEMOREPROD - HD)
There were a few other bounces between there and here: a bidding war for the feature rights that ended with Adam Sandler's company Happy Madison Productions buying the rights, a script by Tim Herlihy (The Wedding Singer), and Jean Patrick moving out of the director's seat for Chris Columbus (Home Alone, the first two Harry Potter pictures) as the picture became a VFX-heavy extravaganza. (Jean stayed on as an Executive Producer.)
Among those who saw Jean's original Pixels short in 2010 and immediately thought it would make a great feature was in fact Sony's Steve Nichols. "It really was the impetus for everything done since then," he says.
"I think that part of the charm of the short is that simple motions and simple characters wreak such realistic physical destruction. I think the Tetris piece in the short works so well, in how the building collapses into dust as the levels are completed.
"I think for us that was our huge cue, to respect that, and keep that feeling of simple characters and simple motions wreaking havoc in the real world. We were able to have these pixelated explosions, but then the dust that came off them was more realistic. It gave the effects some anchoring weight in reality and made them somewhat terrifying."
In trying to raise the "scary" game, Steve, director Columbus, and the game companies wanted to be sure that the original style was still honored. "We wanted to come up with something that was still that simple style . You didn't WANT it to look too creature-y, or insectoid or terrifying. You wanted to have those bright colors and simple shapes, then what they DO is what sets them apart.
"For example, Nintendo gave us a lot of notes for Duck Hunt Dog, where he hops up onto the couch of an old lady and licks her face. They were very specific that they wanted to have certain key moments from the video game in the actual motion, so it was fun.
Duck Hunt Dog GIF
They gave us model sheets, and we were hand-crafting our poses and animation to match what was in the artwork as closely as possible. Then we went through the same process when then characters were voxelized, to make sure that, like the eyes were were two cubes high," he laughs. "We wanted to make sure to be as dead-on as possible, because that's everybody recognizes that, and there's an immediate reaction."
Steve notes that Sony's work with Digital Domain was very much a collaboration. He and Mårten Larsson had worked together on Pirates of the Caribbean: At World's End, and it happens that Sony and Digital Domain's Vancouver offices are very close to each other. Steven and Mårten bounced ideas through the show's VFX Supervisor Matthew Butler (Titanic, Fight Club, many others), and found that their suggestions sometimes overlapped. "They had several sequences that we also worked on," Steve says, "to the point that our Qbert was in their Donkey Kong shots."
Steve was worried about little Qbert from the beginning. "He's unlike a lot of the end invasion scenes with a lot of exploding and chasing and angry characters. Qbert was much more about emotions, and we were very concerned about how we were going to make him emote -- because he doesn't have a mouth," Steve laughs. "It's all down to the eyes. That's all. When it comes right down to it, he's a BALL, so there's the poses of the body and the eyes.
Adam Sandler with Qbert
"I was definitely concerned that when we voxelate him, and we go from the smooth, quite appealing character, do we lose everything? We had to do a lottttt of iterations," he says, "working closely with evvverbody."
Steve pauses before he adds, "It's actually probably the show that I've worked on most with the other departments. Because our animation was so dependent on our effects division that would voxelate the characters, then lighting, and compositing, making sure that all those elements married together for even the simplest things. So we could see his eyebrows move," he laughs.
FROM 8-BIT TO EPIC: IT'S NO GAME
The challenge weaving through all of this, for Mårten, Steve, and everyone else who worked on Pixels, was keeping the characters anchored in our world. "Dan Kramer, our Visual Effects Supervisor here at Sony Pictures Imageworks, was great with helping me make sure that Qbert was really grounded in the environment. We had to be really on it. He had a WEIGHT to him. If he came off floaty, it would have felt WRONG. You didn't want him to look like this giant uncomfortable Japanese lantern," Steve laughs. "It was definitely a challenge."
What Steve has taken away from this is something he's already implementing on his next show. (No, he won't tell you which movie it is. Stop asking.) "Especially nowadays, viewers demand so much from visual effects. It has to all work TOGETHER.
"Before, animation would pass off their work to effects and compositing, and just let it be. 'We're done with it, you can carry on with it and fix all our problems while we keep going'," he laughs. "We HAVE to be conscious as animators how our work impacts effects, and how much their work is so important to what we do."
This is it in the end: no matter how difficult the effects are to produce, they have to LOOK easy - even if it turns out that what looks quite simple turns out to be very difficult indeed.
Pixels - Trailer