LIBRARY: Tutorials Reviews Interviews Editorials Features Business Authors RSS Feed

Pixels: Going From 8-bits to Epic is No Game

COW Library : TV & Movie Appreciation : Tim Wilson : Pixels: Going From 8-bits to Epic is No Game
CreativeCOW presents Pixels: Going From 8-bits to Epic is No Game -- TV & Movie Appreciation Feature
Palm Springs, CA All rights reserved.

The temptation is easy. Take lo-res characters built on iconic video game characters like Pac-Man and Donkey Kong, make 'em big, put 'em in the movie, and leave it at that.

When it came to putting these characters and more into the motion picture Pixels, though, the project became considerably more difficult. As the story goes, space aliens misunderstand transmissions of classic arcade games as actual attacks. In return, they send big, destructive versions of the same game characters to our planet to wreak havoc on an epic scale. Hijinks ensue.

(Worth noting for those who find these things worth noting: while the US opening grosses for Pixels somewhat underperformed expectations, global box office is over $155 million after its fourth weekend. With another dozen-ish weeks to go in theaters, Pixels is doing fine, thanks for asking.)

Mårten Larsson, Visual Effects Supervisor for Digital Domain, is no stranger to creating epics. His previous work includes lead roles on Letters from Iwo Jima, Pirates of the Caribbean: At World's End, and 2012, for which he was nominated for a VES Award for his work as Effects Lead.

"I was on Pixels very early," says Mårten. "I did tests before the movie was greenlit, starting in October 2013. We started shooting in early summer of 2014, plates started rolling in in August-September 2014, and we delivered in July.

"The big thing we were doing is making these game characters come to life and terrorize the streets of New York. From the concept art we knew they would be emitting light, and that was pretty much it going in."

The composition of the characters as we first met them was as lo-res pixels, but this needed adaptation for their incorporation into the "real-world." First, they needed to exist in the same 3D space as New York and the rest of the world, hence their first conversion to "voxels," or "volume elements" – in practice, 3D boxes.

An additional problem immediately presented itself. Characters like Donkey Kong and Pac-Man strike recognizable poses and have distinctive movements in their original contexts, and we've been familiar with those for 35 years.

Donkey Kong Pixels
Donkey Kong, Pixels. Digital Domain has supplied us with beautiful big images, so please click for larger view.

"There was a lot of back and forth to make sure that Donkey Kong stayed looking like Donkey Kong, even when he was in angles that you'd never seen in the game," Mårten says. "Or when he moved between poses," he adds with a laugh. "In the game, that's a one-frame swap. Obviously we needed the in-between motion to make him look somewhat real too, but the tricky thing was to maintain that characteristic look even in non-characteristic angles and poses.

"In the end we tried to minimize that, and not use angles that were too far off from what the game is."

A problem that occurred to me was the challenge of adapting lo-res, 8-bit characters to 4K digital large-format screens. It turns out that the lo-res origins of the characters was actually helpful, Mårten says. "Looking at Donkey Kong for example, his eyes were exactly 4 voxels large in the film, because they were 4 pixels in the game.

"That meant that we had very simple models – but to fight that, and be able to add detail to fit them into the real world." Digital Domain also wanted to ramp up the idea that the characters emitted light – just as the games did of course.

"To play on the whole light-energy idea, we added patterns and light-emitting patterns to the voxels themselves, so the individual boxes had different textures on them. Those patterns would move across the individual voxels, and then across the character. We had edges glowing and light moving, and it gave a lot of detail that worked even if you get up close – but we still retained that lo-res throwback feeling. I like where we landed with that."


The pipeline had to take all of that into account: how the characters would look, how they would move, and how they would balance the need for integration into our world without losing their roots in the game world.

"We started modeling the characters more like a regular skinned character, starting with iconic angles and key poses. Those were rigged for animation, but not as voxelized versions, because that process was fairly heavy. They would animate in Maya, and a plug-in for the Houdini engine would allow them to turn on the voxels to check those key poses, to make sure feet weren't penetrating the ground, things like that. They then published that data, and the post process would turn it all to cubes that were sent to lighting.

"If something needed to happen to those cubes that was specific to the shot – something would break, or some energy had to run through it, then the cubed character was handed over to our effects department, and they would run simulations on top of that.

Along the route, around 160 people from DD touched the film at one point or another

VFX pipeline render passes, click for larger view: 1) Background plate; 2) Animation render; 3) Animation voxelized render; 4)FX sim; 5) Character shadows and reflections; 6) Final comp.


Mårten notes that there have been 3D versions of Pac-Man before, which gave Digital Domain a pretty good start. "We knew how far the mouth needed to open for example, but we were still messing with the eyes quite a bit in terms of placement, size, and resolution that they needed to be for our purposes and still look like Pac-Man."

Previs had also given them some idea of Pac-Man's size, which turned out to be 16 feet. "We knew that he needed to be big enough to bite things – in the movie it was a fire truck, but in the previs it was a bus – so we knew it needed to be big enough to do that, but still fit in the street, so he wouldn't also be running through the sides of the buildings.

PIXELS: Official Pac-Man Clip – Happy 35th Birthday! ©Sony Pictures Entertainment.

"Motion blurring him was an interesting thing as well," says Mårten. "As he opens his mouth, his sphere is deforming, but because the voxels aren't actually moving – the voxels were just turning off and on – it tended to look a little weird. He almost got a little rubbery, so we played with velocities to see if we could make it blur differently."

Compositing was simple enough on the surface, but Digital Domain wanted to take the extra steps to make it really work. "We knew that Pacman would be a 16 foot glowing sphere, in a night scene in New York. To help with interactive lighting, we had a Mini Cooper with big yellow light panels with generators on top. That stood in for Pac-Man to create the emitted light.

"Sometimes it was in the shot, and we put Pacman right on top. We couldn't always shoot with that car, though. It depended on how hard it would be to remove the car from a specific shot, or if Pac-Man was doing something that we couldn't do physically with that car. In those cases, it was a reference pass.

"We also did Lidar scans of all the environments. Even if we shot a frame with the interactive light vehicle, the reflections would be wrong, so we modeled quite a bit of the street, the buildings and the parked cars, just to add interactive light specifically for Pac-Man."


Other shots presented additional challenges, of course. For example, with Centipede, there was just a lot of STUFF – mushrooms in the air, centipedes flying around and exploding into boxes as they were shot, and having to animate around the many soldiers in the scene.

Adam Sandler and Josh Gad taking on Centipede
Adam Sandler and Josh Gad taking on Centipede.

(Oh yeah, and the human characters, including star Adam Sandler with Josh Gad beside him. Nothing personal, guys. This is a VFX story.)

Overarching all of it, though, was the biggest challenge of all. "This was the first comedy I've worked on!" says Mårten. "The biggest revelation is that comedies are a different beast. There are a lot of edit changes – 'oh, this line is funnier than that line' – so things shifted around a little more in the edit than I'm used to."

A comedy cameo that by now is no longer a spoiler but very much of note: lo-res 80s icon Max Headroom. "We got the original actor, Matt Frewer, to do a facial capture, and we used that data with our Direct Drive technology to transfer it onto a model – very different than the rest of the movie, but still a lot of fun."

:o-res 80s icon Max Headroom
Lo-res 80s icon Max Headroom

When I asked for details on Direct Drive, Mårten says, "Direct Drive is a proprietary facial capture system that drives a modeled mesh without having to have a facial rig, so it's a very fast turnaround and very accurate.

"It's so accurate that we had to fight the system a little bit. Back in the day when Matt wore makeup as Max Headroom, there were a number of fixed prosthetics, so parts of his face didn't move as much. But we were getting a lot of movement in those parts, so we had to go in as a post process and try to make the capture a little less accurate!


You may have seen the most recent picture for which Sony Pictures Imageworks' Steve Nichols filled the role of Animation Supervisor, a Marvel Pictures joint called Guardians of the Galaxy. While there are obvious ways in which that one was more challenging than Pixels, there are ways in which Pixels is was the tough one.

"How do we take something that everybody knows and loves, with 8-bit games like Galaga, and then realize them as 3D characters in a real world environment, interacting with real world things?" Steve asks. "For me, that was a big challenge on the animation side, coming up with hybrid characters. Keeping the design of the characters, keeping the spark of what's in the video games, that simple style – then making it more.

"Then also the motion. There's a nod to what was done in the past, but you have to do something more with those motions to make them look like a character is flying, or attacking. It was getting those two things together and coming up with a cool, creative hybrid."


"There was a lot of trial and error in going from the pixels to the voxels to create a look, a lot of different ideas about how they emitted light, how they moved, even how their surfaces moved – did the surfaces have a fixed sort of feel or not.

"The Space Invaders very much kept the design of the original characters, only 3D-ified. They had to be tangible all of a sudden, they had to have volume, but it was all there for us.

"Other characters with more 'deforming' structures, that was where it became much more difficult. A LOT of trial and error," he laughs. "It came to the point where we were splitting up characters, so that some parts of their bodies would 'voxelate' and actually change voxels as the characters deformed. There were other parts where it was way too distracting, so we held those."

Not every reader will be familiar with Pixels the 2010 short by French filmmaker, which set in motion the events that led to the Adam Sandler feature...but then again, maybe you are. As soon as it went up on YouTube, it went viral, quickly racking up millions of views.

It's only 2:34, and in your scribe's opinion, is a giddy delight: essential viewing. We'll hold the rest of the article while you take a moment to watch.


There were a few other bounces between there and here: a bidding war for the feature rights that ended with Adam Sandler's company Happy Madison Productions buying the rights, a script by Tim Herlihy (The Wedding Singer), and Jean Patrick moving out of the director's seat for Chris Columbus (Home Alone, the first two Harry Potter pictures) as the picture became a VFX-heavy extravaganza. (Jean stayed on as an Executive Producer.)

Among those who saw Jean's original Pixels short in 2010 and immediately thought it would make a great feature was in fact Sony's Steve Nichols. "It really was the impetus for everything done since then," he says.

"I think that part of the charm of the short is that simple motions and simple characters wreak such realistic physical destruction. I think the Tetris piece in the short works so well, in how the building collapses into dust as the levels are completed.

"I think for us that was our huge cue, to respect that, and keep that feeling of simple characters and simple motions wreaking havoc in the real world. We were able to have these pixelated explosions, but then the dust that came off them was more realistic. It gave the effects some anchoring weight in reality and made them somewhat terrifying."

In trying to raise the "scary" game, Steve, director Columbus, and the game companies wanted to be sure that the original style was still honored. "We wanted to come up with something that was still that simple style . You didn't WANT it to look too creature-y, or insectoid or terrifying. You wanted to have those bright colors and simple shapes, then what they DO is what sets them apart.

"For example, Nintendo gave us a lot of notes for Duck Hunt Dog, where he hops up onto the couch of an old lady and licks her face. They were very specific that they wanted to have certain key moments from the video game in the actual motion, so it was fun.

Duck Hunt Dog GIF

They gave us model sheets, and we were hand-crafting our poses and animation to match what was in the artwork as closely as possible. Then we went through the same process when then characters were voxelized, to make sure that, like the eyes were were two cubes high," he laughs. "We wanted to make sure to be as dead-on as possible, because that's everybody recognizes that, and there's an immediate reaction."


Steve notes that Sony's work with Digital Domain was very much a collaboration. He and Mårten Larsson had worked together on Pirates of the Caribbean: At World's End, and it happens that Sony and Digital Domain's Vancouver offices are very close to each other. Steven and Mårten bounced ideas through the show's VFX Supervisor Matthew Butler (Titanic, Fight Club, many others), and found that their suggestions sometimes overlapped. "They had several sequences that we also worked on," Steve says, "to the point that our Qbert was in their Donkey Kong shots."

Steve was worried about little Qbert from the beginning. "He's unlike a lot of the end invasion scenes with a lot of exploding and chasing and angry characters. Qbert was much more about emotions, and we were very concerned about how we were going to make him emote -- because he doesn't have a mouth," Steve laughs. "It's all down to the eyes. That's all. When it comes right down to it, he's a BALL, so there's the poses of the body and the eyes.

Adam  Sandler  with  Qbert
Adam Sandler with Qbert

"I was definitely concerned that when we voxelate him, and we go from the smooth, quite appealing character, do we lose everything? We had to do a lottttt of iterations," he says, "working closely with evvverbody."

Steve pauses before he adds, "It's actually probably the show that I've worked on most with the other departments. Because our animation was so dependent on our effects division that would voxelate the characters, then lighting, and compositing, making sure that all those elements married together for even the simplest things. So we could see his eyebrows move," he laughs.

"When Qbert is in the DARPA research lab, where he gets to have some dialogue, and act, seeing that payoff and saying, (laughs, in a surprised voice, "Hey, that actually worked out really well."


The challenge weaving through all of this, for Mårten, Steve, and everyone else who worked on Pixels, was keeping the characters anchored in our world. "Dan Kramer, our Visual Effects Supervisor here at Sony Pictures Imageworks, was great with helping me make sure that Qbert was really grounded in the environment. We had to be really on it. He had a WEIGHT to him. If he came off floaty, it would have felt WRONG. You didn't want him to look like this giant uncomfortable Japanese lantern," Steve laughs. "It was definitely a challenge."

What Steve has taken away from this is something he's already implementing on his next show. (No, he won't tell you which movie it is. Stop asking.) "Especially nowadays, viewers demand so much from visual effects. It has to all work TOGETHER.

"Before, animation would pass off their work to effects and compositing, and just let it be. 'We're done with it, you can carry on with it and fix all our problems while we keep going'," he laughs. "We HAVE to be conscious as animators how our work impacts effects, and how much their work is so important to what we do."

This is it in the end: no matter how difficult the effects are to produce, they have to LOOK easy - even if it turns out that what looks quite simple turns out to be very difficult indeed.

Pixels - Trailer


Related Articles / Tutorials:
TV & Movie Appreciation
Designing Tarantino's Once Upon A Time in Hollywood

Designing Tarantino's Once Upon A Time in Hollywood

Barbara Ling, production designer for Once Upon a Time in Hollywood, discusses how she was able to turn back time and recreate 1960’s Hollywood. Barbara and Go Creative Show host Ben Consoli discuss how Quentin Tarantino kept the film’s script a secret, how she restored Hollywood to the 1960’s, not using green screen, sourcing vintage props from eBay, filming the real Playboy Mansion, the challenges of filming on Hollywood Boulevard, and much more.

Ben Consoli
TV & Movie Appreciation
Great Effects for Good Omens: Collaboration with Milk VFX

Great Effects for Good Omens: Collaboration with Milk VFX

Wouldn’t you know it? You’re a demon living on Earth for the past 6000 years, and you’ve finally persuaded your angelic counterpart to help you stop the end of the world, and now you can’t remember where you left the 11 year old Antichrist. Such are the tribulations of Good Omens, the newly posted Amazon Prime Video series based on the beloved 1990 classic fantasy novel by Terry Pratchett and Neil Gaiman. We spoke with the Oscar, Emmy, and BAFTA-winning team at Milk VFX about how they brought this epic comic fantasy to life.

Tim Wilson
TV & Movie Appreciation
Gods, Marvel, & Maui: Making VFX on a Pacific Island

Gods, Marvel, & Maui: Making VFX on a Pacific Island

It's safe to say that there's no other VFX vendor in the world quite like capital T, let alone one that is constantly contributing to films like Marvel Studios' Avengers: Infinity War and Black Panther, as well as the critically acclaimed TV series, American Gods. What makes capital T unique is that they’re a two person, husband and wife team, who work from their home office ??" a beach house in Hawaii. You'll be inspired by what 2 people can do, if also a little envious of where they're doing it!

Feature, People / Interview
COW News
TV & Movie Appreciation
Avengers: Infinity War - Thanos, Titan, and Weta Digital

Avengers: Infinity War - Thanos, Titan, and Weta Digital

At the center of one of the biggest films in history, Marvel Studios' Avengers: Infinity War, stands Thanos, a CG character whose emotional range became a major contribution to the film's success. Creative COW Contributing Editor Hillary Lewis spoke with Weta Digital VFX Supervisor Matt Aitken about the challenges of their work on this remarkably compelling character and his world.

Feature, People / Interview
Hillary Lewis
TV & Movie Appreciation
VFX Legion Completes Effects for SUPERFLY Remake

VFX Legion Completes Effects for SUPERFLY Remake

Producer Joel Silver and Director X called on VFX Legion to tackle 100+ shots designed to amp up the impact of the raw violence in Sony’s reboot of the iconic ‘70’s film. The reboot of ‘Superfly’ puts a modern, stylish spin on the original 1972 film about a Harlem drug dealer trying to score one last deal before getting out of ‘the game.’ Set in present-day Atlanta, the Mecca of today’s popping music scene, the action is driven by a hip-hop soundtrack curated by Future. The city’s distinctive style is the backdrop for a new generation of affluent, extravagant drug kingpins that takes violence to the extreme.

Editorial, Feature, Project
VFX Legion
TV & Movie Appreciation
Star Wars: How Much Is Too Much?

Star Wars: How Much Is Too Much?

When Disney announced that they would be making a new Star Wars movie every year for at least 10 years I was both excited and a bit skeptical. In 2012 when Lucas sold his company to Disney for $4billion, he included his outlines of Episodes VII, VIII and IX. But Disney and Co. decided to discard these stories and start over, also discarding the extended universe of comics and books that millions of SW fans had grown to love. Adding JJ Abrams to the mix was icing on the cake for SW fans who have become critical of SW. But Lawrence Kasdan was the saving grace, who wrote a script for VII that the original actors could get behind. So, how much Star Wars is too much?

Review, Editorial, Feature
Mike Cohen
TV & Movie Appreciation
The Long Road Home with Seth Reed

The Long Road Home with Seth Reed

Seth Reed is the Emmy®-nominated production designer for the National Geographic miniseries The Long Road Home. Seth joins Go Creative Show host Ben Consoli to discuss the challenges and benefits of shooting at Fort Hood in Texas and how he created the biggest standing set in North America.

Editorial, Feature, People / Interview
Ben Consoli
TV & Movie Appreciation
Favreau, Technicolor & MPC Make The Jungle Book Come Alive

Favreau, Technicolor & MPC Make The Jungle Book Come Alive

Todd McCarthy, veteran film critic and historian, in his review of director Jon Favreau's new, stunning adaptation of Rudyard Kipling's The Jungle Book declared, "...the visual effects team led by Robert Legato and (MPC's) Adam Valdez has both created sumptuous settings that look as lifelike as any CGI ever presented in a studio feature and integrated both humans and animal characters in them in seamless ways."

Editorial, Feature
Jon Favreau
TV & Movie Appreciation
VFX Legion | Hardcore Henry breakdown reel

VFX Legion | Hardcore Henry breakdown reel

Remote post-production and visual effects studio VFX Legion has released its breakdown reel for the incendiary Hardcore Henry. The reel reveals the work that went into the first-person perspective action film, from augmenting violence to stitching shots together into one continuous sequence.

Editorial, Feature
COW News
TV & Movie Appreciation
Renaissance Masters Go 3D with Nuke

Renaissance Masters Go 3D with Nuke

VFX legend Steve Wright helped Italy's Sky 3D tackle an epic project, as Italian all-3D television station set out to present the city of Florence and the masterpieces of Renaissance art housed in the Uffizi Gallery in a spectacular stereoscopic 3D movie shown in 60 countries around the world. While the majority of the film was shot stereoscopically, Steve's challenge was to use Nuke to present some of the world's most precious artworks fully dimensionalized. Here's how he pulled it off.

Editorial, Feature
Steve Wright
© 2020 All Rights Reserved