Renaissance Masters Go 3D with Nuke
COW Library : TV & Movie Appreciation : Steve Wright : Renaissance Masters Go 3D with Nuke
In 2015 Sky 3D, an Italian all-3D television station, embarked on an epic project to present the city of Florence and the masterpieces of Renaissance art housed in the Uffizi Gallery in a spectacular stereoscopic 3D movie shown in 60 countries around the world. This would give movie-goers a chance to see some of the greatest works of art ever produced by man (and one woman ) with a never before seen intimacy that placed the viewer at eye level with the statue of David and face to face with the mighty works of Michelangelo and Leonardo da Vinci by using the latest Hollywood blockbuster techniques of 3D movie making.
Titled "Florence and the Uffizi Gallery 3D," much of the movie would be shot with state-of-the-art stereo camera rigs. But 27 works of art would be converted to stereo 3D, or "dimensionalized," from high resolution photographs using the same Hollywood tech used to convert "Clash of the Titans" to 3D. In this article we will see the range of techniques used for the stereo conversion and the unique problems associated with applying it to fine art.
Madonna of the Harpies in 3D. 17th century female painter Artemisia Gentilieschi painted Judith Slaying Holofernes.
Some of the images herein are 3D so you will want to fetch your red and blue anaglyph 3D glasses to view them. The masterpiece Madonna of the Harpies by Andrea del Sarto shown here (time to don your glasses) is a classic example of one if the 27 works of art converted to stereo 3D. We can only view a small version here, but imagine the impact of seeing this in 3D on the big screen.
Red and blue anaglyph 3D glasses.
I first became aware of this project when I was contacted by a former student, Daniele Pugni, who was now the head of Sky Production & Creative Hub, the motion graphics department of Sky 3D. He had taken one of my courses on compositing feature film visual effects using Nuke, the dominant visual effects software in the entertainment industry. But now he needed to use Nuke on a whole new level – to dimensionalize classic works of art to convert them to 3D – so we began a series of personal training sessions so I could bring him up to speed on the arcane art of stereo conversion. The training was conducted remotely using desktop sharing to his office in Milan, Italy.
How 3D Movies Are Made
Before we begin I must confess that the subject of 3D movie-making has an unfortunate terminology conflict. On the movie-making (production) side it is referred to as "stereo". We have stereo camera rigs, stereographers, and stereo visual effects. On the display side (TV, movie theatre, photographs) it suddenly becomes 3D. We have 3D movies, 3D TV's, and 3D glasses. That is why in this article you will see the term "stereo" when we are discussing the making of the pictures but "3D" when we talk about viewing them.
Let us begin with a brief review of how stereo 3D movies are made. Our vision is stereoscopic (3D) because we get two slightly different views of a scene, one from each eye. To make a 3D movie we also need two points of view. The technique for shooting a stereo 3D movie is illustrated here. Starting on the left, we have the stereo camera rig with two cameras photographing an object from slightly different angles (the angles are exaggerated for illustration purposes). In the middle are the two views of the object seen at slightly different angles. These two views are then combined to form the 3D image on the right (3D glasses on).
Stereo camera rig (left). (Right) Two views, resulting 3D image.
For the observer to see the image in 3D the display system (photograph, TV, movie screen) must somehow show a different view to each eye. In the movie theatre big digital projectors project the two images with polarized light that the audience's 3D glasses can separate into the two views. For photographs like in this article we have to fall back on the ancient red and blue (red and cyan actually) anaglyph system which uses colored filters to separate the two views.
How Stereo Conversion Is Done
The movie industry's recent transition to digital production has lead to a boom in 3D movies because digital technology makes 3D much more practical and aesthetically much better than the old 35mm film technologies. But digital technology has also made a whole new idea possible, namely stereo conversion – converting a movie originally shot in 2D into a 3D movie. This is the technology that made the 3D versions of the Renaissance masterpieces. But how do you start with one view of a painting and turn it into the two slightly different views that are required to make the 3D version? An understanding of this process will reveal the conundrum presented to the digital artists on this particular project.
(Left) 2D shot converted to 3D. (Middle) Original 2D shot. (Right) Second view.
Let's look (with your 3D glasses) at how to make the converted 3D shot shown in the left panel of the illustration above. The middle panel shows a closeup of the original 2D shot of the boy. The right panel shows how the boy needs to be shifted to the right to create the second view for stereo conversion. However, when he is shifted right it reveals previously covered regions of the background which are marked in red.
If we were converting a feature film this would not be an issue. In visual effects we have a variety of methods for re-creating the missing regions of the background. However, if we are trying to dimensionalize a masterpiece of art, who are we to make up the missing pieces of a masterpiece? And that is the conundrum of the digital artists trying to dimensionalize these works of art. The artists on this job were forbidden to simply "invent" the missing background regions for historical works of art created by Renaissance masters.
There was a two-fold solution to this problem. The first was to keep the dimensionalization and 3D camera moves to a minimum so only small regions of the background are affected. The more the picture is "extruded" in depth and the greater the 3D camera moves the more background that must be restored. The second was to ever so gently stretch the pixels in the background to cover the affected regions. Between these two strategies a balance was struck that would not offend either the Renaissance masters or the viewing audience.
Leonardo da Vinci Self Portrait
There are a range of techniques used to dimensionalize pictures and which one to use is based on the content and complexity of the scene. On the "simpler" end of the stereo conversion spectrum is what is often referred to as the "rubber sheet" method which was used for the Leonardo da Vinci Self Portrait. It is as though the image to be converted is on a sheet of rubber that is vacuum formed over a mold. The mold provides the depth dimensionality for the stereo cameras while the image provides the picture content. Photographed by the stereo cameras from the two slightly different angles provides the two views used to create the stereo 3D effect.
(Left) Original photograph. (Middle) Hand painted depth map. (Right) Displaced 3D mesh.
How it is done is shown in the three illustrations above. On the left is the original artwork photographed at very high resolution. From this photograph an artist uses Photoshop to paint a "depth map" (middle grayscale image). The depth map represents the distance from the camera to the painting with brighter pixels being closer to the camera. This depth map is then used to displace the vertices of a 3D mesh shifting the bright pixels closer to the camera and darker pixels further from the camera as seen in the image on the right. This is the "mold" mentioned above upon which the original image will be "vacuum formed" to dimensionalize it when re-photographed by the stereo cameras.
Camera and 3D geometry setup in Nuke
The illustration above shows the 3D arrangement required to render each view. The rear-most layer is the original painting placed as the background. The da Vinci character is isolated using Photoshop then projected onto the displaced 3D mesh in front of the painting. This way the isolated and dimensionalized character is rendered separately so its edges can be artfully blended with the original painting background. In front of all that is the 3D camera used to render this view. A parallel setup is used for the other view but with the da Vinci element offset a bit to introduce the parallax shift required to create the stereo effect.
Here is the resulting stereo 3D image produced from the two views rendered in Nuke.
Converted da Vinci 3D image.
The Head of Medusa by Caravaggio
The head of Medusa by Michelangelo Caravaggio (not "the" Michelangelo) presented a truly unusual stereography problem. As you can see from the production photo here that it was painted on a convex shield (curved outward). However, Caravaggio, an avid bar-room brawler and miscreant, was also a master of realism and lighting. He actually painted it to make the convex shield appear concave (curved inward) with the Medusa head suspended within the curvature of the shield. This posed a vexing problem for the stereographers.
Head of Medusa on shield.
Physically, the Medusa was painted on the surface of a shield that was curved outward but artistically it was intended to look like it was suspended within a shield that was curved inward. Daniel Pugni chose to go with the artistic intent rather than physical reality, so the stereo version would be made to look like the shield was concave and the Medusa head was suspended within it. But how might this be done?
3D model of the Head of Medusa painting.
To pull off this visual inversion required extraordinary skill and technique because the stereo cameras simply cannot be fooled. It was necessary to actually build a 3D model of what the Medusa and shield would look like if it were suspended within a concave shield, project the original Head of Medusa photograph onto the 3D geometry, then re-photograph it with a pair of stereo cameras. As you can see from the picture above the 3D model was very complex and much had to be inferred about the depth of each element by carefully interpreting the lighting painted into the original masterpiece.
Converted stereo 3D photo of Head of Medusa on shield.
The deliciously horrific 3D results can be seen in the resulting 3D image above. The shield appears concave and Medusa's head is floating within it just like Michelangelo Caravaggio envisioned it, but was never actually able to see it himself. A dramatic camera move was added to further emphasize the 3-dimensional horror.
One other unique issue in a project like this was that the stereo artists were not permitted to fix any of the defects in the art. These sculptures and paintings are hundreds of years old and have accumulated scratches and mars that are now considered part of the art. From a feature film standpoint, our natural instincts are to fix every blemish in every frame for a perfect viewing experience. But not this time. The Sky 3D stereographers had to stay their hand.
There is no feature film stereo conversion industry in Italy so this project represented a major step forward both technically and artistically. Sky 3D's team has successfully made the transition from a motion graphics department to a full-up visual effects department and have already embarked on their next exciting adventure for Sky 3D – another 3D spectacular movie "St. Peter's and the Papal Basilicas of Rome 3D" which will give the world a 3D tour of four magnificent cathedrals in Rome. With such spectacular subject matter we are expecting more great things from the stereo artists of Sky 3D where classic art meets Hollywood tech.
Courses on Lynda.com - http://www.lynda.com/Steve-Wright/127-1.html
Facebook - https://www.facebook.com/nukeguru
Compositing Visual Effects – introductory book http://www.amazon.com/exec/obidos/ASIN/0240817818/wwwswdfxcom-20
Digital Compositing for Film and Video – advanced book http://www.amazon.com/exec/obidos/ASIN/024081309X/wwwswdfxcom-20