LIBRARY: Tutorials Reviews Interviews Editorials Features Business Authors RSS Feed

Procedural Matte Creation

Procedural Matte Creation

by Barend Onneweer, Gouda/Rotterdam, The Netherlands
©2001 by Barend Onneweer. All rights are reserved. Used at CreativeCow.net by kind permission of the author.


Barend Onneweer

ARTICLE FOCUS:
Barend Onneweer explains Procedural matte creation by combining multiple keying types into a single matte for extensive control over the final matte. This is essential for pulling those difficult mattes from badly lit greenscreens.



INTRODUCTION

An always recurring problem in digital video compositing is the creation of a matte from blue or green screen footage. Most of these problems are due to lack of experience in this field (it definitely is not as easy as it would seem, I learned the hard way…). Also, the nature of DV as a compressed video tends to be problematic for this kind of work. The DV compression produces tiny artefacts in the image, especially around the edges, which is exactly where the problems will arise when trying to pull a decent matte from the footage.

Most of the quality of the final matte and composite is actually established during the shoot. There are a couple of basic guidelines to consider when shooting the footage that will make life a lot easier in post.

That said, this article focuses on the approach after the harm has been done. This approach is actually widely used at high-end composite studios but it's essential to pulling a 'difficult matte'.

The following is a step-by-step walkthrough for what's called procedural matte creation. The examples were made using Adobe After Effects 4.1 Production Bundle. The same approach could also be used using other compositing software of course. I've tried to use only the standard plug-ins that come with the Production Bundle, the results might even be a little bit better using 3rd party keying filters like Ultimatte or Pinnacle's Primatte.

Usually you're not going to be able to produce a decent matte from simply applying one keying filter or effect to one piece of footage. Basically what we're going to do is using different versions of the same clip and different type keying filters to produce the final matte. I have de-interlaced my footage and rendered it out as full frames before I imported them back into this project. I find it easier to create mattes from de-interlaced footage then from interlaced footage.

STEP 1: Setting up

After importing the piece of footage (which I called Original.tif for this tutorial) into an After Effects project, I create two compositions, make sure all the composition settings are identical for these. The first one is named Final and the other is called Matte.

I drag the footage into the Final comp. Then I copy this layer from the Final comp into the Matte comp. This is important, because in other situations where you have a more complex comp, you have to make sure the copy of the clip in the Matte comp is placed at the same in- and out-points, so that the matte remains synchronized to the original clip.

STEP 2: The matte

In the Matte comp, I'm going to create several mattes that will be combined to create the complete matte. This comp has produce a high contrast black and white image to be used as the matte source for the original clip. I put a black solid at the bottom of the comp, because of possible interference of alpha channels.

The basic approach uses a Core Matte and an Edge Matte. The Edge Matte will look nice around the edges, a lot of semi-transparent tones and detail. I often use the After Effects' Color Difference Key filter for this, it's in the Production Bundle. I like the detailed edges this filter produces and it doesn't show a lot of the DV artefacts.

I select the layer of footage in the Matte Comp, and add the Color Difference Key.

It's important to set the view option in the Effect Control window to [matte corrected]. This way the effect produces a black and white matte.

As you can see the Edge Matte has some nice features around the edges, especially along the left side of the girl and the barrel of the gun. This happens a lot with edge mattes, that only one side has a really sharp edge.

CLICK ON THE IMAGES TO SEE IT AT THE ORIGINAL SIZE
EDGE MATTE
CORE MATTE
COMBINED MATTE
EDGE MATTE (detail)
CORE MATTE (detail)
COMBINED MATTE (detail)

So another Matte is needed to fill the holes left in the Edge Matte. This is accomplished by duplicating the layer we have been working on in the Matte comp, removing the effects in the duplicate and setting the transfer mode of the top layer to [screen], so that the results of both layers are combined. It's best to turn off the visibility of the Edge Matte while working on this Core Matte.

I produced the Core Matte using the Color Range keyer, found in the keying filters menu. This filter keys out a range of colors or shades of the same color, the way you specify them. You start by clicking the color picker in the small preview window in the Effects Control window. With the add-color picker (with the little +) you can add points in the preview window until the whole screen is keyed out, but the foreground is still intact. I had the fuzziness set at 20 for this effect.

As I pointed out earlier, the Matte Comp is supposed to be black and white, so I'll add the Levels filter to this layer, found in the Adjust filters. In this effect I set the Output Black to 255, effectively making this whole layer white. Now there's a white silhouette in this layer, but the edges are quite rough as you can see in the details below. So that is why I'll add a last effect to the Core Matte, the Matte Choker, found in the Matte Tools in the effect list. This effect basically smoothes the outlines of the matte and gives the option to shrink and soften it.

THE DIFFERENT STAGES OF THE CORE MATTE
COLOR RANGE KEYER
ADJUST LEVELS
MATTE CHOKER

The Matte Comp would now look something like this:

Creating the matte this way gives a lot of control over the combined matte. Sometimes it is impossible to get a satisfactory matte by just using keying filters. I have on occasion animated small circles on top of these layers to fill certain holes that I couldn't fix with the Core Matte.

Another possibility is the addition of what is called a Garbage Matte. If for instance the camera is static, and there are static objects in the foreground, you could create a black and white image in Photoshop to mask out these objects. This would give you more room to fiddle around with the other layers, since you wouldn't have to worry about those objects.

WARNING: especially with choked mattes, 'fluttering' of the edges can occur, so it's wise to test-render or RAM-preview some sequences instead of only relying on the still.

If you want to see my effect settings for these two layers, click here.

STEP 3: Applying the matte

Now that we've finished the Matte, we'll have it applied to the original footage. I drag the Matte comp from the Project Window into the Final Comp. Again make sure that it starts at frame 0, so that it is in sync with the original footage. The visibility of this layer can be turned off, it will only serve as a matte layer in this comp.

Select the Original footage in the Final Comp, and add the effect Set Matte, found under the Channel effects. This is where I apply the Matte comp as a source for the transparency of the Original footage.

The important settings here are the two roll-down menu's. The upper one defines which layer is set as a source for the matte, the lower defines from which parameter the transparency is derived.

Now the Final comp should look something like this:

As you can see I have added a background layer, to check the matte. If there is still a serious aura of green (or blue, depending on the colour of the screen) surrounding your objects, you can go back and choke the Core Matte a little more, or adjust the settings of the Edge Matte. There are also some other tools available to tweak the edges of a matte, which I'll discuss in the last part of this article.

Step 4: Fine-tuning the final image

To make this effect convincing there are a couple of important things to consider. First of all, the colours and brightness and contrast of the foreground have to match the background. You can adjust these by applying Levels or Color Balance effects to the background or foreground, or both.

Then there are some possibilities of removing what's called green or blue 'spill', the tiny green or blue edges around the foreground. The most basic way to remove these is After Effects own Spill Suppressor, found in the Keying effects. This filter will let you choose a color, which it will then remove from the layer it's applied to. You tend to lose a lot of color in the whole layer though, while you actually only want to remove the spill from the edges.

There are some 3rd party filters that succesfully blend in the colors of a specified background layer around the edges of the foreground layer. This way it looks as if some of the color or light from the background reflects around the edges of the forground. I use the Light Wrap filter for this, which is included in Puffin Designs Composite Wizard, but there are others. These filters also succesfully mask any residual screen that comes through, like it can be seen above in the lower left corner of my Combined Matte.

CLICK ON THE IMAGE TO OPEN THE ORIGINAL SIZE IMAGE
ORIGINAL FOOTAGE
BACKGROUND FOOTAGE
FINAL COMPOSITE

 

The images are taken from my (at the moment almost finished) graduation project, and ultra low-budget science-fiction short, completely shot on DV and edited on off-the-shelf desktop equipment. Check it out at www.raamw3rk.net.

(c) 2001 Barend Onneweer

 

Barend Onneweer is a leader in the Adobe After Effects Creative COW. Drop by and discuss this or other effects. Like to see who Barend is? Click here.

Barend has a new article in the library called 'Shooting for Chromakeying'. In it he has put together a list of things to consider when shooting for chromakeying. Would you like to see that article? Click here.

The Adobe After Effects Creative COW is a part of CreativeCOW.net -- online Creative Communities of the World. This tutorial is for the use of Creative COW members and visitors and may not be reproduced without permission by the author and CreativeCOW.net.

 


 

Visit Creative Cow's website and forums if you got here by direct link to this article...



 

Comments

...edge matte!
by Anders Hattne
Darn it!
After two days of fixing dirty green screen footage, and even clearing a green clad man from a green background, I read this which makes me think I should have re-done all the keys!!

I'll give the edge and core matte technique a try so I remember it for next time!!
Thanks



Related Articles / Tutorials:
Adobe After Effects
Imagineer mocha Pro 5 Plug-In for Adobe: An In Depth Review

Imagineer mocha Pro 5 Plug-In for Adobe: An In Depth Review

Imagineer mocha Pro 5 Plug-in for Adobe brings all the amazing features of the professional version of the mocha Planar Tracker directly into After Effects and Premiere Pro in the form of a plugin. In this in-depth review, After Effects tutorial guru Tobias Gleissenberger of Surfaced Studio will show you what you can do with this new plug-in, and discuss what he likes and doesn't like about the new update.

Tutorial
Tobias Gleissenberger
Adobe After Effects
After Effects 2015.3 - My Favorite Features

After Effects 2015.3 - My Favorite Features

Learn why you should upgrade to After Effects CC 2015.3 - 13.8.1 - a close and detailed look at the latest release of After Effects (August 2016). Roei Tzoref will be focusing on his favorite features that set this release apart from previous versions: Performance, Queue in AME, Lumetri Color new features, and more.

Tutorial
Roei Tzoref
Adobe After Effects
Advanced Masking in Adobe After Effects

Advanced Masking in Adobe After Effects

Some of the coolest stuff you can do inside of Adobe After Effects is only possible once you unlock the power of masks. Join After Effects whiz Tobias Gleissenberger of Surfaced Studio to learn about mask animation and interpolation, using the variable width feathering tool, managing mask modes and ordering, and more.

Tutorial
Tobias Gleissenberger
Adobe After Effects
Fixing Common After Effects Problems and Mistakes

Fixing Common After Effects Problems and Mistakes

Got problems using Adobe After Effects? Exported files too large, Expressions not working, mixing shapes and makes, modes/switches, selecting previews for layers vs. comps - Surfaced Studio's Tobias Gleissenberger shows you fast fixes for these and more!

Tutorial
Tobias Gleissenberger
Adobe After Effects
mocha AE Planar Tracker for Absolute Beginners

mocha AE Planar Tracker for Absolute Beginners

Want to learn how to create advanced visual effects? Learn how to use mocha AE to track your shot and add advanced visual effects to live action footage inside Adobe After Effects. mocha can help you track shots that would be hard to track using traditional 2D point or feature trackers because it is a PLANAR TRACKER. A planar tracker uses planes and textures to track as opposed to points or groups of pixels. This allows the tracker to stay on track even if your shot contains motion blur or a very shallow depth of field. mocha AE comes included with Adobe After Effects and is available since CS3 and there is no reason for you not to use this awesome tool to make it easier for you to track your shot, replace screens or rotoscope!

Tutorial
Tobias Gleissenberger
Adobe After Effects
How to Make After Effects Faster with Proxies

How to Make After Effects Faster with Proxies

Learn how to make Adobe After Effects faster by using PROXIES! A proxy is a placeholder that stands in place for a very large video file or image sequence in your project. You can easily create proxies for the large source files that you are using and After Effects will automatically link them to the item in your project panel.

Tutorial
Tobias Gleissenberger
Adobe After Effects
Stabilize & Smooth: mocha 5 Plug-in for Adobe & Avid

Stabilize & Smooth: mocha 5 Plug-in for Adobe & Avid

Imagineer Systems and Boris FX product specialist Mary Poplin shows you how to stabilize with the new mocha Pro 5 plug-in inside of Avid Media Composer, Adobe Premiere Pro and After Effects. This tutorial covers artistic stabilization, such as smoothing out camera movements or stabilizing around moving objects, completely locking down shots, and automatically replacing edge fill on planar backgrounds.

Tutorial
Mary Poplin
Adobe After Effects
Adobe After Effects Puppet Tool

Adobe After Effects Puppet Tool

Become a puppet master by learning how to use the Puppet Tool in Adobe After Effects! This intermediate-level tutorial from After Effects guru Tobias will show you how the Puppet Tool allows you to add joints and animations to bring life to any static image!

Tutorial
Tobias Gleissenberger
Adobe After Effects
How to Spawn A Clone in Adobe After Effects

How to Spawn A Clone in Adobe After Effects

Want to learn how to create a cool clone spawn effect in Adobe After Effects? Follow along with After Effects whiz Tobias from Surfaced Studio in this exciting new visual effects tutorial that combines green screen using Keylight, CC Vector Blur, the Liquefy Effect, CC Particle World, and much more, delivered in Tobias' inimitable style!

Tutorial
Tobias Gleissenberger
Adobe After Effects
Creating A Flame on Your Finger with After Effects

Creating A Flame on Your Finger with After Effects

It is easy to do some motion tracking and attach a basic stock footage element of fire onto your hands, but there is a little bit of work involved if you actually want to make it look good. In this intermediate tutorial by After Effects expert Tobias, you'll see how to use a fire stock footage element to set your thumb on fire! There are lots of useful tricks for null objects, expressions, and more in this tutorial that will help you create all sorts of other cool visual effects -- or set even more things on fire!

Tutorial
Tobias Gleissenberger
MORE
© 2016 CreativeCOW.net All Rights Reserved
[TOP]