Turning Paint Into Masks in After Effects

Timothy Kurkoski Beaverton, Oregon, USA ©2003 Timothy Kurkoski and Creativecow.net. All rights reserved.

Article Focus:
Paint strokes and masks in After Effects are both forms of vector paths, but converting one into the other isn't always straightfoward. This tutorial shows how to take strokes from the Vector Paint effect, as well as AE 6's new Paint tools, and turn them into masks.

Paint strokes and masks are both versatile tools in After Effects, but sometimes you may find you've used one when you should have used the other. Fortunately, both features are based on vector paths, although going from the Vector Paint effect to a mask isn't the most straightforward procedure. In this tutorial we'll walk through that process, and we'll also show how much easier it is to do with AE6's new Paint tools. Finally, we'll look at an alternate method of getting a mask from your paint strokes by using AE6's Auto-Trace feature. It's a little rougher, but the potential time savings are enormous.

Note: The project file download includes both AE 5.5 and 6.0 version files. For the first part of the tutorial, you must have AE 5.5 Production Bundle or AE 6.0 Professional. For the second section, you need AE 6.0 Standard or Pro. For the third part, AE 6.0 Pro is required.

Start by opening on the Paint_to_Mask.aep project file. Look at the Bessy Roto - Start comp. Here we have Ol' Bessy in front of her barn. We want to isolate just Bessy's head without the background or the text of the logo. I've started rotoscoping her out using the eraser brush in Vector Paint, but have very quickly realized that using a mask will be faster. Rather than start the mask from scratch, what we're going to do is take that paint stroke and convert it into a mask. Start by opening the effects for the Bessy layer and selecting the Vector Paint effect. The Vector Paint tools will appear on left hand side of the comp window.

1. In the Vector Paint tools, choose the selection tool (the arrow).
   
2. Select the paint stroke. It will become highlighted by showing a thin line inside of it.
   
   
   
3. Copy the paint stroke by using the keyboard shortcut (CTRL+C on Windows, Command+C on Mac). It is important to use the keyboard shortcut here instead of using the Edit > Copy menu command. Using the menu command would copy the whole Vector Paint effect instead of just the paint stroke.
   
4. Create an adjustment layer: Layer > New > Adjustment Layer
   
5. Select the adjustment layer and choose Edit > Paste, or CTRL/CMD+V (it's safe to use the keyboard shortcuts now that we're out of Vector Paint). Press U, and the Position and Scale properties of the adjustment layer will show the new keyframes.
   
   
   
6. Deselect the Position and Scale properties (F2 or click away from the layers).
   
7. Select the Position property. All of the position keyframes will become highlighted.
   
8. Edit > Copy or CMD/CTRL+C to copy the position keyframes.
   
9. Go back to the Bessy layer and draw a new mask. It can be any shape or size.
   
10. Press M on the keyboard, which will show the mask values for the Bessy layer. Select the Mask Shape property.
    
11. Edit > Paste or CTRL/CMD+V to paste the position keyframes into the mask.
    
    We should now see our mask, although because there are so many handles it kind of looks like a double mask. Zooming in gives you a better view.
    
    
12. Zooming in also reveals another problem: the mask is open. Go to the Layer menu > Mask > Closed (or context-click on the mask and choose Closed). Make sure to zoom back out to see the result: the background of the logo has been masked out.
    
    
    

We've accomplished our goal. The paint stroke has been transformed into a mask, and now Bessy is rotoscoped out of her logo. A little cleanup will make Bessy look prettier, though. Delete the adjustment layer and turn off the Vector Paint effect. I tweaked the mask feather and expansion properties to give Bessy a smoother edge. Then I added a drop shadow, animated the position and rotation properties, and gave Bessy a little dialogue. Preview the Bessy Roto - Final comp to see the results. Ta da!

If your project involves a lot of Vector Paint work, it's clear that converting all of the paint strokes to masks could be a time consuming process. AE only allows you to copy one paint stroke at a time. After Effects 6.0 has two new tools that can make the process a lot easier: the new Paint effect and the Auto-Trace command.

Turning paint strokes into masks is far less complicated when you start with AE6's new Paint effect. It's just a matter of copying the paint stroke shape to the mask shape. Note that I said "start with"- there's no easy way to convert Vector Paint strokes into the new Paint strokes, other than turning them into masks and copying those into Paint. So plan ahead. The new Paint tools are worth the time to learn.

Open the Paint Roto - Start comp. This is identical to the Bessy Roto comp, except that the roto painting has been done using the eraser tool of the new After Effects 6.0 Paint effect instead of Vector Paint.

1. Twirl down the Effects for the Bessy layer in the timeline window, then twirl down the Paint effect and open the Eraser 1 stroke.
   
2. Select the Shape property and choose Edit > Copy or CTRL/CMD+C.
   
   
   
3. Draw a new mask on the Bessy layer, any shape or size will do.
   
4. With the mask still selected, choose Edit > Paste or CTRL/CMD+V.
   
   Just like with Vector Paint, the mask appears doubled, and it also needs to be closed.
   
   
5. Go to the Layer menu and choose Mask > Closed (or context-click on the mask and choose Closed).
   
   

Just like with the Vector Paint instructions, a little cleanup work is desirable. Turn off the visibility of the Paint effect (or just delete it), and adjust the mask feather and expansion to clean up the edges. I also added some text and a couple of drop shadows. Preview the Paint Roto - Final comp for the result.

It's worth mentioning that not only can Paint strokes be copied into masks, but masks can be copied into Paint strokes (something that can't be done with Vector Paint). To do so, just reverse the process: copy the Mask Shape, draw a paint stroke, and paste into the stroke's Shape property. If you want to use the mask to roto-paint, make sure you draw your stroke with the eraser tool. (TIP: You can also convert any existing paint stroke to an eraser stroke by changing the color to 100% black and the Channels property to Alpha.)

Converting paint strokes into masks can be tiresome if you have a lot of strokes. What if you don't have the time or patience? There is a quick and dirty alternative to have AE6 make a mask out of paint strokes for you: the Auto-trace feature. The difference is that instead of using the vector path of the paint stroke, Auto-trace creates the mask from of the outline of your paint strokes.

Open the Speed Bessy - Start comp and preview it. Here's Bessy, happily zooming around the barnyard. I've added a Vector Paint stroke as a cartoon speed trail. It's kind of cheesy looking. With the help of Auto-Trace, let's turn it into a smoke trail instead.

1. Open the effects for the Bessy layer, and double-click on the Vector Paint effect. In the Effects Control window, change the Composite Paint property to "Only". This will isolate the paint strokes by themselves.
   
   
   
2. Go to the Layer menu and choose Auto-trace...
   
3. In the Auto-trace dialog, under Time Span choose "Work Area", choose "Alpha" for the Channel, and check the "Apply to new layer" option. Click OK.
   
   
   
4. A new white solid layer will be created, with a mask that follows the outline of our vector paint stroke. Rename the layer to "Smoke" and move it two layers down, so it is directly below the Bessy layer.
   
   
   
5. On the Bessy layer, turn off the visibility of the Vector Paint effect.
   
6. Select the Smoke layer and apply the Fractal Noise effect (Effects > Noise > Fractal Noise).
   
7. Open the controls for Mask 1 on the Smoke layer, and increase the Mask Feather property to give the smoke effect a softer edge. I animated the value with hold keyframes: 9 pixels on Bessy's first pass, 12 on the second, and 20 for Bessy's closest flyby.
   
   

Presto! Bessy is now burning rubber, er, pixels across the countryside. Preview the Speed Bessy - Final comp to see the results.

In this example, I started with the Vector Paint effect. If you were to start with AE6's new Paint effect instead, the process requires only a single adaptation: In step 1, open the Paint effect settings and switch the Paint on Transparent property to On.

Practical application of using Auto-trace with your own comps may vary in difficulty. Most likely you'll need to play with the tolerance and threshold values in the Auto-trace dialog to get an accurate mask, especially when dealing with airbrushed or feathered paint strokes.

We've covered a lot of detail about paint strokes and masks. Let's run through the important points one more time:

• To turn Vector Paint stroke into a mask: Copy the stroke and paste it onto another layer. Then copy the position keyframes of that layer into the Mask Shape property of a mask.
• To turn a stroke form AE6's Paint effect into a mask: Copy the paint stroke's Shape property and paste it into the Mask Shape property of a mask.
• To create a mask from paint strokes using the Auto-trace feature: Set the paint effect to "Only" or "Paint on Transparent", then choose Layer > Auto-trace.

Questions? Comments? Discuss this tutorial in the After Effects forum at CreativeCow.net.