LIBRARY: Tutorials Reviews Interviews Editorials Features Business Authors RSS Feed

Turning Paint Into Masks in After Effects

Turning Paint Into Masks in After Effects
A Creative COW After Effects Tutorial




Timothy Kurkoski Timothy Kurkoski
Beaverton, Oregon, USA

©2003 Timothy Kurkoski and Creativecow.net. All rights reserved.

Article Focus:
Paint strokes and masks in After Effects are both forms of vector paths, but converting one into the other isn't always straightfoward. This tutorial shows how to take strokes from the Vector Paint effect, as well as AE 6's new Paint tools, and turn them into masks
.


Project File .zip Project File .sit

Paint strokes and masks are both versatile tools in After Effects, but sometimes you may find you've used one when you should have used the other. Fortunately, both features are based on vector paths, although going from the Vector Paint effect to a mask isn't the most straightforward procedure. In this tutorial we'll walk through that process, and we'll also show how much easier it is to do with AE6's new Paint tools. Finally, we'll look at an alternate method of getting a mask from your paint strokes by using AE6's Auto-Trace feature. It's a little rougher, but the potential time savings are enormous.

Note: The project file download includes both AE 5.5 and 6.0 version files. For the first part of the tutorial, you must have AE 5.5 Production Bundle or AE 6.0 Professional. For the second section, you need AE 6.0 Standard or Pro. For the third part, AE 6.0 Pro is required.


Part 1: Converting Vector Paint strokes into masks

Start by opening on the Paint_to_Mask.aep project file. Look at the Bessy Roto - Start comp. Here we have Ol' Bessy in front of her barn. We want to isolate just Bessy's head without the background or the text of the logo. I've started rotoscoping her out using the eraser brush in Vector Paint, but have very quickly realized that using a mask will be faster. Rather than start the mask from scratch, what we're going to do is take that paint stroke and convert it into a mask. Start by opening the effects for the Bessy layer and selecting the Vector Paint effect. The Vector Paint tools will appear on left hand side of the comp window.

  1. In the Vector Paint tools, choose the selection tool (the arrow).
  2. Select the paint stroke. It will become highlighted by showing a thin line inside of it.


  3. Copy the paint stroke by using the keyboard shortcut (CTRL+C on Windows, Command+C on Mac). It is important to use the keyboard shortcut here instead of using the Edit > Copy menu command. Using the menu command would copy the whole Vector Paint effect instead of just the paint stroke.
  4. Create an adjustment layer: Layer > New > Adjustment Layer
  5. Select the adjustment layer and choose Edit > Paste, or CTRL/CMD+V (it's safe to use the keyboard shortcuts now that we're out of Vector Paint). Press U, and the Position and Scale properties of the adjustment layer will show the new keyframes.


  6. Deselect the Position and Scale properties (F2 or click away from the layers).
  7. Select the Position property. All of the position keyframes will become highlighted.
  8. Edit > Copy or CMD/CTRL+C to copy the position keyframes.
  9. Go back to the Bessy layer and draw a new mask. It can be any shape or size.
  10. Press M on the keyboard, which will show the mask values for the Bessy layer. Select the Mask Shape property.
  11. Edit > Paste or CTRL/CMD+V to paste the position keyframes into the mask.

    We should now see our mask, although because there are so many handles it kind of looks like a double mask. Zooming in gives you a better view.

  12. Zooming in also reveals another problem: the mask is open. Go to the Layer menu > Mask > Closed (or context-click on the mask and choose Closed). Make sure to zoom back out to see the result: the background of the logo has been masked out.


We've accomplished our goal. The paint stroke has been transformed into a mask, and now Bessy is rotoscoped out of her logo. A little cleanup will make Bessy look prettier, though. Delete the adjustment layer and turn off the Vector Paint effect. I tweaked the mask feather and expansion properties to give Bessy a smoother edge. Then I added a drop shadow, animated the position and rotation properties, and gave Bessy a little dialogue. Preview the Bessy Roto - Final comp to see the results. Ta da!


If your project involves a lot of Vector Paint work, it's clear that converting all of the paint strokes to masks could be a time consuming process. AE only allows you to copy one paint stroke at a time. After Effects 6.0 has two new tools that can make the process a lot easier: the new Paint effect and the Auto-Trace command.


Part 2: Converting After Effects 6.0 Paint effect strokes into masks

Turning paint strokes into masks is far less complicated when you start with AE6's new Paint effect. It's just a matter of copying the paint stroke shape to the mask shape. Note that I said "start with"- there's no easy way to convert Vector Paint strokes into the new Paint strokes, other than turning them into masks and copying those into Paint. So plan ahead. The new Paint tools are worth the time to learn.

Open the Paint Roto - Start comp. This is identical to the Bessy Roto comp, except that the roto painting has been done using the eraser tool of the new After Effects 6.0 Paint effect instead of Vector Paint.

  1. Twirl down the Effects for the Bessy layer in the timeline window, then twirl down the Paint effect and open the Eraser 1 stroke.
  2. Select the Shape property and choose Edit > Copy or CTRL/CMD+C.


  3. Draw a new mask on the Bessy layer, any shape or size will do.
  4. With the mask still selected, choose Edit > Paste or CTRL/CMD+V.

    Just like with Vector Paint, the mask appears doubled, and it also needs to be closed.

  5. Go to the Layer menu and choose Mask > Closed (or context-click on the mask and choose Closed).

Just like with the Vector Paint instructions, a little cleanup work is desirable. Turn off the visibility of the Paint effect (or just delete it), and adjust the mask feather and expansion to clean up the edges. I also added some text and a couple of drop shadows. Preview the Paint Roto - Final comp for the result.

It's worth mentioning that not only can Paint strokes be copied into masks, but masks can be copied into Paint strokes (something that can't be done with Vector Paint). To do so, just reverse the process: copy the Mask Shape, draw a paint stroke, and paste into the stroke's Shape property. If you want to use the mask to roto-paint, make sure you draw your stroke with the eraser tool. (TIP: You can also convert any existing paint stroke to an eraser stroke by changing the color to 100% black and the Channels property to Alpha.)


Part 3: Using Auto-trace to turn paint strokes into masks

Converting paint strokes into masks can be tiresome if you have a lot of strokes. What if you don't have the time or patience? There is a quick and dirty alternative to have AE6 make a mask out of paint strokes for you: the Auto-trace feature. The difference is that instead of using the vector path of the paint stroke, Auto-trace creates the mask from of the outline of your paint strokes.

Open the Speed Bessy - Start comp and preview it. Here's Bessy, happily zooming around the barnyard. I've added a Vector Paint stroke as a cartoon speed trail. It's kind of cheesy looking. With the help of Auto-Trace, let's turn it into a smoke trail instead.

  1. Open the effects for the Bessy layer, and double-click on the Vector Paint effect. In the Effects Control window, change the Composite Paint property to "Only". This will isolate the paint strokes by themselves.


  2. Go to the Layer menu and choose Auto-trace...
  3. In the Auto-trace dialog, under Time Span choose "Work Area", choose "Alpha" for the Channel, and check the "Apply to new layer" option. Click OK.


  4. A new white solid layer will be created, with a mask that follows the outline of our vector paint stroke. Rename the layer to "Smoke" and move it two layers down, so it is directly below the Bessy layer.


  5. On the Bessy layer, turn off the visibility of the Vector Paint effect.
  6. Select the Smoke layer and apply the Fractal Noise effect (Effects > Noise > Fractal Noise).
  7. Open the controls for Mask 1 on the Smoke layer, and increase the Mask Feather property to give the smoke effect a softer edge. I animated the value with hold keyframes: 9 pixels on Bessy's first pass, 12 on the second, and 20 for Bessy's closest flyby.

Presto! Bessy is now burning rubber, er, pixels across the countryside. Preview the Speed Bessy - Final comp to see the results.

In this example, I started with the Vector Paint effect. If you were to start with AE6's new Paint effect instead, the process requires only a single adaptation: In step 1, open the Paint effect settings and switch the Paint on Transparent property to On.

Practical application of using Auto-trace with your own comps may vary in difficulty. Most likely you'll need to play with the tolerance and threshold values in the Auto-trace dialog to get an accurate mask, especially when dealing with airbrushed or feathered paint strokes.


Summary

We've covered a lot of detail about paint strokes and masks. Let's run through the important points one more time:

  • To turn Vector Paint stroke into a mask: Copy the stroke and paste it onto another layer. Then copy the position keyframes of that layer into the Mask Shape property of a mask.
  • To turn a stroke form AE6's Paint effect into a mask: Copy the paint stroke's Shape property and paste it into the Mask Shape property of a mask.
  • To create a mask from paint strokes using the Auto-trace feature: Set the paint effect to "Only" or "Paint on Transparent", then choose Layer > Auto-trace.


Questions? Comments? Discuss this tutorial in the After Effects forum at CreativeCow.net.


If you found this page from a direct link, please visit our forums or read other articles at CreativeCOW.net

Comments

Reply to Timk reply to Harry
by Harry Putnam
> Reply to Harry
> by Timothy Kurkoski on Nov 30, 2009
> Wow. I haven't looked at this tutorial in ages.

Thanks so much for the reply. And you hit it dead right about how I goofed it up. I skated right over that open path part thinking (and this is pretty dumb) that since I knew I had entered a closed path, it didn't apply to me. But of course when I entered the position data.... that made it an open path, and I'm still thinking about the path I entered... hoboy. So sorry to have missed the boat there.
Reply to Harry
by Timothy Kurkoski
Wow. I haven't looked at this tutorial in ages.

Harry, it sounds like you've missed step 12. When you paste the motion path (the position keyframes) onto the mask shape, it is an open path and can't exclude pixels. You need to close the mask in order to make the background disappear. In After Effects CS4, the name of the menu command has changed slightly; it's now Layer menu > Mask And Shape Path > Closed.

For what it's worth, Vector Paint is very outdated now and I'd concentrate on Part 2 of this tutorial for practical uses. But Part 1 is still useful in that it demonstrates that Position keyframes (and other X/Y dimension properties) can be turned into vector masks.
Backgroud did not disappear
by Harry Putnam
I hope I haven't just completely missed boat on this brief
Tutorial

Authored by Timothy Kurkoski

Just the first section kind of left things hanging far as I can tell.

The tutor says `ta da' as if something has happened but I don't see any difference other than my path is outlined in a heavy yellow line.

I still see the logo background... but apparently it is supposed to have disappeared. Can anyone tell me what is supposed to have happened there?

The tutorial then jumps to the finished product, where the logo background is now gone, but never really says how that happens.

=================================

In point 12 where the author says:

"We've accomplished our goal. The paint stroke has been transformed into a mask, and now Bessy is rotoscoped out of her logo. [...] "

I still see what is shown in point 11.

That is, the logo background is still there. Author says it is rototscoped out so I'm guessing I should not be seeing it?

I have followed all the points. I do have the mask... so when or how does the rotoscoping happen and logo background disappear?


Related Articles / Tutorials:
Adobe After Effects
Adobe After Effects Glitter Text - No Plug-ins Required!

Adobe After Effects Glitter Text - No Plug-ins Required!

Learn how to create a GLITTER TEXT EFFECT in Adobe After Effects without ANY plugins! VFX guru and filmmaker, Surfaced Studio's Tobias G, introduces you to CC Particle World, a powerful particle generator included in After Effects! You can use it to create all sorts of cool particle effects including explosions, smoke, fairy glitter, stars and much more. Along the way, Tobias brings the remarkable breadth of in-depth tips and tricks that will help you raise the level of your own AE expertise for every kind of creative task.


Tobias G
Adobe After Effects
After Effects Advanced Content-Aware Fill With Photoshop

After Effects Advanced Content-Aware Fill With Photoshop

In part three of his series on Adobe After Effects Content Aware Fill, filmmaker and After Effects artist Cody Pyper takes his deepest dive yet! Following requests from viewers in the series so far, Cody takes a closer look at how Adobe Photoshop can help you remove unwanted objects from your video footage in After Effects.


Cody Pyper
Adobe After Effects
Best Results with After Effects Content-Aware Fill: Reference Frames

Best Results with After Effects Content-Aware Fill: Reference Frames

Join filmmaker and After Effects artist Cody Pyper for a deep dive into how to get the absolute best results using the Content-Aware Fill tool in After Effects. Locked-down shots with simple backgrounds are one thing, but Cody shows the details of how to get fantastic results with complicated backgrounds and a moving camera using reference frames.


Cody Pyper
Adobe After Effects
Adobe After Effects Reverse Stabilization

Adobe After Effects Reverse Stabilization

You're going to be blown away by how you can power up your After Effects workflow with reverse stabilizing your footage! By separating your tracking from your compositing, you can focus on each step, and in addition, overcome the render order complexities when match moving elements and effects on a moving shot.


Roei Tzoref
Adobe After Effects
After Effects Content Aware Fill: When It Doesn't  Work

After Effects Content Aware Fill: When It Doesn't Work

There’s a new artificial intelligence-powered feature in Adobe After Effects called Content-Aware Fill that allows you to remove anything from your shots fairly easily! It's powerful, but if you’ve tried it you know that it doesn’t always work perfectly. So what do you do when it doesn’t work as well as you'd hoped? Filmmaker Cody Pyper is here to show what to try next!


Cody Pyper
Adobe After Effects
VFX in Adobe After Effects: An Artists Panel (NAB)

VFX in Adobe After Effects: An Artists Panel (NAB)

Join panelists Andrew Kramer of Video Copilot, Jayse Hansen, and Mary Poplin of Boris FX, along with moderator, Victoria Nece of Adobe, as they discuss the world of visual effects from an artist’s perspective.


Adobe Creative Cloud
Adobe After Effects
Scars, Tattoos & Digital Makeup: After Effects & Mocha Pro

Scars, Tattoos & Digital Makeup: After Effects & Mocha Pro

How do you add SCARS, TATTOOS or DIGITAL MAKEUP to a person's face? Learn how to use Mocha Pro's planar tracker and the Mesh Warp tool to insert a flat image/video on a (non flat) human face! Join VFX guru Tobias G from Surfaced Studio for a closer look!


Tobias G
Adobe After Effects
Compositing Secrets Everyone Can Use, Pt 5: Brightness

Compositing Secrets Everyone Can Use, Pt 5: Brightness

The first challenge to understanding the nature of brightness in compositing starts with remembering that we're not actually seeing color at all, but rather something of an illusion that appears to us as color! Join longtime VFX artist, editor, software developer, and business owner Simon Ubsdell for Part 5 of the best look behind the technology of compositing that you've ever seen, as he takes a look at the math behind brightness, and how to apply that to the compositing toolsets in your favorite editing, compositing, and color grading applications.

Tutorial
Simon Ubsdell
Adobe After Effects
60 Second Tutorial: Alien Invasion in Adobe After Effects

60 Second Tutorial: Alien Invasion in Adobe After Effects

Here's a fast, fun one! Sixty seconds to create and animate an alien using the 3D and Track Camera tools to animate your character in Adobe After Effects. Use the included files, or bring your own!

Tutorial
Adobe Creative Cloud
Adobe After Effects
What Are Adobe Motion Graphics Templates?

What Are Adobe Motion Graphics Templates?

A Motion Graphics Template, referred to as a MOGRT, is an animated sequence that is self-contained and can be used in Adobe Premiere Pro, Adobe Premiere Rush and Adobe After Effects, combining graphics, text, audio and video files, as well as vector or still images (including logos), to create a still or animation that can then be customized by the MOGRT user. The result is a dynamic creative tool that provides design freedom and is consistent to its users across apps and devices. Reuse, share, and even sell them!


Rod Harlan
MORE
© 2020 CreativeCOW.net All Rights Reserved
[TOP]