LIBRARY: Tutorials Reviews Interviews Editorials Features Business Authors RSS Feed

SynthEyes 3-D Camera Tracker

SynthEyes 3-D Camera Tracker
A CreativeCOW.net Product Review and Tutorial

Karl Holt gives an indepth look at SynthEyes 3-D Camera Tracker

Karl Holt
Karl Holt
Darkline Entertainment
UK

© Karl Holt and Creativecow.net. All rights are reserved.

Article Focus:
If you are unfamiliar with 3D camera tracking software, SynthEyes 3-D Camera Tracker is a tool which will analyze moving 2D video footage by tracking points of reference frame by frame. From this, it can reproduce an accurate 3D camera which matches the properties and movement of the real-world camera you used to shoot the sequence. In this review/tutorial, Karl Holt gives us an indepth look along with a lot of tips for using the software.



What does this do for us? Well once you have an accurate 3D camera track, you can bring this data into software such as 3DSmax and place 3D objects or characters within your filmed scene. You can completely reconstruct a scene, build 3D set extensions, insert 3D rain, or replace background footage and elements; all with moving footage. Ever wondered how they made the T-rex run through a moving camera shot in Jurassic Park? Or placed a CG ocean around a boat shot only against green screen? Or added thousands of troops running across the shores to attack the enemy? In short, it opens up a whole new level of visual effects possibilities that were previously limited to big production houses. Even if you do not own an application such as 3DSMax, this data can also be brought into compositing applications such as After Effects and Combustion where you can still use this camera information to achieve complex effects with 2D layers.

If you’ve heard of or used Boujou, Matchmover or PFtrack, you’ll know that the cost of this software has always been high. The current version of Boujou has a list price of $10,000. SynthEyes comes in a touch cheaper at $350. So what does it offer us for this price? And is it any good?


The rules of camera tracking.

Planning your shot, and understanding what can and cannot be tracked is essential before you even try to film a sequence. Waving the camera around blindly in a 360 degree circle and expecting SynthEyes to solve it, is not going to work. There are a few things you should consider :

Firstly there must be sufficient references or features within the scene. So filming a white room isn’t so clever (unless you have markers all over it which you can remove later in post). There needs to be significant depth and perspective change of these features in order to produce a good camera solution. There needs to be an abundance of features over the course of the whole shot, not just part of it; and they need to be as spread out as much as possible within the image to produce the most accurate track.

Where is the 3D object going to be? It helps enormously if you have features that SynthEyes can track around this point. Normally during an effects shot you will be following a 3D character in a scene or focussing on the inserted object. These areas need good tracking references around them to get ‘locked’ tracks. So always have a particular shot in mind when you shoot your footage and what the 3D component will be. It’s easy to overlook it but 3D space consists of 3 axis (X,Y,Z); so make sure you have good references spread out on the ground plane and some on a perpendicular plane.


Features

What is a feature? A feature is a consistent trackable point in your image sequence which exists for quite some time. You could choose the top of a pole, a leaf on the ground, an indicator on a car, the corner of a building, a mark on the wall. You get the idea. Try to pick out areas which are small, definable, and have a high contrast in relation to their immediate surroundings. Remember what you are trying to do is build a 3D approximation of your scene through single point markers. So what you should not choose are features that will change due to camera perspective. For example, an intersection where one object crosses another, a highlight which moves across glass, or an object that moves in the frame (a swaying branch). All of these will produce markers that do not actually represent any static locked property in your scene. For environments that do not contain many features it is common to add track markers or tennis balls to help with the tracking process. Of course these need to be cloned out later.


Tracking with SynthEyes

On opening SynthEyes I've chosen to import a 10 second sequence which I shot handheld in the forest. It guesses the setting correctly (PAL DV, 25fps, progressive, 250 frames).




You can customise the input here and SynthEyes can work at any frame rate and any resolution. I accept the default settings. Once it's imported, clicking play in the transport controls buffers the shot into RAM.


Click on the graphic above to view larger image


SynthEyes main work area contains 4 viewports, similar to that of a 3D application. Front, Left, Top and Camera. You can play the sequence by using the transport controls or by scrubbing in the time bar. At the top of the screen are access to several function panels. When active the left hand side of the screen changes to reflect the controls within that panel


Click on the graphic above to view larger image


SynthEyes has a 'Full Automatic' tracking mode which is available from the ‘summary’ panel. This is very useful for smooth shots, dolly/streadicam/crane etc. SynthEyes will try to track features itself, then take the best ones and produce a solution from this. It’s a one click process. However, you’ll find more often than not that automatic tracking just wont cut it for those hand held or slightly blurry movements you might throw at it. All it takes is for a few auto trackers to lose their references and you’re solution is down the pan. For the purpose of this tutorial (I couldn't cover everything SynthEyes is capable of) we will focus on solving a manually tracked, handheld, fixed lens DV shot. Why? Hand held is much harder to track, and you will learn all the basics of camera tracking and solving from it.

Although we are not going to go ‘Automatic’ in this tutorial, you can use automatic tracking to get a head start with your shot. There are two ways to do this; open the 'Features' panel, make sure ‘Link after blips’ is ticked and click the button 'Blips all frames'.




SynthEyes processes the whole shot and tracks hundreds of features.




Next click the 'Peel All' button. This takes all of the automatic tracked references and keeps only the best ones; turning them into tracker points which are used in the solution.




Unlike 'Full Automatic' it hasn't tried to produce a solution just yet. Alternatively in the summary panel you could have clicked 'Run Auto-tracker', this will do the same process as blipping and peeling, but in one-click. It’s helpful to do the former process the first time to understand what SynthEyes is doing. You don’t need to click ‘Run Auto-tracker’ if you have already blipped and peeled.




Expand the camera view to fill the screen by clicking 'Quad' in the drop down menu of the toolbar and changing this to ‘Camera’. Now go to the 'Tracker' panel, you will see diamonds in the viewport representing the automatic tracked points that SynthEyes has kept.



Click on the graphic above to view larger image


Scrub the timeline to see if these features stay locked to the object they are tracking. Although SynthEyes keeps the best tracks it does not always mean these are accurate as a feature may jump to a similar looking feature close to it by mistake. On a ground full of similar leaves this is likely to happen during fast movements. This is where it becomes obvious why we didn't use 'Full Automatic'. In a hand held shot some of the trackers will jump around; if you had calculated a solution with these trackers then you would be waiting a long time for a solution that just won’t work. Select the faulty trackers if there are any and delete them. Sometimes this is all you will need to do, you are ready to solve. In my shot SynthEyes has placed a lot of tracker points on the floor but none in the distance. Few are above the ground plane. This is unlikely to give a great solution at present.

Now I will add a more even distribution of tracker points to help SynthEyes produce an accurate track. Scrub through your sequence to find a suitable tracking point (remembering the rules described earlier). In the tracker panel click the 'create tracker' button (which looks like a wand) and then click over the point you wish to track in the viewport.



Once you do this the feature you are tracking will appear in the tracker window and 2 red boxes will appear in the viewport.

An important thing to remember here is to create the tracker at the point you want it to exist; scrub to check the timing of the feature before you create the tracker. This may sound obvious but it’s a common error to create a new tracker midway through the sequence when it is also in shot at the start too. If the reference only comes into shot at frame 60, then create it on this frame and track forwards. If the reference exists for the whole shot, then create the tracker on the first or last frame, tracking forward or backwards.

In my example I'm tracking a piece of blue tape which I put onto a tree trunk. This stays in my shot until frame 125, so this should be a good reference. Once I have placed the tracker on frame 0 all I have to do is click 'play' in the transport controls for tracking to begin. But first I change the tracker type to 'white spot on dark background'.



This will help it to track my marker easier. There are different options to use here which depends on the type of reference you are tracking. Some features will track better using 'symmetry' mode. So I continue to track my marker, and at frame 35 I lose my tracker. SynthEyes can not track this properly and the marker goes flying off the screen.



If SynthEyes loses it's tracking feature it displays these frames as a light blue coloured segment within the time bar; a feature matched correctly frame to frame is shown as dark blue.



One way to stop this from happening is to increase the size of the search area of the tracker. You will notice that the tracker has two squares. The inner square is the area you are tracking; the outer square is the search radius for the feature. With handheld shots simply increasing the range in the 'search' box is good enough to track a reference more accurately.



By how much depends on how shaky the camera movement is, so I increase my search area by 2 or 3 times in both directions. Then I scrub back to frame 30 or until you can see your reference in the tracker window again, and press play. Now my tracker is re-tracked from frame 30 onwards. It stays locked to my reference up to frame 125.

If increasing the search area still produces an incorrect track you can try to change the tracker type to another search type (symmetry instead of spot). You may have to place keyframes each time you lose the reference. Go back to the frame in which you lost the tracker and reposition it over the reference again. Be careful that when you reposition a tracker you do not create a new one by clicking in the viewport if you miss the edge of the tracker (it happens often!). I find it useful to click and drag in the tracker window, this will move the existing tracker and you know you haven’t created a new one accidentally. Alternatively you can click and drag the 'Pos' spinners to move the tracker.



From here, press play to track again, and press stop to reposition the tracker when SynthEyes loses it. In the time bar these keyframes can be seen as black markers. Keep repositioning and tracking as necessary.



A useful feature here is 'Preroll by Key Smooth'. This feature tries to smooth out the transition of your newly placed keyframe with the preceding good track points. Turning this on in the 'track' menu and then entering a value of 3 in the 'Key Smooth' box will mean that syntheyes will smooth the new keyframe out over the 3 previous frames.



Too much smoothing may produce inaccurate tracks, especially on a handheld shot. If you are accurate with your tracker placement you may not need this feature at all. However on smooth dolly/crane shots it may be beneficial to have this feature enabled to avoid a sudden glitch when you re-position the lost tracker.

I have a problem. After frame 125 my tracker goes behind another tree and the feature box loses its reference and flies around the screen again.



If this occurs, scrub back to the last frame at which the feature was locked. You can use the tracker window as a guide, or look at the where the segment has turned light blue in the time bar. Now push the 'enable/disable tracker button'.



This will delete the frames after this point so they are not taken into the solution. If a tracker is disabled on a frame then the tracker window does not display the content of the tracker and instead shows a pink colour. If at any point your reference comes back into shot you can scrub to that frame and re-enable the tracker by pressing the 'enable tracker' button again, or click and drag in the tracker window to activate it. Now press play to track from here onward. Do not create a new tracker for a feature that disappears and comes back into shot later; you will end up with 2 separate trackers which is not ideal. Enabling and disabling the tracker is the best way. Always look at the blue bar in the time controls to see which frames the tracker is active for, and to see if it is accurate on these frames.



Most bad solutions can be put down to a few rogue frames where a tracker has lost its feature. Scrubbing and looking in the viewport will show you if the tracker is indeed locked to the feature. Using the tracker window is not always the best way as it could jump to a similar looking feature which you won’t notice. Once your tracker is done push the 'lock tracker' button to prevent further tampering of this tracker.



This is important as SynthEyes can track multiple trackers at once; so if you do not lock it you may end up tracking it again incorrectly when you add a new tracker. When trackers are locked the 2 red boxes will disappear and be replaced as a box with a crosshair.



That’s the basic process of tracking features. There are other tools to help you track your features. You can track a feature backwards through a shot by first clicking the ‘track backwards’ button in the tracker panel.



And then reversing the direction of the play controls in the time bar.



Click on the graphic above to view larger image


Now when you press play the feature will track in reverse which can be useful for features which exist mid-way through a shot to the end. You can filter/smooth the results, and you can delete or create as many trackers as you like. SynthEyes recommends a minimum of 6 trackers present at a time for a shot. In my experience you should use as many as you need; the more you have evenly distributed the more accurate the solution will be. I normally run an automatic track, delete the bad trackers, and then add as many manual trackers as needed. Give SynthEyes references which are as close to, and as far away from the camera as possible. Also choose points that stay on screen for a long time and not just a few frames. Remember you will be able to see some great tracking references that SynthEyes will not see automatically.



Click on the graphic above to view larger image


Solving

Once you have enough trackers distributed through your shot and placed at different depths in your scene, you are ready to solve. This is pretty simple. Go to the solver panel and click the 'Go!' button.



SynthEyes can normally produce a solution in a few minutes. It is very fast. If it starts to take very long then it could be a sign that it cannot solve correctly. Look out for the RMS error in the optimization window, typically this is very low (0.003). If it's high then the chances are SynthEyes is going to report a bad solution. You'll need to go back and look for rogue trackers.



Click on the graphic above to view larger image


Once it is solved you will see yellow crosses appearing in your viewports. These are true 3D points within a 3D environment. Scrub the timeline. If they appear to be locked to your references then you have done a good job.



Look in the Solver panel at the text marked 'Error'. Ideally you're looking for the error to be between 1 and 2 hpix. This is the norm for a good 3D solution, but I have had decent solutions between 2 and 3 hpix also.




Remember it's never going to be perfect due to lenses, distortion and tracking accuracy of the features. Perhaps shooting 35mm with a whopping lens and flat tracking markers in your scene will produce a smaller error, but for most purposes this error is fine.

The only way really to check whether you have a good solution is to put some geometry in the scene. Before you do that you need to set-up a co-ordinate system. By that I mean orienting your scene so that the ground plane of the image sequence lines up with the ground plane you place your geometry on. There is a whole process in the SynthEyes manual about setting up a co-ordinate system which I won't go into here. In my experience I have had great success 'eyeballing' the co-ordinate system myself, and this is how you do it. Click the 3D object panel and change your viewports to 'quad'.




Here you can play the sequence again and view the path of the camera in any of the viewports. Hopefully it looks good.



Click on the graphic above to view larger image


In my view I have a cluster of trackers on the leafy ground. I can simply go into my ‘3D Objects’ panel and click the button 'whole'. Now clicking the 'rotate' or 'move' button I can select the camera in the viewports and rotate and move it until all the ground points line up with the ground plane (the 'whole' button makes sure the entire scene is rotated and not just the camera). The same process can be achieved with relatively few ground markers. The more spread out they are the more accurately you can line up the ground plane.



With a bit of tweaking and rotating in different viewports you should be able to set-up your co-ordinate system easily. Use your mouse wheel to zoom in and out of viewports to make sure your points are accurately lined up with the ground plane. This is another good reason to have tracker markers on your ground, it really helps you to set-up the co-ordinate system. It would be nice if you could select some tracking points and change them to a different colour, so they stood out from the cluster when you were rotating your camera and aligning; but this is a minor quibble. Next click on the 'create object' button and in the top viewport place whatever objects you like (boxes, cylinders, spheres, humans) in your scene.





Use the scale and transform tools to move and position the objects to the ground plane.



Switch back to camera view and play your shot.



You should see if the objects are locked to the ground. If there is some slight object sliding during playback you may find it's something simple like the object is just a few inches above the ground plane. As our scene is not a totally flat floor, some fine-tuning may be required. This is where eyeballing the co-ordinate system is helpful as you can raise, lower or rotate the scene to fine-tune the position of the ground plane. Equally the object may be sitting on a raised part of the floor which would cause object sliding. This is another reason to have tracking points not just on the ground, but at the points where you want to insert the 3D objects. You can easily line up the 3D object to sit on top of the tracker.



Click on the graphic above to view larger image


If you have a more serious error like a spike or judder in the camera movement; the chances are some trackers are not correctly locked and are producing the error. Check the camera path in one of the viewports, if there is a spike in the path then there are probably loose trackers on that frame, or not enough trackers. There are lots of options to help you spot these spikes, such as camera and tracker graphs.

It may be the case that you just need more trackers. The 'Tracker Bars' panel is very useful for seeing an overview of all the trackers and where problems might lie. It shows how many trackers exist at a certain point in time, and whether you need more. The areas where more trackers are needed are highlighted by a yellow vertical bar. Manual or supervised trackers are indicated by dark blue bars for correctly tracked references, and light blue for where it has lost that reference. It's easy to pick out the problematic trackers. Automatic trackers are represented by a series of black arrows.



Click on the graphic above to view larger image


From this information it’s clear that you need more trackers from frame 34 to 61. Also some of the trackers have light blue bars where they may have lost their reference and need to be checked. The good thing about SynthEyes is you can reposition, re-track, delete or add trackers at any stage in the process. You may need to give SynthEyes more depth perspective, or more tracker points around the inserted 3D object. Go back to the 'Tracker panel' to add more trackers and when you are done come back to the 'solver panel' and click 'Refine' from the drop down menu and 'Go!'.




If the Hpix error goes down, then you have increased the accuracy of the solution. Sometimes a glitch can be because of too many incorrectly tracked markers. If you spot a rogue tracker, simply click on it and delete it, and then you can ‘Refine’ the solution once again. When you are ready you can export this solution to a variety of 2D and 3D programs. You can click on trackers and name them in the tracker panel, which is very helpful especially if you want to match up 3D elements with a specific object in your shot.




I am naming the trackers around the base of a cluster of trees so I can locate that tracker in my 3D application, and insert geometry accurately. Basic objects you create in SynthEyes such as Boxes, cylinders and spheres will carry over into 3DSMax as well as the naming of any trackers.


Click on the graphic above to view larger image



Conclusion

I have not covered many of the features within SynthEyes. It can track camera zooms (something which even Boujou could not do until the latest version). You can track moving objects in your scene (lets say you had a moving bus you wanted to track separately and add a CG character on top of it), tripod shots, roto splines for removing elements from an automatic track, lens distortion compensation, multi-shot tracking, it can work at any resolution and frame rate and it can even work out the position of the sun in the scene from the shadows.

On top of this technical support is nothing short of excellent. Before I even purchased the product I e-mailed a barrage of questions to Mr Andersson. He answered all of them and helped me over the next few days to get to grips with this software very quickly; and I hadn't even purchased it. The only down side I can think of are the minimal tutorials in the online help, they only cover basics of the software, leaving you to read the rest of the manual like a glossary.

Having not used any of the other tracking software for a long time, I cannot really give it a fair comparison to the latest versions from the competitors. The last time I used Boujou was version 1 and it didn't solve anything I threw at it. However, what I can say is that SynthEyes does everything I want it to, and more. It's simple to use, very fast, very intuitive, and lets you solve the most complex of shots with flexibility and ease. For $350 this software is practically free when you take into consideration what it can do. What's great about it is that it's affordable enough for people who are curious about this kind of work to get into it, and it’s certainly robust enough for a production environment too. It has been used on many shots for big budget feature films with great success where other camera trackers failed. I can now offer my clients a level of work previously dedicated to the larger production houses. I really don’t know what you get out of Boujou for an extra $9,650 but I won’t be splashing out the cash to find out.

I give SynthEyes 3-D Camera Tracker 4.5 COWs.





SynthEyes


Please visit our forums at Creativecow.net if you found this page from a direct link.



Related Articles / Tutorials:
Adobe After Effects
Imagineer mocha Pro 5 Plug-In for Adobe: An In Depth Review

Imagineer mocha Pro 5 Plug-In for Adobe: An In Depth Review

Imagineer mocha Pro 5 Plug-in for Adobe brings all the amazing features of the professional version of the mocha Planar Tracker directly into After Effects and Premiere Pro in the form of a plugin. In this in-depth review, After Effects tutorial guru Tobias Gleissenberger of Surfaced Studio will show you what you can do with this new plug-in, and discuss what he likes and doesn't like about the new update.

Tutorial
Tobias Gleissenberger
Adobe After Effects
After Effects 2015.3 - My Favorite Features

After Effects 2015.3 - My Favorite Features

Learn why you should upgrade to After Effects CC 2015.3 - 13.8.1 - a close and detailed look at the latest release of After Effects (August 2016). Roei Tzoref will be focusing on his favorite features that set this release apart from previous versions: Performance, Queue in AME, Lumetri Color new features, and more.

Tutorial
Roei Tzoref
Adobe After Effects
Advanced Masking in Adobe After Effects

Advanced Masking in Adobe After Effects

Some of the coolest stuff you can do inside of Adobe After Effects is only possible once you unlock the power of masks. Join After Effects whiz Tobias Gleissenberger of Surfaced Studio to learn about mask animation and interpolation, using the variable width feathering tool, managing mask modes and ordering, and more.

Tutorial
Tobias Gleissenberger
Adobe After Effects
Fixing Common After Effects Problems and Mistakes

Fixing Common After Effects Problems and Mistakes

Got problems using Adobe After Effects? Exported files too large, Expressions not working, mixing shapes and makes, modes/switches, selecting previews for layers vs. comps - Surfaced Studio's Tobias Gleissenberger shows you fast fixes for these and more!

Tutorial
Tobias Gleissenberger
Adobe After Effects
mocha AE Planar Tracker for Absolute Beginners

mocha AE Planar Tracker for Absolute Beginners

Want to learn how to create advanced visual effects? Learn how to use mocha AE to track your shot and add advanced visual effects to live action footage inside Adobe After Effects. mocha can help you track shots that would be hard to track using traditional 2D point or feature trackers because it is a PLANAR TRACKER. A planar tracker uses planes and textures to track as opposed to points or groups of pixels. This allows the tracker to stay on track even if your shot contains motion blur or a very shallow depth of field. mocha AE comes included with Adobe After Effects and is available since CS3 and there is no reason for you not to use this awesome tool to make it easier for you to track your shot, replace screens or rotoscope!

Tutorial
Tobias Gleissenberger
Adobe After Effects
How to Make After Effects Faster with Proxies

How to Make After Effects Faster with Proxies

Learn how to make Adobe After Effects faster by using PROXIES! A proxy is a placeholder that stands in place for a very large video file or image sequence in your project. You can easily create proxies for the large source files that you are using and After Effects will automatically link them to the item in your project panel.

Tutorial
Tobias Gleissenberger
Adobe After Effects
Stabilize & Smooth: mocha 5 Plug-in for Adobe & Avid

Stabilize & Smooth: mocha 5 Plug-in for Adobe & Avid

Imagineer Systems and Boris FX product specialist Mary Poplin shows you how to stabilize with the new mocha Pro 5 plug-in inside of Avid Media Composer, Adobe Premiere Pro and After Effects. This tutorial covers artistic stabilization, such as smoothing out camera movements or stabilizing around moving objects, completely locking down shots, and automatically replacing edge fill on planar backgrounds.

Tutorial
Mary Poplin
Adobe After Effects
Adobe After Effects Puppet Tool

Adobe After Effects Puppet Tool

Become a puppet master by learning how to use the Puppet Tool in Adobe After Effects! This intermediate-level tutorial from After Effects guru Tobias will show you how the Puppet Tool allows you to add joints and animations to bring life to any static image!

Tutorial
Tobias Gleissenberger
Adobe After Effects
How to Spawn A Clone in Adobe After Effects

How to Spawn A Clone in Adobe After Effects

Want to learn how to create a cool clone spawn effect in Adobe After Effects? Follow along with After Effects whiz Tobias from Surfaced Studio in this exciting new visual effects tutorial that combines green screen using Keylight, CC Vector Blur, the Liquefy Effect, CC Particle World, and much more, delivered in Tobias' inimitable style!

Tutorial
Tobias Gleissenberger
Adobe After Effects
Creating A Flame on Your Finger with After Effects

Creating A Flame on Your Finger with After Effects

It is easy to do some motion tracking and attach a basic stock footage element of fire onto your hands, but there is a little bit of work involved if you actually want to make it look good. In this intermediate tutorial by After Effects expert Tobias, you'll see how to use a fire stock footage element to set your thumb on fire! There are lots of useful tricks for null objects, expressions, and more in this tutorial that will help you create all sorts of other cool visual effects -- or set even more things on fire!

Tutorial
Tobias Gleissenberger
MORE
© 2016 CreativeCOW.net All Rights Reserved
[TOP]