Matchmoving by Hand

Today we’re going to take a quick look at matchmoving objects – but not by tracking multiple features and solving for the camera and object. Basically, not the normal way you would expect. This is how to hack and cheat matchmoving things that have no markers.

3D matchmoving is an absolutely essential aspect of visual effects . Ideally you’ll have some tracking markers and camera information, but if not, don’t fear – visual effects is all about cheating! And you can absolutely cheat an object matchmove. I’ve used this technique for years on TV shows like Gotham, Pennyworth, Agents of S.H.I.E.L.D., and Swamp Thing.

Here’s a couple examples of what we’re doing. These two videos each have slightly different effect elements added, but are both based off the same tracks.

For a real world example, in the show Agents of S.H.I.E.L.D., a character named Hellfire can charge things with energy and turn them into bombs. (I did not work on the shot below. Notice even his fingers have energy in them.)

https://www.tumblr.com/agentsofshield/143466579830/fireball-anyone

You can also see it starting at 1:30 in this scene, and at about 17 seconds into this shot. And below is a still image I found online from one of the shots I worked on where Hellfire charges up a martini shaker.

More often than not, I don’t do anything to the default camera when I use this technique (other than roughly lining up the ground plane). However, you could be a bit more accurate by first using fSpy to calculate the camera position and focal length, and then using the manual techniques from there.

I created the electricity bolts for the bottle in Natron, and am including the Natron project in the Projects download if you want to take a look. I’d be happy to make a quick tutorial on it if anyone’s interested, just leave a comment below or message me HERE. (Edit: I went ahead and made the tutorial, it’s Part 4 below)

I’ve also demonstrated this technique a couple times during World Blender Meetup Day. In 2017 I did a quick walk-through of another Agents of S.H.I.E.L.D. shot, which you can see HERE, and then in 2019 I used the same footage I shot for this tutorial, which you can see HERE.

One of the most powerful things you can do with simple face tracking is blemish clean up and de-aging (making an actor appear younger or older). You could also add things like heat ripple, other kinds of particles, or cool sci-fi stuff like subdermal glowing veins or a HUD like in Iron Man. If your manual track is good enough, you could add makeup fx like the Joker, or even remove chunks of the person! Although, keep in mind that if you do try a digital makeup effect, manual tracking is not ideal for that. It’s more useful for things that don’t need to stick 100%.

Another thing I use this technique for a lot is gunshot woulds to the head, complete with chunks and blood mist blowing out. But that’s a tutorial for the future (I have big plans for that one!)

In the commercial software world, Nuke has a great plugin called GeoTracker that does an amazing job of automating this process. I also know of a tool in Syntheyes called GeoH that does the same thing, although I have never used that tool. There’s also a pretty cool tool called Character Face Gen. None of these options are free and open source, however, so hopefully this tutorial can fill the gap.

Actor – Stephan Singh

Assets required:

Pre-rendered elements:

Tutorial 1 – Modeling & matchmoving the canteen.

Tutorial 2 – Matchmoving the face

Tutorial 3 – Adding the cool elements!

Tutorial 4 – Creating the electricity element in Natron

28 comments

    1. Thanks, Igor! Head over to the Contact Me page and send me a message telling me some details about what this Natron fire/ice tutorial is that you’d like to see!

      Reply

  5. Good to see you back, Sean. Great stuff. That sphere idea – so simple and yet I needed that in a past project where I had things spinning. One effect, that I’d be curious to see how you approach it, is where there is something crawling under the skin, you know, the alien creature inside the body thing!

    Reply

    1. Ah, yes, that’s a popular effect. I remember seeing Andrew Kramer do it over on Videocopilot years ago, and honestly, I’m not sure I’d do it any differently than he does there. It’s definitely something I’ve done at work a couple times over the years!

      Reply

    1. Hmmm, Youtube should let you adjust the quality to whatever you like. Is it not letting you do that? They should be full HD.

      Reply

      1. thanks for your fast reply.
        At least in my browsers (Chrome/Firefox), I can choose up to 1080p but it only works on max 360p. After that there is only a spinning wheel on black background.
        But never mind. If you say its in full HD I will try it on another pc in a few days.
        Maybe youtube doesnt like some of my adblockers.

        The tutorials are great, even though I cant see all of the details, your discription is very good.
        I hope to see some more of them in the next time 🙂

      2. Yeah, very strange. For me here on my laptop, it loads up by default at 720p. Keep me posted if you find that keeps happening on another computer. Glad you like the tutorials!

    1. Thanks for checking out the tutorial! And yes, I do think Natron could have a future, if there was a person or group interested in picking it up and continuing development. 3D compositing is not necessary for a compositing program, even though compositors are coming to rely on it more and more. There’s a ton of other things I use Natron for that can’t be done yet in other open source software.

      Reply

  9. Hi Sean, I’m learning Blender at the moment, and I want to learn Rotomation and matchmoving for a personal project I’m working on, which is to colorize black and white footage using detailed (but not hyperrealistic) 3D Models, which I can overlay the colors from over the original footage. I’m concerned about the process as I am animating over black and white footage, and since camera tracking wont work with low quality B&W footage, I want to see if it can be done manually, and be easily able to animate geometry and models over the objects in the original footage.

    I’m very pleased with your tutorial, and while I lament that Keentools’ Facebuilder plugin for Blender is now no longer free, I can’t thank you enough for this tutorial, as it has helped me immensely.

    Will it be also possible for you to make some more tutorials of this process about how to animate full bodies, buildings, sky, etc?

    Reply

    1. Wow, interesting that you want to colorize footage using 3d animated things rather than the standard technique of just rotoscoping. I’m very curious as to why you’re going that way instead of roto. Also, you mention camera tracking not working with low quality B&W footage. Camera tracking definitely works on B&W footage, but I’m not sure how low quality your footage is that camera tracking wouldn’t help out on some shots.

      Glad you liked the tutorial, I hope it helps you on your project! Yeah, Facebuilder was never intending to be a free product, it only had a free trial period. Keep in mind it’s very easy to create a head without using Facebuilder. Grab a decent head model from the internet (like maybe Blendswap) and just using soft selection, you should be able to push parts of the head around until it matches photos. You’ll have to switch out the photos you’re using as reference a few times, but it’s still totally possible, and even easy. 🙂

      As for animating full bodies, buildings, etc, well, I probably won’t be doing a tutorial on a full body, although, the process would be pretty much identical to what I show in these tutorials. I would build each body part separately and manually track each part separately. It would take a long time, but it’s perfectly possible. Also, not sure how to hand matchmove a building because buildings don’t really move. Perhaps you mean hand animating a matchmove? This is also very possible, but honestly, it’s not very accurate at all. I would only use it for vfx that don’t need to be touching the ground or a wall or something. For example, it would work great for smoke or particle effects, but not something like tracking a matte painting in. Hope that makes sense.

      Reply

      1. Hi Sean

        1. The reason why I am choosing to colorize footage using CGI and 3D Geometry is merely for a project that I am working on. I already have a fairly effective process of keyframe interpolation using a code called Deepremaster. However, it doesn’t really work well for extreme movements leading me to create more keyframes, so I am using Matchmoving techniques to compensate for where I possibly would’ve had to rotoscope by animating thousands of polygons over objects in the original footage.

        This brings me to the reason why I chose Rotomation and Matchmoving instead of simple rotoscoping. The reason for this is because unlike Rotoscoping where you have to animate dosens of masks in order to get a more realistic detail, Rotomation and Matchmoving may inevitably compromise by allowing me to animate simple models which while not intended to be hyperrealistic, I intend for them to have basic detail that may add to the realistic image such as subtleties on the skin tones.

        2. While I am admittedly disappointed that the process of animating a full body will not inevitably be covered in the near future, I am slightly reassured about the prospect of just simply animating separate body parts, though it is inevitably possible they might go out of sync.

        3. As for the Black and White footage, the footage is about 480p, so hopefully you’ll understand where I am coming from with this

        4. I do think I might practice morphing the head model to better match photos and try to recreate a photo from scratch as part of the practice.

        5. As for whether or not the manual matchmoving process may work well for objects like buildings in a tracking shot, perhaps there is always the possibility of resizing and rotating the object frame by frame to better match the video’s movements

      2. Very interesting! I’m going to check out Deepremaster, sounds very interesting. As for the 480p, it should still be trackable. For years before HD, that’s all us micro budget indie filmmakers had, so I got lots of practice matchmoving 480p footage back then. 🙂 But yeah, you certainly may run into problems now and again. And maybe I’ll add a rough hand-camera-tracking tutorial to my list of topics.

      4. It’s also very fascinating to hear from your experience as an indie filmmaker having to matchmove on poor quality 480p footage, and I do find the fact that sometimes 480p footage can be trackable quite reassuring.

        Hand Camera Tracking also sounds very interesting too

Leave a Reply to SeanCancel reply