How PatchMaker works?

The generic workflow in PatchMaker is as follows:

Select, by assigning a mask to a frame, an object, the motion of which is to be tracked. Define the tracking interval (segment) around this frame.
Run the tracking algorithm on frames of the given segment.
Apply the computed motion to warp the overlay or eliminate the movement of the underlay.

The computation step is mostly veiled from the user. In this section, we outline what is going on during this step.

Pairs of nearby frames in the segment are selected and sequentially processed. In each pair, one if the frames (a reference frame) has an already determined mask (drawn by the user or previously computed). The other frame is the target frame, for which the position of the masked object is to be computed.
The mask in the reference frame is split in blocks of a fixed size.
For each mask block its the best fit on the target frame is determined.
Displacements of individual blocks are integrated in the unified motion described by several parameters according to the selected motion model.

GENERATING reference frames

When tracking starts, the mask (determining the position of the object) is defined only in a single (key) frame, whichis to become the first reference frame. As the tracking proceeds an more frames get their data, the choice of reference frames becomes wider.

Generally speaking, we can do with a single reference frame—the key frame—and compare all other frames to it. This is recommended when the image of the tracked object does not change much within the tracking bounds. However, this is not always the case, especially when the aspect or the illumination in the scene changes. After several frames, the object image may change too much to be accurately recognized by the algorithm. For this reason, new reference frames need to be introduced to update the reference object image.

By decreasing the value of the Reference frame pitch property of the segment you make the algorithm more adaptable to change (but slower). The value of N means that PatchMaker skips N-1 frames before setting a new reference frame. Hence, N is the period of reference frames in the segment.

The maximum adaptivity is attained with the period of 1, when the motion is estimated between frames 1 and 2, then 2 and 3, 3 and 4, and so on. A small reference frame pitch not only slows down the computation but also causes a larger number of transformations to be composed to yield the final object position, making it more prone to errors.

Splitting the mask into blocks

The masked region in each reference frame is splitted into elemental square blocks. Their size is determined by the value of the Tracker size parameter:

Splitting the mask into blocks.
One of the blocks is denoted by '1'.

MATCHING elementaL blocks

Each block of the reference frame mask looks for the best match within a bounded region of the target frame (its size determined by the value of Matching radius parameter).

A block with a distinct internal color pattern is more likely to accurately find its match in the the target frame. The more such blocks in the mask, the higher the accuracy of the computed transformation. The blocks, however, should not be too small, as they might loose their "individuality".

Elemental block matching is the most time-consuming operation. To speed up tracking, the most likely position for each block is estimated based on its previous motion.This can be done for all but two frames adjacent to the key frame, where no previous motion exists. The maximum block match departure from its estimate is defined by the Max motion change parameter. Decreasing this parameter speeds up tracking.

integrating block displacements into a unified transformation

The output transformation parameters are computed from displacements of all elemental blocks. The motion thus found is accurate to within a pixel. This might suffice when it is fast (more than 34 pixels per frame), or not smooth (jerky) and, therefore, hardly-predictable.

To achieve a "subpixel" accuracy, the computed transformation parameters are refined by another, non block-based technique.

If the motion issmooth and slow, the refinement "subpixel" procedure might do the job alone. In this case, you can turn off the first block-matching step by setting the value of the Estimation method parameter to the Fastest.

If you the Estimation method is set to Fast, the first step is done only once, in passing from the key frame to its neighbours. After that, the refinement algorithm takes over and relies only on the available motion estimates. It works well with fast motion, but the smoothness is still required.

When the Estimation method is set to Normal (the default) or Intensive, both steps are performed.

By default, the transformation parameters are determined for the affine motion model. The affine model applies well to plane surface motion when the distance from the camera is large compared to object dimensions. You can adjust the Motion model parameter, to select another and more "conservative" model with a smaller number of parameters.