How AI puts you in a video: motion transfer and character replacement explained

Tap a button, wait a few minutes, and there you are, doing a dance you have never done in a room you have never been in. It feels like magic, but the process is surprisingly logical once you see the steps. Here is what happens inside an app like SynthGen AI.

Step 1: Split the video into motion and appearance

Every person in a video is two things at once: how they move and what they look like. The first job is to pull those apart.

• Body motion is read as a skeleton: the position of the head, shoulders, elbows, hands, hips, knees and feet in every frame.
• Facial motion is read separately: where the eyes look, how the mouth opens, how the head tilts. Modern models such as Wan-Animate use features learned from the face rather than a fixed set of points, which keeps expressions natural.
• Appearance, meaning the original person’s face, hair and clothes, is thrown away. That part will come from your photo.

What is left is a moving stick figure with expressions: pure motion, no identity.

Step 2: Read who you are from one photo

Your photo is encoded into a description the model understands: face shape, skin tone, hair, eyes, and, where visible, body shape and outfit. One good photo is enough for current models because they have learned from enormous amounts of video what people look like from other angles. This is also why a sharp, front facing photo matters so much: the model can only be as accurate as what it is given.

Step 3: Generate the new video

Now a video diffusion model draws the result. Diffusion models start from noise and refine it, step by step, into frames that match their instructions. Here the instructions are: this person (your photo), in these poses (the skeleton), with these expressions (the face signal), over time.

Generating frames together, instead of one at a time, is what keeps the result smooth. Older approaches drew each frame separately, and the face flickered from frame to frame. Video models keep your identity stable across the whole clip.

Two ways to use the same idea

This is where SynthGen AI’s two modes come from.

Character replacement (Swap me in)

The model keeps the original background and draws you into it, in place of the original person. The hard part is making you look like you were really there, so models add a relighting step: Wan-Animate’s authors describe a dedicated relighting module that adjusts your lighting and color tone to match the scene. SynthGen AI uses Wan 2.2 Animate in replacement mode for this.

Motion transfer (Copy the moves)

The model takes your photo as the scene, background included, and animates you with the extracted motion. Nothing of the original video’s world remains except the moves and the sound. SynthGen AI uses Kling motion control for this, which follows one character through one continuous shot of 3 to 30 seconds.

Why the rules about clips exist

Each limitation you hear about comes straight from the steps above:

• Cuts break the motion. A hard cut makes the skeleton jump to a different place or person, so motion models stop at the cut.
• Several people confuse the tracker. The model follows one skeleton, and has to pick which.
• Limbs out of frame get invented. No skeleton points means no motion to copy, so the model guesses.
• Covered faces look less like you. Missing information in your photo is filled in from what is typical, not from you.

Our guide to picking the right video and photo turns these into a checklist.

Why it takes minutes

A 15 second clip at 30 frames per second is 450 frames. Each one goes through a model with billions of parameters, many times over, on a data center GPU. SynthGen AI sends the job to Replicate, which runs the models, and notifies you when the result is back. A typical video takes 3 to 6 minutes.

What it means for you

You do not need to know any of this to use the app. But it explains the two things that matter most for a great result: a clean clip, so the motion is easy to read, and a clear photo, so your appearance is easy to read. Get those right and the model does the rest. Get SynthGen AI on the App Store.