Skip to main content
QwenCloud home page
Docs
Video gen & edits

Lip-sync replacement (VideoRetalk)

Replace the lip movements in a video so that they match an audio file

videoretalk generates a new video in which the speaker's lip movements match an input audio file, based on a person video. Use it to dub a video, correct mismatched audio, or translate speech into another language while keeping the original performance. Quick links: API reference: Create a task, Query the result Supported regions: China North 2 (Beijing) only. Use an API key created in that region and the https://cn-beijing.maas.qwencloudapi.com endpoint. See Regions and endpoints.

Input requirements

InputRequirements
VideoMP4, AVI, or MOV; up to 300 MB; 2-120 seconds; 15-60 fps; H.264 or H.265; side length 640-2,048 pixels
Video contentClose-up, front-facing person. Avoid extreme angles or very small faces.
AudioWAV, MP3, or AAC; up to 30 MB; 2-120 seconds; clear, loud human voice without ambient noise or background music
Reference image (optional)JPEG, JPG, PNG, BMP, or WebP; up to 10 MB; aspect ratio no more than 2; longest side no more than 4,096 pixels; clear frontal face of a person in the video
URLFiles must be publicly accessible over HTTP or HTTPS. Local paths are not supported.

Billing

ItemValue
Unit price$0.011469 per second of generated video
Task submission rate1 request per second
Concurrent tasks1 (additional tasks are queued)
You are charged only for successful tasks.

Request example

import os
import time
import requests

# China North 2 (Beijing). This model is available only in this region.
# API keys are region-specific. See https://docs.qwencloud.com/developer-guides/getting-started/regions
BASE_URL = "https://cn-beijing.maas.qwencloudapi.com/api/v1"
headers = {
  "Authorization": f"Bearer {os.getenv('DASHSCOPE_API_KEY')}",
  "Content-Type": "application/json",
  "X-DashScope-Async": "enable",
}

response = requests.post(
  f"{BASE_URL}/services/aigc/image2video/video-synthesis",
  headers=headers,
  json={
    "model": "videoretalk",
    "input": {
      "video_url": "https://example.com/video.mp4",
      "audio_url": "https://example.com/audio.wav"
    },
    "parameters": {"video_extension": False}
  },
)
task_id = response.json()["output"]["task_id"]

while True:
  result = requests.get(f"{BASE_URL}/tasks/{task_id}", headers=headers).json()
  status = result["output"]["task_status"]
  if status in ("SUCCEEDED", "FAILED"):
    break
  time.sleep(10)

print(result)

Parameters

ParameterTypeRequiredDescription
input.video_urlstringYesPublic URL of the person video.
input.audio_urlstringYesPublic URL of the audio that drives the lip movements.
input.ref_image_urlstringNoPublic URL of a reference face image. Use it to choose which face to sync when the video contains multiple people. If omitted, the largest face in the first frame is used.
parameters.video_extensionbooleanNoExtends the video when the audio is longer. true loops the video in reverse-forward playback until it matches the audio length. false truncates the audio. Default: false.
parameters.query_face_thresholdintegerNoFace matching confidence when a reference image is provided. Range: 120-200. Lower values match more loosely. Default: 170.

FAQ

What happens when the video and audio have different durations? By default, the longer file is truncated to match the shorter one. Set video_extension to true to loop the video until it matches the audio length. How are silent segments in the audio handled? The model generates closed-mouth frames for silent audio segments. What happens when a frame contains no face but the audio has speech? The original frame is preserved and the audio continues. Lip sync applies only to frames with a detectable face. How do I choose a person in a video with multiple people? The model syncs one person only. Set input.ref_image_url to the target face. If omitted, the largest face in the first frame is synced. Can I test this model in the console? No. VideoRetalk is pay-as-you-go and requires API calls.