Skip to main content
Speech-to-speech

Real-time audio and video translation - Qwen

This topic introduces Qwen real-time speech and audiovisual translation, including model capabilities, supported models, and integration. The models use audio and image input to translate in real time and output text or speech in the target language for voice communication and video translation.

Try an online demo with one-click deployment using Function Compute .

Features

  • Multi-language support: Translates between 60 languages — 29 with audio and text output, 31 with text-only output — including Chinese, English, French, German, Russian, Japanese, Korean, Spanish, Portuguese, and Arabic.
  • Visual enhancement: Analyzes visual cues, such as lip movements, gestures, and on-screen text, to improve translation accuracy, especially in noisy environments or for ambiguous words.
  • 2.3-second latency: Delivers simultaneous interpretation with latency as low as 2.3 seconds.
  • Real-time speaker diarization: Distinguishes different speakers and their speech when multiple people take turns speaking, so listeners can clearly understand who said what.
  • Lossless simultaneous interpretation: Predicts semantic units to resolve cross-language word order differences, achieving quality comparable to offline translation.
  • Natural voice: Matches the intonation and emotion of the source audio automatically.
  • Hotword configuration: Configurable hotwords improve translation accuracy for specific terms.
  • Voice cloning: Clones the speaker's voice for translated output. Supports real-time cloning and pre-cloned voice profiles.

Procedure

1. Configure the connection

Set model to a supported model when connecting. The connection example below uses qwen3.8-livetranslate-flash-realtime. When connecting over WebSocket, use the following parameters:
AOQ, WebRTC, and WebSocket access are supported. For client-side integration that prioritizes stable latency, interaction under poor network conditions, and noise suppression and echo cancellation for real-time full-duplex communication, consider AOQ first. For protocol comparison and selection, see the API reference.
ParameterDescription
endpointSingapore: wss://ap-southeast-1.maas.qwencloudapi.com/api-ws/v1/realtime. For China North 2 (Beijing), replace ap-southeast-1 with cn-beijing. API keys are region-specific. See Regions.
query parameterThe model query parameter must be set to the model name. Example: ?model=qwen3.8-livetranslate-flash-realtime
message headerUse a Bearer Token for authentication: Authorization: Bearer $DASHSCOPE_API_KEY > DASHSCOPE_API_KEY is your API key from QwenCloud.
Sample connection code (Python):
# pip install websocket-client
import json
import websocket
import os

API_KEY=os.getenv("DASHSCOPE_API_KEY")
# Singapore (ap-southeast-1). For another region, replace ap-southeast-1 in the URL.
# API keys are region-specific. See https://docs.qwencloud.com/developer-guides/getting-started/regions
API_URL = "wss://ap-southeast-1.maas.qwencloudapi.com/api-ws/v1/realtime?model=qwen3.8-livetranslate-flash-realtime"

headers = [
    "Authorization: Bearer " + API_KEY
]

def on_open(ws):
    print(f"Connected to server: {API_URL}")
def on_message(ws, message):
    data = json.loads(message)
    print("Received event:", json.dumps(data, indent=2))
def on_error(ws, error):
    print("Error:", error)

ws = websocket.WebSocketApp(
    API_URL,
    header=headers,
    on_open=on_open,
    on_message=on_message,
    on_error=on_error
)

ws.run_forever()

2. Configure language, modality, and voice

Configure the session through session.update:
  • Target language: Set session.translation.language, such as en for English. With qwen3.8-livetranslate-flash-realtime, the source language is detected automatically.
With qwen3.5-livetranslate-flash-realtime, set session.input_audio_transcription.language to specify the source language, or omit it for automatic detection.
  • Output modalities: With qwen3.8-livetranslate-flash-realtime, set session.output_modalities to ["text"] for text only or ["text", "audio"] for text and audio (default).
With qwen3.5-livetranslate-flash-realtime, use session.modalities. The supported values and default are the same.
  • Source transcription: With qwen3.8-livetranslate-flash-realtime, source transcription is always enabled and cannot be disabled. The LiveTranslate model itself generates the source transcript. Relevant information in images can help resolve ambiguities in the source speech. Receive increments through conversation.item.input_audio_transcription.delta and complete transcripts through conversation.item.input_audio_transcription.completed.
With qwen3.5-livetranslate-flash-realtime, source transcription is enabled by default. A separate ASR model transcribes the input audio. Set session.input_audio_transcription.model to qwen3-asr-flash-realtime to enable it or null to disable it. Receive increments through conversation.item.input_audio_transcription.text and complete transcripts through conversation.item.input_audio_transcription.completed.
  • Audio and voice: With qwen3.8-livetranslate-flash-realtime, defaults are 16000 Hz PCM input, 24000 Hz PCM output, and voice Tina. You can also send Opus audio by setting session.audio.input.format.type to opus. Check session.audio.output.voice in session.created for the voice. To preserve the speaker’s voice, see Voice cloning.
With qwen3.5-livetranslate-flash-realtime, configure audio through session.sample_rate, session.input_audio_format, and session.output_audio_format, and the voice through session.voice. The default voice is also Tina.
  • Hotwords: Set session.translation.corpus.phrases to specify preferred translations. See Configure hotwords.
We recommend using no more than 1,000 hotwords.

3. Input audio and images

Send Base64-encoded audio and image data using the input_audio_buffer.append and input_image_buffer.append events. Audio input is required; image input is optional.
Images can be from a local file or captured in real time from a video stream.
With qwen3.8-livetranslate-flash-realtime, the default is session.audio.input.turn_detection.type = speaker_detection. Send audio continuously; the server detects speech boundaries and generates translation responses without a manual commit for each utterance. See Identify speakers for configuration and speaker identifiers.
How the model determines that an utterance is complete depends on the VAD mode or Manual mode configured via the turn_detection parameter:
  • VAD mode (default): The client continuously sends input_audio_buffer.append events. When the server detects speech start/end, it returns input_audio_buffer.speech_started and input_audio_buffer.speech_stopped events respectively, automatically commits the audio buffer, and triggers translation. Translation responses are generated synchronously with the streaming audio and typically begin during audio input, without waiting for the speech to end.
  • Manual mode: Set session.turn_detection to null. After the client finishes sending a complete utterance, it sends an input_audio_buffer.commit event to commit the audio buffer. After the server returns an input_audio_buffer.committed event to confirm, it automatically starts generating the translation response; the client does not need to send any other event to trigger the response. To clear uncommitted audio before committing, send an input_audio_buffer.clear event.

4. Receive the model response

With qwen3.8-livetranslate-flash-realtime, handle responses according to the output modalities:
  • Text only: Append the delta values from response.text.delta to the subtitles in order, without overwriting text already displayed.
  • Text and audio: Concatenate delta from response.audio_transcript.delta for text, and Base64-decode delta from response.audio.delta to obtain audio chunks.
response.done indicates the end of a response. See Server events for event fields.
  • Output source language recognition results Set session.input_audio_transcription.model to qwen3-asr-flash-realtime. The server then returns both the translation and the speech recognition result (original text) for the input audio. The server returns these events:
    • conversation.item.input_audio_transcription.text: Streams the recognition results.
    • conversation.item.input_audio_transcription.completed: Returns the final result after the recognition is complete.
    • conversation.item.input_audio_transcription.failed: Returns error information when recognition fails.
Translation responses are generated synchronously with the streaming audio and typically do not require waiting for speech to end (see the VAD/Manual mode description in the previous section). The response format depends on the output modality.
  • Text-only output The server streams incremental translated text (including confirmed text and tentative predicted text) through response.text.text events; upon completion, the full translated text is returned in a response.text.done event. stash contains provisional translation text. It lets subtitles appear sooner, but later events may rewrite it. Depending on your application's needs, display text and stash together in the client, or display only the confirmed text. Use item_id to update the matching subtitle, then show the complete translation when response.text.done arrives.
  • Text and audio output
qwen3.5-livetranslate-flash-realtime uses the response.text.text event for incremental text delivery, which differs from the response.text.delta event used by Omni (full-duplex voice conversation) models. These events have different field structures and semantics — do not use them interchangeably.

5. End the session

After sending all audio, send a session.finish event, then wait for the server to return a session.finished event before closing the WebSocket connection. If you close the WebSocket without sending session.finish, the server's VAD cannot detect the end of the final speech segment. This causes translation results for that segment to be lost entirely, and the connection may hang indefinitely. Always send this event before disconnecting.

Supported models

ModelVersionContext windowMax inputMax output
(tokens)
qwen3.8-livetranslate-flash-realtimeStable53248491524096
qwen3.5-livetranslate-flash-realtime
Alias for qwen3.5-livetranslate-flash-realtime-2026-05-19
Stable53248491524096
qwen3.5-livetranslate-flash-realtime-2026-05-19Snapshot

Getting started

Translate microphone input in real time

  1. Use Python 3.10 or later and install pyaudio:
  • macOS
  • Debian/Ubuntu
  • CentOS
  • Windows
   brew install portaudio && pip install pyaudio
Install the WebSocket dependency: pip install websocket-client.
  1. Set this environment variable. The API key must belong to the region in the connection URL:
  1. Save the code below as livetranslate_client.py, run python livetranslate_client.py, and speak into the microphone.
The example uses qwen3.8-livetranslate-flash-realtime to translate microphone input into English, print the text, and play the translated speech in real time. Input is mono, 16-bit, 16000 Hz PCM; playback is mono, 16-bit, 24000 Hz PCM. Press Ctrl+C to stop recording. The example sends session.finish, receives and plays the remaining results, and closes the connection after session.finished. Use headphones to keep translated speech from being recorded again.
import base64
import json
import os
import signal
import threading
import time

import pyaudio
import websocket

model = "qwen3.8-livetranslate-flash-realtime"
api_key = os.environ["DASHSCOPE_API_KEY"]
url = (
    # Singapore (ap-southeast-1). For another region, replace ap-southeast-1 in the URL.
    # API keys are region-specific. See https://docs.qwencloud.com/developer-guides/getting-started/regions
    f"wss://ap-southeast-1.maas.qwencloudapi.com"
    f"/api-ws/v1/realtime?model={model}"
)
stop_recording = False
send_errors = []
ws = None
audio = None
input_stream = None
output_stream = None
sender = None
previous_sigint_handler = None


def receive():
    event = json.loads(ws.recv())
    if event.get("type") == "error" or event.get("code"):
        raise RuntimeError(event)
    return event


def stop_input(signum, frame):
    global stop_recording
    stop_recording = True


def send_audio():
    try:
        while not stop_recording:
            chunk = input_stream.read(1600)
            ws.send(json.dumps({
                "type": "input_audio_buffer.append",
                "audio": base64.b64encode(chunk).decode("ascii"),
            }))
    except Exception as error:
        send_errors.append(error)
    finally:
        try:
            ws.send(json.dumps({"type": "session.finish"}))
        except Exception as error:
            send_errors.append(error)
            ws.close()


def receive_translation():
    while True:
        event = receive()
        event_type = event.get("type")
        if event_type in ("response.text.delta", "response.audio_transcript.delta"):
            print(event["delta"], end="", flush=True)
        elif event_type == "response.audio.delta":
            output_stream.write(base64.b64decode(event["delta"]))
        elif event_type == "response.done":
            if event["response"]["status"] != "completed":
                raise RuntimeError(event["response"])
            print()
        elif event_type == "session.finished":
            return


try:
    ws = websocket.create_connection(
        url, header={"Authorization": f"Bearer {api_key}"}, timeout=30
    )
    while receive().get("type") != "session.created":
        pass
    ws.send(json.dumps({
        "type": "session.update",
        "session": {
            "output_modalities": ["text", "audio"],
            "translation": {"language": "en"},
        },
    }))
    while receive().get("type") != "session.updated":
        pass
    audio = pyaudio.PyAudio()
    input_stream = audio.open(
        format=pyaudio.paInt16, channels=1, rate=16000,
        input=True, frames_per_buffer=1600,
    )
    output_stream = audio.open(
        format=pyaudio.paInt16, channels=1, rate=24000, output=True,
    )
    print("Start speaking. Press Ctrl+C to stop recording and wait for the remaining translation.")
    previous_sigint_handler = signal.signal(signal.SIGINT, stop_input)
    sender = threading.Thread(target=send_audio, daemon=True)
    sender.start()
    receive_translation()
    stop_recording = True
    sender.join()
    if send_errors:
        raise send_errors[0]
except Exception:
    if send_errors:
        raise send_errors[0]
    raise
finally:
    stop_recording = True
    if ws is not None:
        ws.close()
    if sender is not None:
        sender.join()
    for stream in (input_stream, output_stream):
        if stream is not None:
            stream.stop_stream()
            stream.close()
    if audio is not None:
        audio.terminate()
    if previous_sigint_handler is not None:
        signal.signal(signal.SIGINT, previous_sigint_handler)

Configure hotwords

Adding hotwords can improve translation accuracy when you need a fixed translation or need to distinguish between homophones. For example:
  • Fixed wording: Alibaba Cloud's slogan, “计算,为了无法计算的价值。”, has the agreed English wording “Computing, for the value beyond computation.”. The wording and word order need to be preserved.
  • Brand names that sound alike: Qoder sounds like the common word coder. When the speaker means the product Qoder, the translation needs to retain the brand's spelling.
The following table compares the outputs of two tests. Each pair uses the same audio and differs only in its hotword configuration.
Content of the input audioWithout hotwordsWith hotwords
计算,为了无法计算的价值。Calculate. For values that cannot be calculated.Computing, for the value beyond computation.
Qoder 是面向真实工作的 Agentic 平台。Coder is an agentic system designed for real-world work. platform.Qoder, is an agentic platform designed for real-world work.

Configuration

Set hotwords through session.translation.corpus.phrases, using key-value pairs to map source-language words or phrases to their preferred translations. In the first session.update request from Getting started, merge the following translation object into the existing session. Keep the output modality, voice, and other settings for the session.
{
  "event_id": "configure-hotwords",
  "type": "session.update",
  "session": {
    "translation": {
      "language": "en",
      "corpus": {
        "phrases": {
          "计算,为了无法计算的价值。": "Computing, for the value beyond computation.",
          "Qoder": "Qoder"
        }
      }
    }
  }
}
Configure the session before sending audio. Wait for the server's session.updated event before sending input_audio_buffer.append events.

Identify speakers

qwen3.8-livetranslate-flash-realtime can identify different speakers and label them with speaker_id. Associate this ID with the source text and translation to display subtitles by speaker. Keep the default speaker_detection, or configure it explicitly in session.update. threshold distinguishes silence from valid speech. Sounds below the threshold are filtered out before they reach the model. The threshold is fixed at 0.5.
{"type": "session.update", "session": {"audio": {"input": {"turn_detection": {"type": "speaker_detection", "threshold": 0.5}}}}}
Associate the events as follows:
  1. Record the speaker_id and source item_id from input_audio_buffer.speech_started.
  2. Read the source text for that item_id from conversation.item.input_audio_transcription.delta.
  3. In the translation's conversation.item.created, associate item.id with the source ID in the event's top-level previous_item_id.
  4. Use each translation increment's item_id to find its source text and speaker, then update the subtitle.
Place these variables and functions before def receive_translation(): in the quick start microphone example. Within that function, call handle_speaker_event(event) immediately after event = receive(), and replace the original print(event["delta"], end="", flush=True) with pass so that the snippet handles all subtitle output. The snippet prints the speaker ID, source text, and translation as they change, so you can display subtitles by speaker. Keep the existing audio and session-completion handling.
speaker_by_source = {}
source_by_translation = {}
source_text = {}
translation_text = {}


def show_subtitle(translation_id):
    source_id = source_by_translation.get(translation_id)
    if source_id not in speaker_by_source or translation_id not in translation_text:
        return
    print({
        "speaker_id": speaker_by_source[source_id],
        "source": source_text.get(source_id, ""),
        "translation": translation_text[translation_id],
    })


def handle_speaker_event(event):
    event_type = event.get("type")
    if event_type in ("response.text.delta", "response.audio_transcript.delta"):
        item_id = event["item_id"]
        translation_text[item_id] = translation_text.get(item_id, "") + event["delta"]
        show_subtitle(item_id)
        return
    if event_type == "conversation.item.created":
        item = event["item"]
        if any(part.get("type") == "input_audio" for part in item.get("content", [])):
            return
        source_id = event.get("previous_item_id")
        if item.get("role") == "assistant" and source_id is not None:
            source_by_translation[item["id"]] = source_id
            show_subtitle(item["id"])
        return
    if event_type == "input_audio_buffer.speech_started" and "speaker_id" in event:
        source_id = event["item_id"]
        speaker_by_source[source_id] = event["speaker_id"]
    elif event_type == "conversation.item.input_audio_transcription.delta":
        source_id = event["item_id"]
        source_text[source_id] = source_text.get(source_id, "") + event["delta"]
    elif event_type == "conversation.item.input_audio_transcription.completed":
        source_id = event["item_id"]
        source_text[source_id] = event["transcript"]
    else:
        return
    for translation_id, mapped_source in source_by_translation.items():
        if mapped_source == source_id:
            show_subtitle(translation_id)
See Speaker IDs and Source/translation association for field definitions.

Improve translation with images

Qwen real-time translation models support image input to help disambiguate homonyms and recognize uncommon proper nouns during translation. Send no more than 2 images per second. Download the following sample images: medical mask.png, masquerade mask.png Reuse the quick start's microphone input and connection settings:
  1. Install Pillow: pip install Pillow.
  2. Set translation.language to zh for Chinese output, then say “What is mask?” into the microphone when you run the example.
  3. Place the image-loading code below before def send_audio(): in the microphone example, and replace the original send_audio() with this version. It uses the existing input_stream, stop_recording, ws, send_errors, and imports.
import io
from PIL import Image

IMAGE_PATH = "medical-mask.png"
# IMAGE_PATH = "masquerade-mask.png"
with Image.open(IMAGE_PATH) as image:
    buffer = io.BytesIO()
    image.convert("RGB").save(buffer, format="JPEG")
image_bytes = buffer.getvalue()
if len(image_bytes) > 500_000:
    raise ValueError("JPEG > 500 KB")
image_b64 = base64.b64encode(image_bytes).decode("ascii")


def send_audio():
    try:
        last_image_time = None
        while not stop_recording:
            chunk = input_stream.read(1600)
            ws.send(json.dumps({
                "type": "input_audio_buffer.append",
                "audio": base64.b64encode(chunk).decode("ascii"),
            }))
            now = time.monotonic()
            if last_image_time is None or now - last_image_time >= 0.5:
                ws.send(json.dumps({
                    "type": "input_image_buffer.append",
                    "image": image_b64,
                }))
                last_image_time = now
    except Exception as error:
        send_errors.append(error)
    finally:
        try:
            ws.send(json.dumps({"type": "session.finish"}))
        except Exception as error:
            send_errors.append(error)
            ws.close()
Audio and images are interleaved, with at most one image every 0.5 seconds. Try each image and check whether “mask” is translated as the Chinese word for the corresponding object. After you press Ctrl+C to stop recording, the example requests session completion and receives the remaining results.

Voice cloning

The model clones the speaker's voice from input audio and uses it for translated output. Use a pre-cloned voice profile or clone in real time during translation. Useful for conference interpreting, live streaming, and video dubbing. Real-time cloning: Both qwen3.8-livetranslate-flash-realtime and qwen3.5-livetranslate-flash-realtime clone voices from input audio during translation without an additional cloning fee. Model calls remain billed by token usage. Create a voice in advance: The Voice Cloning API charges $0.01 per voice; failed creation is not billed. See that billing section for free-quota regions, limits, and validity. Translation using the created voice is billed by model token usage. Set the following parameters in session.update to enable voice cloning:
  • session.enable_voice_clone: Set to true to enable voice cloning.
  • session.voice_clone_options.frequency: Controls when voice cloning occurs. Accepted values:
    • never: Use a previously created voice without extracting a new voice during this session. Set session.voice to your cloned voice ID.
    • once: Extract the voice once from input audio at the start of the session and reuse it for subsequent translations in this session. “Once” describes extraction, not a single audio output. Suitable for a single-speaker presentation; set session.voice to default.
    • always: Extract the voice before each translated-audio response, so output follows speaker changes. Suitable for multiple speakers; set session.voice to default.
  • session.voice: Specifies the output voice. The value depends on the frequency setting:
    • Set to default: Use with frequency set to once or always. Clones the speaker's voice from the input audio. A default voice is used until cloning completes.
    • Set to a custom cloned voice ID (for example, qwen-translate-vc-xxx-yyy-zzz): Use with frequency set to never. Prepare the voice in advance through the Voice cloning API, with target_model set to the translation model you use.
When frequency is set to once or always , the voice parameter must be set to default . Any other value causes the server to return an error.

Voice cloning configuration examples

With qwen3.8-livetranslate-flash-realtime, replace the session.update configuration in its Quick start example with the configuration for your scenario.
With qwen3.5-livetranslate-flash-realtime, replace session.output_modalities with session.modalities in the examples. The other cloning parameters are the same. Specify this model in the connection’s model parameter and handle its response events as described in Receive model responses.
Pre-cloned voice profile (for scenarios that require a fixed voice):
{
    "type": "session.update",
    "session": {
        "output_modalities": ["text","audio"],
        "voice": "qwen-translate-vc-xxx-yyy-zzz",
        "translation": {
            "language": "en"
        },
        "enable_voice_clone": true,
        "voice_clone_options": {
            "frequency": "never"
        }
    }
}
Reuse one voice in the current session (single-speaker presentations):
{
    "type": "session.update",
    "session": {
        "output_modalities": ["text","audio"],
        "voice": "default",
        "translation": {
            "language": "en"
        },
        "enable_voice_clone": true,
        "voice_clone_options": {
            "frequency": "once"
        }
    }
}
Follow the current speaker’s voice (multiple speakers):
{
    "type": "session.update",
    "session": {
        "output_modalities": ["text","audio"],
        "voice": "default",
        "translation": {
            "language": "en"
        },
        "enable_voice_clone": true,
        "voice_clone_options": {
            "frequency": "always"
        }
    }
}

One-click Function Compute deployment

To deploy the application:
  1. Open the Function Compute template, enter your API key, and click Create and Deploy Default Environment to test the application.
  2. Wait for about a minute. In Environment Details > Environment Context, retrieve the endpoint, change the protocol from http to https (for example, https://qwen-livetranslate-flash-realtime-intl.fcv3.xxx.ap-southeast-1.fc.devsapp.net/), and open the URL in a browser to interact with the model.
This endpoint uses a self-signed certificate and is for temporary testing only. Your browser will display a security warning on your first visit. This is expected behavior. Do not use this endpoint in a production environment. To proceed, follow the on-screen instructions (for example, click Advanced → Proceed to (unsafe)).
If you are prompted to configure Resource Access Management permissions, follow the on-screen instructions.
To view the project source code, go to Resource Information > Function Resources .
Both Function Compute and QwenCloud provide a free quota for new users, sufficient for basic debugging. After the free quota is used up, pay-as-you-go billing applies.

Interaction flow

With qwen3.8-livetranslate-flash-realtime, the server detects turns and generates responses by default. The following table summarizes the main interaction stages. See Server events for event fields.
StageClient actionServer events
Create and configure the sessionConnect and send session.updatesession.created, session.updated
Send audioinput_audio_buffer.appendSource transcription is streamed through conversation.item.input_audio_transcription.delta, followed by conversation.item.input_audio_transcription.completed.
Receive translation and audioKeep receiving server eventsTranslation is returned through response.text.delta (text only) or response.audio_transcript.delta (text and audio). Audio is returned through response.audio.delta. response.done marks the end of a response.
End the sessionsession.finishClose the connection after receiving session.finished.
Translation uses an event-driven WebSocket model. How speech boundaries are determined depends on VAD mode or Manual mode (see 3. Input audio and images). The table below is based on VAD mode (default) and annotates the different server events in Manual mode.
LifecycleClient eventServer event
Session initializationsession.update > Session configurationsession.created > Session created session.updated > Session configuration updated
User audio inputinput_audio_buffer.append > Append audio to the buffer input_image_buffer.append > Append image to the buffer input_audio_buffer.commit > (Manual mode only) Commit the audio bufferVAD mode: input_audio_buffer.speech_started > Speech start detected input_audio_buffer.speech_stopped > Speech end detected; server automatically commits the audio buffer Manual mode: input_audio_buffer.committed > Returned after the client sends input_audio_buffer.commit, confirming the audio buffer has been committed
Server audio outputNoneresponse.created > Signals that the server starts generating a response. response.output_item.added > Signals that a new output item is available. conversation.item.created > A new message item is created in the conversation. response.content_part.added > Signals that a new content part has been added to the assistant message. response.text.text > Incremental translated text in text-only modality response.audio_transcript.text > Incremental translated text in audio+text modality response.audio.delta > Contains an incremental chunk of the synthesized audio. response.text.done > Translation text complete in text-only modality response.audio_transcript.done > Translation text complete in audio+text modality response.audio.done > Signals that the synthesized audio is complete. response.content_part.done > Signals that a text or audio content part for the assistant message is complete. response.output_item.done > Signals that the entire output item for the assistant message is complete. response.done > Signals that the entire response is complete.
Session terminationsession.finish > Notifies the server that audio input is completesession.finished > Server processing complete; session ended
After sending all audio, send a session.finish event and wait for session.finished before closing the WebSocket. If you close the connection without sending session.finish, the server cannot know that audio input has ended, and the recognition and translation results for the last speech segment will be lost.

API

For the AOQ access flow and examples, see AOQ access. For protocol selection and reference links, see the API reference.

Billing

Qwen3.8-LiveTranslate-Flash-Realtime and Qwen3.5-LiveTranslate-Flash-Realtime
  • Audio: 7 tokens per second of input audio; 12.5 tokens per second of output audio.
  • Image: Every 32×32 pixels consumes 0.5 tokens. For token prices for each model, see Model pricing and billing.
Real-time cloning: Both qwen3.8-livetranslate-flash-realtime and qwen3.5-livetranslate-flash-realtime clone voices from input audio during translation without an additional cloning fee. Model calls remain billed by token usage. Create a voice in advance: The Voice Cloning API charges $0.01 per voice; failed creation is not billed. See that billing section for free-quota regions, limits, and validity. Translation using the created voice is billed by model token usage.

Rate limits

For information about model rate limits, see Rate limiting.

Supported languages

Use the following language codes to specify the source and target languages.
Some target languages only support text.
Language codeLanguageOutput
zhChineseAudio + text
enEnglishAudio + text
arArabicAudio + text
deGermanAudio + text
frFrenchAudio + text
esSpanishAudio + text
ptPortugueseAudio + text
idIndonesianAudio + text
itItalianAudio + text
koKoreanAudio + text
ruRussianAudio + text
thThaiAudio + text
viVietnameseAudio + text
jaJapaneseAudio + text
trTurkishAudio + text
hiHindiAudio + text
msMalayAudio + text
nlDutchAudio + text
urUrduAudio + text
nbNorwegian BokmålAudio + text
svSwedishAudio + text
daDanishAudio + text
heHebrewAudio + text
fiFinnishAudio + text
plPolishAudio + text
isIcelandicAudio + text
csCzechAudio + text
filFilipinoAudio + text
faPersianAudio + text
yueCantoneseText
elGreekText
afAfrikaansText
astAsturianText
beBelarusianText
bgBulgarianText
bnBengaliText
bsBosnianText
caCatalanText
cebCebuanoText
etEstonianText
glGalicianText
guGujaratiText
hrCroatianText
huHungarianText
jvJavaneseText
kkKazakhText
knKannadaText
kyKyrgyzText
lvLatvianText
mkMacedonianText
mlMalayalamText
mrMarathiText
paPunjabiText
roRomanianText
skSlovakText
slSlovenianText
swSwahiliText
tgTajikText
azAzerbaijaniText
ukUkrainianText

Supported voices

For supported voices and the corresponding voice parameter values, see Voice list.

FAQ

Is there a charge while the connection is open but nobody speaks?

Keeping a connection open is not billed. If you send silent audio, whether it counts as audio input depends on the mode you use:
  • Server-side speech detection: Filtered silence does not count as audio input.
  • Manual mode: Silence you explicitly commit counts as audio input.
The audio input token count is returned in response.done, under response → usage → input_tokens_details → audio_tokens. The model is billed by input and output tokens. See Billing for prices and calculation rules.

How do I migrate from Gummy to LiveTranslate?

gummy-realtime-v1 and qwen3.8-livetranslate-flash-realtime use different protocols. Besides changing the model name, update the code that opens the connection, sends audio, receives translations, and ends the session. For English captions with the Python SDK, change four calls. Reuse your 16 kHz mono 16-bit PCM recording.

Step

Gummy

LiveTranslate

Connect and configure

TranslationRecognizerRealtime(..., translation_target_languages=["en"])

start()

client.audio.realtime.connect(model="qwen3.8-livetranslate-flash-realtime")

session.update_session(output_modalities=["text"], translation_params={"language": "en"})

Wait for session.updated before sending audio.

Send audio

send_audio_frame(data)

session.append_audio(data)

Display captions

on_event(..., translation_result, ...)

translation_result.get_translation("en").text

session.events()

response.text.delta

Read events while sending audio. Append each delta to the caption.

End the session

stop()

session.end_session()

Read the last caption. Close after session.finished.

Set DASHSCOPE_API_KEY to your QwenCloud API key. Save the code as subtitles.py and the recording as input.wav. Run:
python -m pip install dashscope==2.0.2
python subtitles.py input.wav
import os
import sys
import time
import wave
from concurrent.futures import ThreadPoolExecutor

import dashscope


def send_audio(session, recording):
    while chunk := recording.readframes(1600):
        session.append_audio(chunk)
        time.sleep(len(chunk) / 32000)
    session.end_session()


def translate(path):
    # Singapore (ap-southeast-1). For another region, replace ap-southeast-1 in the URL.
    # API keys are region-specific. See https://docs.qwencloud.com/developer-guides/getting-started/regions
    client = dashscope.Client(api_key=os.environ["DASHSCOPE_API_KEY"],
                              region="ap-southeast-1",
                              base_url="https://ap-southeast-1.maas.qwencloudapi.com/api/v1")
    with wave.open(path, "rb") as recording:
        if (recording.getframerate(), recording.getnchannels(), recording.getsampwidth()) != (16000, 1, 2):
            raise ValueError("Expected a 16 kHz mono 16-bit PCM WAV file")
        with client.audio.realtime.connect(model="qwen3.8-livetranslate-flash-realtime", timeout=30) as session:
            events = session.events()
            for event in events:
                if event["type"] == "session.created":
                    session.update_session(output_modalities=["text"], translation_params={"language": "en"})
                elif event["type"] == "session.updated":
                    break
                elif event["type"] == "error":
                    raise RuntimeError(event["error"])
            else:
                raise RuntimeError("Session closed before configuration completed")

            with ThreadPoolExecutor(max_workers=1) as pool:
                sender = pool.submit(send_audio, session, recording)
                try:
                    for event in events:
                        kind = event["type"]
                        if kind == "response.text.delta":
                            print(event["delta"], end="", flush=True)
                        elif kind == "response.text.done":
                            print(flush=True)
                        elif kind == "response.done" and event["response"]["status"] != "completed":
                            raise RuntimeError(event["response"].get("status_details"))
                        elif kind == "error":
                            raise RuntimeError(event["error"])
                        elif kind == "session.finished":
                            break
                    else:
                        raise RuntimeError("Session closed before session.finished")
                except BaseException:
                    session.close()
                    raise
                sender.result()


if __name__ == "__main__":
    translate(sys.argv[1])
Direct WebSocket connections
  1. Endpoint Change /api-ws/v1/inference to /api-ws/v1/realtime?model=qwen3.8-livetranslate-flash-realtime.
  2. Audio Replace binary frames with Base64 PCM in the audio field of each input_audio_buffer.append event.
  3. Captions Replace result-generated with response.text.delta. Append each delta to the caption.
After sending the audio, send session.finish. Wait for session.finished before closing. Full example

What does RPM limit? Is there a limit on the total number of simultaneous connections?

RPM limits the number of WebSocket connection requests per minute. Responses on an existing connection do not count toward RPM. Reconnecting counts as a new request. There is currently no separate limit on the total number of connections open at the same time. If a WebSocket closes during setup with code 1007 and reason Requests rate limit exceeded, please try again later., the request rate limit was reached. Reduce the connection rate and reconnect with exponential backoff.

How do I resume translation after a disconnection?

If you receive no semantic result after merged audio exceeded 60s and the WebSocket connection closes, create a new session to resume translation:
  1. Stop sending audio to the old connection.
  2. Reconnect and send session.update to restore the target language, output modalities, and other settings.
  3. Wait for session.updated, then resume sending audio.
If a parameter error occurs and the connection remains open, correct the request and continue using that connection. When using dashscope.Client(), also resume event reception.
Real-time audio and video translation - Qwen - QwenCloud