This topic introduces Qwen real-time speech and audiovisual translation, including model capabilities, supported models, and integration. The models use audio and image input to translate in real time and output text or speech in the target language for voice communication and video translation.
Try an online demo with one-click deployment using Function Compute .
Features
- Multi-language support: Translates between 60 languages — 29 with audio and text output, 31 with text-only output — including Chinese, English, French, German, Russian, Japanese, Korean, Spanish, Portuguese, and Arabic.
- Visual enhancement: Analyzes visual cues, such as lip movements, gestures, and on-screen text, to improve translation accuracy, especially in noisy environments or for ambiguous words.
- 2.3-second latency: Delivers simultaneous interpretation with latency as low as 2.3 seconds.
- Real-time speaker diarization: Distinguishes different speakers and their speech when multiple people take turns speaking, so listeners can clearly understand who said what.
- Lossless simultaneous interpretation: Predicts semantic units to resolve cross-language word order differences, achieving quality comparable to offline translation.
- Natural voice: Matches the intonation and emotion of the source audio automatically.
- Hotword configuration: Configurable hotwords improve translation accuracy for specific terms.
- Voice cloning: Clones the speaker's voice for translated output. Supports real-time cloning and pre-cloned voice profiles.
Procedure
1. Configure the connection
Set model to a supported model when connecting. The connection example below uses qwen3.8-livetranslate-flash-realtime.
When connecting over WebSocket, use the following parameters:
| Parameter | Description |
|---|---|
| endpoint | Singapore: wss://ap-southeast-1.maas.qwencloudapi.com/api-ws/v1/realtime. For China North 2 (Beijing), replace ap-southeast-1 with cn-beijing. API keys are region-specific. See Regions. |
| query parameter | The model query parameter must be set to the model name. Example: ?model=qwen3.8-livetranslate-flash-realtime |
| message header | Use a Bearer Token for authentication: Authorization: Bearer $DASHSCOPE_API_KEY > DASHSCOPE_API_KEY is your API key from QwenCloud. |
2. Configure language, modality, and voice
Configure the session through session.update:
- Target language: Set
session.translation.language, such asenfor English. Withqwen3.8-livetranslate-flash-realtime, the source language is detected automatically.
qwen3.5-livetranslate-flash-realtime, set session.input_audio_transcription.language to specify the source language, or omit it for automatic detection.- Output modalities: With
qwen3.8-livetranslate-flash-realtime, setsession.output_modalitiesto["text"]for text only or["text", "audio"]for text and audio (default).
qwen3.5-livetranslate-flash-realtime, use session.modalities. The supported values and default are the same.- Source transcription: With
qwen3.8-livetranslate-flash-realtime, source transcription is always enabled and cannot be disabled. The LiveTranslate model itself generates the source transcript. Relevant information in images can help resolve ambiguities in the source speech. Receive increments throughconversation.item.input_audio_transcription.deltaand complete transcripts throughconversation.item.input_audio_transcription.completed.
qwen3.5-livetranslate-flash-realtime, source transcription is enabled by default. A separate ASR model transcribes the input audio. Set session.input_audio_transcription.model to qwen3-asr-flash-realtime to enable it or null to disable it. Receive increments through conversation.item.input_audio_transcription.text and complete transcripts through conversation.item.input_audio_transcription.completed.- Audio and voice: With
qwen3.8-livetranslate-flash-realtime, defaults are 16000 Hz PCM input, 24000 Hz PCM output, and voiceTina. You can also send Opus audio by settingsession.audio.input.format.typetoopus. Checksession.audio.output.voicein session.created for the voice. To preserve the speaker’s voice, see Voice cloning.
qwen3.5-livetranslate-flash-realtime, configure audio through session.sample_rate, session.input_audio_format, and session.output_audio_format, and the voice through session.voice. The default voice is also Tina.- Hotwords: Set
session.translation.corpus.phrasesto specify preferred translations. See Configure hotwords.
3. Input audio and images
Send Base64-encoded audio and image data using the input_audio_buffer.append and input_image_buffer.append events. Audio input is required; image input is optional.
Images can be from a local file or captured in real time from a video stream.With
qwen3.8-livetranslate-flash-realtime, the default is session.audio.input.turn_detection.type = speaker_detection. Send audio continuously; the server detects speech boundaries and generates translation responses without a manual commit for each utterance. See Identify speakers for configuration and speaker identifiers.
qwen3.5-livetranslate-flash-realtime: VAD and Manual modes
qwen3.5-livetranslate-flash-realtime: VAD and Manual modes
- VAD mode (default): The client continuously sends input_audio_buffer.append events. When the server detects speech start/end, it returns
input_audio_buffer.speech_startedandinput_audio_buffer.speech_stoppedevents respectively, automatically commits the audio buffer, and triggers translation. Translation responses are generated synchronously with the streaming audio and typically begin during audio input, without waiting for the speech to end. - Manual mode: Set
session.turn_detectiontonull. After the client finishes sending a complete utterance, it sends an input_audio_buffer.commit event to commit the audio buffer. After the server returns aninput_audio_buffer.committedevent to confirm, it automatically starts generating the translation response; the client does not need to send any other event to trigger the response. To clear uncommitted audio before committing, send an input_audio_buffer.clear event.
4. Receive the model response
With qwen3.8-livetranslate-flash-realtime, handle responses according to the output modalities:
- Text only: Append the
deltavalues fromresponse.text.deltato the subtitles in order, without overwriting text already displayed. - Text and audio: Concatenate
deltafromresponse.audio_transcript.deltafor text, and Base64-decodedeltafromresponse.audio.deltato obtain audio chunks.
response.done indicates the end of a response. See Server events for event fields.
qwen3.5-livetranslate-flash-realtime: Response event differences
qwen3.5-livetranslate-flash-realtime: Response event differences
-
Output source language recognition results
Set
session.input_audio_transcription.modeltoqwen3-asr-flash-realtime. The server then returns both the translation and the speech recognition result (original text) for the input audio. The server returns these events:conversation.item.input_audio_transcription.text: Streams the recognition results.conversation.item.input_audio_transcription.completed: Returns the final result after the recognition is complete.conversation.item.input_audio_transcription.failed: Returns error information when recognition fails.
-
Text-only output
The server streams incremental translated text (including confirmed text and tentative predicted text) through response.text.text events; upon completion, the full translated text is returned in a response.text.done event.
stashcontains provisional translation text. It lets subtitles appear sooner, but later events may rewrite it. Depending on your application's needs, displaytextandstashtogether in the client, or display only the confirmedtext. Useitem_idto update the matching subtitle, then show the complete translation whenresponse.text.donearrives. -
Text and audio output
- Text The server streams incremental translated text through response.audio_transcript.text events; upon completion, the full translated text is returned in a response.audio_transcript.done event.
- Audio The server returns incremental, Base64-encoded audio data in response.audio.delta events.
qwen3.5-livetranslate-flash-realtime uses the response.text.text event for incremental text delivery, which differs from the response.text.delta event used by Omni (full-duplex voice conversation) models. These events have different field structures and semantics — do not use them interchangeably.5. End the session
After sending all audio, send a session.finish event, then wait for the server to return a session.finished event before closing the WebSocket connection.
If you close the WebSocket without sending session.finish, the server's VAD cannot detect the end of the final speech segment. This causes translation results for that segment to be lost entirely, and the connection may hang indefinitely. Always send this event before disconnecting.
Supported models
Recommended models
| Model | Version | Context window | Max input | Max output |
|---|---|---|---|---|
| (tokens) | ||||
| qwen3.8-livetranslate-flash-realtime | Stable | 53248 | 49152 | 4096 |
qwen3.5-livetranslate-flash-realtimeAlias for qwen3.5-livetranslate-flash-realtime-2026-05-19 | Stable | 53248 | 49152 | 4096 |
| qwen3.5-livetranslate-flash-realtime-2026-05-19 | Snapshot | |||
Getting started
Translate microphone input in real time
- Use Python 3.10 or later and install
pyaudio:
- macOS
- Debian/Ubuntu
- CentOS
- Windows
pip install websocket-client.
- Set this environment variable. The API key must belong to the region in the connection URL:
DASHSCOPE_API_KEY: An API key for the region. See Get an API key.
- Save the code below as
livetranslate_client.py, runpython livetranslate_client.py, and speak into the microphone.
qwen3.8-livetranslate-flash-realtime to translate microphone input into English, print the text, and play the translated speech in real time. Input is mono, 16-bit, 16000 Hz PCM; playback is mono, 16-bit, 24000 Hz PCM. Press Ctrl+C to stop recording. The example sends session.finish, receives and plays the remaining results, and closes the connection after session.finished. Use headphones to keep translated speech from being recorded again.
Configure hotwords
Adding hotwords can improve translation accuracy when you need a fixed translation or need to distinguish between homophones. For example:
- Fixed wording: Alibaba Cloud's slogan, “计算,为了无法计算的价值。”, has the agreed English wording “Computing, for the value beyond computation.”. The wording and word order need to be preserved.
- Brand names that sound alike: Qoder sounds like the common word coder. When the speaker means the product Qoder, the translation needs to retain the brand's spelling.
| Content of the input audio | Without hotwords | With hotwords |
|---|---|---|
| 计算,为了无法计算的价值。 | Calculate. For values that cannot be calculated. | Computing, for the value beyond computation. |
| Qoder 是面向真实工作的 Agentic 平台。 | Coder is an agentic system designed for real-world work. platform. | Qoder, is an agentic platform designed for real-world work. |
Configuration
Set hotwords through session.translation.corpus.phrases, using key-value pairs to map source-language words or phrases to their preferred translations.
In the first session.update request from Getting started, merge the following translation object into the existing session. Keep the output modality, voice, and other settings for the session.
session.updated event before sending input_audio_buffer.append events.
Identify speakers
qwen3.8-livetranslate-flash-realtime can identify different speakers and label them with speaker_id. Associate this ID with the source text and translation to display subtitles by speaker.
Keep the default speaker_detection, or configure it explicitly in session.update. threshold distinguishes silence from valid speech. Sounds below the threshold are filtered out before they reach the model. The threshold is fixed at 0.5.
- Record the
speaker_idand sourceitem_idfrominput_audio_buffer.speech_started. - Read the source text for that
item_idfromconversation.item.input_audio_transcription.delta. - In the translation's
conversation.item.created, associateitem.idwith the source ID in the event's top-levelprevious_item_id. - Use each translation increment's
item_idto find its source text and speaker, then update the subtitle.
def receive_translation(): in the quick start microphone example. Within that function, call handle_speaker_event(event) immediately after event = receive(), and replace the original print(event["delta"], end="", flush=True) with pass so that the snippet handles all subtitle output. The snippet prints the speaker ID, source text, and translation as they change, so you can display subtitles by speaker. Keep the existing audio and session-completion handling.
Improve translation with images
Qwen real-time translation models support image input to help disambiguate homonyms and recognize uncommon proper nouns during translation. Send no more than 2 images per second.
Download the following sample images: medical mask.png, masquerade mask.png
Reuse the quick start's microphone input and connection settings:
- Install
Pillow:pip install Pillow. - Set
translation.languagetozhfor Chinese output, then say “What is mask?” into the microphone when you run the example. - Place the image-loading code below before
def send_audio():in the microphone example, and replace the originalsend_audio()with this version. It uses the existinginput_stream,stop_recording,ws,send_errors, and imports.
Ctrl+C to stop recording, the example requests session completion and receives the remaining results.
Voice cloning
The model clones the speaker's voice from input audio and uses it for translated output. Use a pre-cloned voice profile or clone in real time during translation. Useful for conference interpreting, live streaming, and video dubbing.
Real-time cloning: Both qwen3.8-livetranslate-flash-realtime and qwen3.5-livetranslate-flash-realtime clone voices from input audio during translation without an additional cloning fee. Model calls remain billed by token usage.
Create a voice in advance: The Voice Cloning API charges $0.01 per voice; failed creation is not billed. See that billing section for free-quota regions, limits, and validity. Translation using the created voice is billed by model token usage.
Set the following parameters in session.update to enable voice cloning:
-
session.enable_voice_clone: Set totrueto enable voice cloning. -
session.voice_clone_options.frequency: Controls when voice cloning occurs. Accepted values:never: Use a previously created voice without extracting a new voice during this session. Setsession.voiceto your cloned voice ID.once: Extract the voice once from input audio at the start of the session and reuse it for subsequent translations in this session. “Once” describes extraction, not a single audio output. Suitable for a single-speaker presentation; setsession.voicetodefault.always: Extract the voice before each translated-audio response, so output follows speaker changes. Suitable for multiple speakers; setsession.voicetodefault.
-
session.voice: Specifies the output voice. The value depends on thefrequencysetting:- Set to
default: Use withfrequencyset toonceoralways. Clones the speaker's voice from the input audio. A default voice is used until cloning completes. - Set to a custom cloned voice ID (for example,
qwen-translate-vc-xxx-yyy-zzz): Use withfrequencyset tonever. Prepare the voice in advance through the Voice cloning API, withtarget_modelset to the translation model you use.
- Set to
Whenfrequencyis set toonceoralways, thevoiceparameter must be set todefault. Any other value causes the server to return an error.
Voice cloning configuration examples
With qwen3.8-livetranslate-flash-realtime, replace the session.update configuration in its Quick start example with the configuration for your scenario.
qwen3.5-livetranslate-flash-realtime, replace session.output_modalities with session.modalities in the examples. The other cloning parameters are the same. Specify this model in the connection’s model parameter and handle its response events as described in Receive model responses.One-click Function Compute deployment
To deploy the application:
- Open the Function Compute template, enter your API key, and click Create and Deploy Default Environment to test the application.
- Wait for about a minute. In Environment Details > Environment Context, retrieve the endpoint, change the protocol from http to https (for example, https://qwen-livetranslate-flash-realtime-intl.fcv3.xxx.ap-southeast-1.fc.devsapp.net/), and open the URL in a browser to interact with the model.
If you are prompted to configure Resource Access Management permissions, follow the on-screen instructions.
To view the project source code, go to Resource Information > Function Resources .
Both Function Compute and QwenCloud provide a free quota for new users, sufficient for basic debugging. After the free quota is used up, pay-as-you-go billing applies.
Interaction flow
With qwen3.8-livetranslate-flash-realtime, the server detects turns and generates responses by default. The following table summarizes the main interaction stages. See Server events for event fields.
| Stage | Client action | Server events |
|---|---|---|
| Create and configure the session | Connect and send session.update | session.created, session.updated |
| Send audio | input_audio_buffer.append | Source transcription is streamed through conversation.item.input_audio_transcription.delta, followed by conversation.item.input_audio_transcription.completed. |
| Receive translation and audio | Keep receiving server events | Translation is returned through response.text.delta (text only) or response.audio_transcript.delta (text and audio). Audio is returned through response.audio.delta. response.done marks the end of a response. |
| End the session | session.finish | Close the connection after receiving session.finished. |
qwen3.5-livetranslate-flash-realtime: Interaction flow
qwen3.5-livetranslate-flash-realtime: Interaction flow
| Lifecycle | Client event | Server event |
|---|---|---|
| Session initialization | session.update > Session configuration | session.created > Session created session.updated > Session configuration updated |
| User audio input | input_audio_buffer.append > Append audio to the buffer input_image_buffer.append > Append image to the buffer input_audio_buffer.commit > (Manual mode only) Commit the audio buffer | VAD mode: input_audio_buffer.speech_started > Speech start detected input_audio_buffer.speech_stopped > Speech end detected; server automatically commits the audio buffer Manual mode: input_audio_buffer.committed > Returned after the client sends input_audio_buffer.commit, confirming the audio buffer has been committed |
| Server audio output | None | response.created > Signals that the server starts generating a response. response.output_item.added > Signals that a new output item is available. conversation.item.created > A new message item is created in the conversation. response.content_part.added > Signals that a new content part has been added to the assistant message. response.text.text > Incremental translated text in text-only modality response.audio_transcript.text > Incremental translated text in audio+text modality response.audio.delta > Contains an incremental chunk of the synthesized audio. response.text.done > Translation text complete in text-only modality response.audio_transcript.done > Translation text complete in audio+text modality response.audio.done > Signals that the synthesized audio is complete. response.content_part.done > Signals that a text or audio content part for the assistant message is complete. response.output_item.done > Signals that the entire output item for the assistant message is complete. response.done > Signals that the entire response is complete. |
| Session termination | session.finish > Notifies the server that audio input is complete | session.finished > Server processing complete; session ended |
session.finish event and wait for session.finished before closing the WebSocket. If you close the connection without sending session.finish, the server cannot know that audio input has ended, and the recognition and translation results for the last speech segment will be lost.
API
For the AOQ access flow and examples, see AOQ access. For protocol selection and reference links, see the API reference.
Billing
Qwen3.8-LiveTranslate-Flash-Realtime and Qwen3.5-LiveTranslate-Flash-Realtime
- Audio: 7 tokens per second of input audio; 12.5 tokens per second of output audio.
- Image: Every 32×32 pixels consumes 0.5 tokens. For token prices for each model, see Model pricing and billing.
qwen3.8-livetranslate-flash-realtime and qwen3.5-livetranslate-flash-realtime clone voices from input audio during translation without an additional cloning fee. Model calls remain billed by token usage.
Create a voice in advance: The Voice Cloning API charges $0.01 per voice; failed creation is not billed. See that billing section for free-quota regions, limits, and validity. Translation using the created voice is billed by model token usage.
Rate limits
For information about model rate limits, see Rate limiting.
Supported languages
Use the following language codes to specify the source and target languages.
Some target languages only support text.
| Language code | Language | Output |
|---|---|---|
| zh | Chinese | Audio + text |
| en | English | Audio + text |
| ar | Arabic | Audio + text |
| de | German | Audio + text |
| fr | French | Audio + text |
| es | Spanish | Audio + text |
| pt | Portuguese | Audio + text |
| id | Indonesian | Audio + text |
| it | Italian | Audio + text |
| ko | Korean | Audio + text |
| ru | Russian | Audio + text |
| th | Thai | Audio + text |
| vi | Vietnamese | Audio + text |
| ja | Japanese | Audio + text |
| tr | Turkish | Audio + text |
| hi | Hindi | Audio + text |
| ms | Malay | Audio + text |
| nl | Dutch | Audio + text |
| ur | Urdu | Audio + text |
| nb | Norwegian Bokmål | Audio + text |
| sv | Swedish | Audio + text |
| da | Danish | Audio + text |
| he | Hebrew | Audio + text |
| fi | Finnish | Audio + text |
| pl | Polish | Audio + text |
| is | Icelandic | Audio + text |
| cs | Czech | Audio + text |
| fil | Filipino | Audio + text |
| fa | Persian | Audio + text |
| yue | Cantonese | Text |
| el | Greek | Text |
| af | Afrikaans | Text |
| ast | Asturian | Text |
| be | Belarusian | Text |
| bg | Bulgarian | Text |
| bn | Bengali | Text |
| bs | Bosnian | Text |
| ca | Catalan | Text |
| ceb | Cebuano | Text |
| et | Estonian | Text |
| gl | Galician | Text |
| gu | Gujarati | Text |
| hr | Croatian | Text |
| hu | Hungarian | Text |
| jv | Javanese | Text |
| kk | Kazakh | Text |
| kn | Kannada | Text |
| ky | Kyrgyz | Text |
| lv | Latvian | Text |
| mk | Macedonian | Text |
| ml | Malayalam | Text |
| mr | Marathi | Text |
| pa | Punjabi | Text |
| ro | Romanian | Text |
| sk | Slovak | Text |
| sl | Slovenian | Text |
| sw | Swahili | Text |
| tg | Tajik | Text |
| az | Azerbaijani | Text |
| uk | Ukrainian | Text |
Supported voices
For supported voices and the corresponding voice parameter values, see Voice list.
FAQ
Is there a charge while the connection is open but nobody speaks?
Keeping a connection open is not billed. If you send silent audio, whether it counts as audio input depends on the mode you use:
- Server-side speech detection: Filtered silence does not count as audio input.
- Manual mode: Silence you explicitly commit counts as audio input.
response → usage → input_tokens_details → audio_tokens. The model is billed by input and output tokens. See Billing for prices and calculation rules.
How do I migrate from Gummy to LiveTranslate?
gummy-realtime-v1 and qwen3.8-livetranslate-flash-realtime use different protocols. Besides changing the model name, update the code that opens the connection, sends audio, receives translations, and ends the session.
For English captions with the Python SDK, change four calls. Reuse your 16 kHz mono 16-bit PCM recording.
Step | Gummy | LiveTranslate |
|---|---|---|
Connect and configure |
|
Wait for |
Send audio |
|
|
Display captions |
|
Read events while sending audio. Append each |
End the session |
|
Read the last caption. Close after |
Python example: translate a recording into English captions
Python example: translate a recording into English captions
DASHSCOPE_API_KEY to your QwenCloud API key. Save the code as subtitles.py and the recording as input.wav. Run:- Endpoint Change
/api-ws/v1/inferenceto/api-ws/v1/realtime?model=qwen3.8-livetranslate-flash-realtime. - Audio Replace binary frames with Base64 PCM in the
audiofield of eachinput_audio_buffer.appendevent. - Captions Replace
result-generatedwithresponse.text.delta. Append eachdeltato the caption.
session.finish. Wait for session.finished before closing. Full example
What does RPM limit? Is there a limit on the total number of simultaneous connections?
RPM limits the number of WebSocket connection requests per minute. Responses on an existing connection do not count toward RPM. Reconnecting counts as a new request. There is currently no separate limit on the total number of connections open at the same time.
If a WebSocket closes during setup with code 1007 and reason Requests rate limit exceeded, please try again later., the request rate limit was reached. Reduce the connection rate and reconnect with exponential backoff.
How do I resume translation after a disconnection?
If you receive no semantic result after merged audio exceeded 60s and the WebSocket connection closes, create a new session to resume translation:
- Stop sending audio to the old connection.
- Reconnect and send
session.updateto restore the target language, output modalities, and other settings. - Wait for
session.updated, then resume sending audio.
dashscope.Client(), also resume event reception.