Qwen-Audio-Realtime WebSocket API

​

Service endpoint

wss://dashscope-intl.aliyuncs.com/api-ws/v1/realtime?model=<model_name>

​

Request headers

​

Core concepts

• Session: Each WebSocket connection corresponds to one session, which maintains configuration and conversation context.
• Conversation item: Each message in the conversation, organized as a linked list in order.
• Response: The output produced by one round of model inference, containing one or more output items. An output item can be an assistant message or a function call.
• Function call: An output item produced when the model requests the client to execute a tool function. After execution, the client writes back the result via function_call_output and triggers the next round of inference with response.create.
• Turn detection: Controls when inference is triggered.

​

Interaction modes

​

Interaction flows

​

server_vad mode

​

A complete conversation turn

1. The client establishes a WebSocket connection; the server returns a session.created event.
2. The client sends session.update to configure session parameters; the server returns session.updated.
3. The client continuously sends input_audio_buffer.append events to append audio data.
4. The server detects the start of speech and returns input_audio_buffer.speech_started, while streaming incremental ASR transcription via conversation.item.input_audio_transcription.delta.
5. The server detects the end of speech and returns input_audio_buffer.speech_stopped, input_audio_buffer.committed, and conversation.item.created.
6. The server automatically generates a response, streaming text and audio deltas (response.audio_transcript.delta, response.audio.delta), and finally returns response.done.

​

User interruption

1. The server is streaming response.audio.delta (loop).
2. The client sends input_audio_buffer.append (the user starts speaking).
3. The server returns response.done (status=cancelled), canceling the current response.
4. The server returns input_audio_buffer.speech_started.
5. The client continues sending input_audio_buffer.append; the server streams conversation.item.input_audio_transcription.delta.
6. The server detects the end of speech and returns input_audio_buffer.speech_stopped and conversation.item.input_audio_transcription.completed.
7. The server commits the audio buffer internally, returning input_audio_buffer.committed and conversation.item.created.
8. The server automatically starts a new round of inference and returns response.created.

​

smart_turn mode

​

A complete conversation turn

• Non-semantic sounds (such as "um" or "uh") do not trigger inference; instead, they are returned via the ambient_audio_transcription events.
• Speech that was initially judged valid may later be retracted (input_audio_buffer.speech_stopped returns with reason=turn_invalid), in which case inference is not triggered.
• While waiting for the user's next input, the client can explicitly send response.create to trigger inference.

​

User interruption

1. The server is streaming response.audio.delta (loop).
2. The client sends input_audio_buffer.append (the user starts speaking).
3. The server returns response.done (status=cancelled), canceling the current response.
4. The server returns input_audio_buffer.speech_started.
5. The client continues sending input_audio_buffer.append; the server streams conversation.item.input_audio_transcription.delta.
6. The server returns conversation.item.input_audio_transcription.completed and input_audio_buffer.speech_stopped.
7. The server commits the audio buffer internally, returning input_audio_buffer.committed and conversation.item.created.
8. The server automatically starts a new round of inference and returns response.created.

​

Invalid turns

1. The client continuously sends input_audio_buffer.append.
2. The server returns input_audio_buffer.speech_started.
3. The client continues sending input_audio_buffer.append; the server streams conversation.item.input_audio_transcription.delta.
4. The server returns input_audio_buffer.speech_stopped (reason=turn_invalid).
5. The client continues sending audio, waiting for the next valid speech turn.

​

Voiceprint enrollment flow

1. The client sends session.update, including the voiceprint audio URL in turn_detection.voiceprint_audio_urls; the server returns session.updated.
2. The server immediately starts voiceprint enrollment asynchronously, pushing a voiceprint_audio_list.in_progress event before returning session.updated, carrying a unique item_id for this enrollment task.
3. The server returns session.updated, confirming that the session configuration has taken effect.
4. After voiceprint enrollment completes, the server pushes a terminal event (with the same item_id as in step 2):
   • On success: voiceprint_audio_list.completed.
   • On failure: voiceprint_audio_list.failed, with a reason field explaining the failure (for example, the audio URL could not be downloaded).

​

push-to-talk mode

​

A complete conversation turn

1. The client continuously sends input_audio_buffer.append events to append audio data.
2. After the user finishes speaking, the client sends input_audio_buffer.commit to commit the buffer.
3. The client sends response.create to manually trigger inference.
4. The server generates a response, streaming text and audio.

​

User interruption

1. The server is streaming response.audio_transcript.delta and response.audio.delta (loop).
2. The client sends response.cancel.
3. The server returns response.done (status=cancelled, reason=client_cancelled).
4. The client continues sending input_audio_buffer.append; the server streams conversation.item.input_audio_transcription.delta.
5. The client sends input_audio_buffer.commit to commit the buffer.
6. The server returns conversation.item.input_audio_transcription.completed, input_audio_buffer.committed, and conversation.item.created.
7. The client sends response.create to manually trigger a new round of inference.
8. The server returns response.created, starting a new round of response.

​

Operation constraints by mode

​

Error handling