Skip to main content
WebSocket

Java SDK

The key interfaces and request parameters for Qwen-Omni real-time DashScope Java SDK.

Prerequisites

Your Java SDK version must be v2.22.15 or later. Before you begin, see Real-time multimodal interaction flow.

Getting started

To use qwen3.8-omni-flash-realtime, set the session constructor's model parameter to this model name. The default voice is Tina. For Java SDK session configuration and video aggregation examples, see the real-time guide. Download the sample code from GitHub. Three calling methods are available:
  1. Audio conversation example: Captures real-time audio from a microphone, enables Voice Activity Detection (VAD) mode, which automatically detects the start and end of speech, and supports voice interruption.
    Set the enableTurnDetection parameter to true.
    Use headphones for audio playback to prevent echoes from triggering voice interruption.
  2. Audio and video conversation example: Captures real-time audio and video from a microphone and camera, enables VAD mode, and supports voice interruption.
    Set the enableTurnDetection parameter to true.
    Use headphones for audio playback to prevent echoes from triggering voice interruption.
  3. Local call: Uses local audio and images as input and enables Manual mode, which lets you manually control the sending pace.
    Set the enableTurnDetection parameter to false.

Request parameters

Configure the following request parameters using the chained methods or setters of the OmniRealtimeParam object, and then pass it to the OmniRealtimeConversation constructor.
ParameterTypeDescription
modelStringThe Qwen-Omni model to use. See Model list.
urlStringService endpoint:
  • Singapore: wss://ap-southeast-1.maas.qwencloudapi.com/api-ws/v1/realtime
  • China North 2 (Beijing): wss://cn-beijing.maas.qwencloudapi.com/api-ws/v1/realtime
API keys are region-specific. See Regions.
Configure the following request parameters using the chained methods or setters of the OmniRealtimeConfig object, and then pass it to the updateSession method.
ParameterTypeDescription
modalitiesList<OmniRealtimeModality>The model output modality. Set to [OmniRealtimeModality.TEXT] for text only, or [OmniRealtimeModality.TEXT, OmniRealtimeModality.AUDIO] for both audio and text.
voiceStringThe voice for audio output. For supported voices, see Voice list. The default voice for the Qwen3.5-Omni-Realtime series is Tina, and for qwen3-omni-flash-realtime-2025-09-15 it is Cherry.
inputAudioFormatOmniRealtimeAudioFormatThe format of the user's input audio. Currently only supports PCM_16000HZ_MONO_16BIT, which represents a PCM audio stream at a 16 kHz sample rate.
outputAudioFormatOmniRealtimeAudioFormatThe format of the model's output audio. Currently only supports PCM_24000HZ_MONO_16BIT, which represents a PCM audio stream at a 24 kHz sample rate.
instructionsStringA system message that sets the goal or role for the model. Example: "You are an AI customer service agent for a five-star hotel. Answer customer questions about room types, facilities, prices, and booking policies accurately and in a friendly manner. Always respond professionally and helpfully. Do not provide unverified information or information outside the scope of the hotel's services." Note: Set instructions using the OmniRealtimeConfig instance's parameters method: See Example 1 below
smooth_outputBooleanThis parameter is supported only by qwen3-omni-flash-realtime-2025-09-15.
  • true: The model provides conversational responses.
  • false: The model provides more formal, written-style responses. However, performance may be suboptimal if the content is difficult to read aloud.
  • null (default): The model automatically chooses between conversational and formal response styles.
Set smooth_output using the OmniRealtimeConfig instance's parameters method, the same as instructions.
enableInputAudioTranscriptionBooleanEnables speech recognition for input audio.
InputAudioTranscriptionStringSpeech recognition model for input audio transcription. The value is always qwen3-asr-flash-realtime. This parameter is not configurable.
enableTurnDetectionBooleanEnables VAD. If disabled, you must manually submit audio to create a response.
turnDetectionTypeStringThe VAD type. Valid values:
  • server_vad (default): Detects the end of user speech based on acoustic features.
  • semantic_vad: Detects the end of user speech based on semantic validity. This mode can filter out meaningless speech, such as backchannels and background noise. This option is supported by Qwen3.8-Omni-Flash-Realtime and the Qwen3.5-Omni-Realtime series.
turnDetectionThresholdFloatVAD detection threshold. Increase in noisy environments and decrease in quiet environments.
  • The closer the value is to -1, the more likely noise is to be detected as speech.
  • The closer the value is to 1, the less likely noise is to be detected as speech.
Default: 0.5. Valid values: [-1.0, 1.0].
turnDetectionSilenceDurationMsIntegerSilence duration that indicates end of speech. If exceeded, the model triggers a response. Default: 800. Valid values: [200, 6000].
turnDetectionParamMapAdditional turn_detection configuration parameters. Currently supports idle_timeout_ms (Integer): the idle timeout in milliseconds. Applies only toqwen3.5-omni-plus-realtimeandqwen3.5-omni-flash-realtimemodels inserver_vadmode. After the server finishes audio playback and the user remains silent beyond this duration (no speech.started triggered), the model proactively generates a response to prompt the user to continue the conversation. Valid range: [5000, 30000]. Example: turnDetectionParam(Map.of("idle_timeout_ms", 5000))
enable_searchBooleanApplies to Qwen3.8-Omni-Flash-Realtime and Qwen3.5-Omni-Realtime models. Enables web search. Default: false. When enabled, the model can search the web to answer real-time questions. > enable_search and search_options parameters must be set using the parameters method of the OmniRealtimeConfig instance. The process is the same for instructions. > Tool calling (tools) and web search (enable_search) are incompatible and cannot be enabled at the same time.
search_optionsObjectWeb search option settings. This parameter takes effect only after enable_search is enabled. You can set enable_source (Boolean) to true to return a list of search result sources. > search_options using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions.
toolsList<Map<String, Object>>Applies to Qwen3.8-Omni-Flash-Realtime and Qwen3.5-Omni-Realtime models. Tool definitions. When provided, the model can call external tools to respond to user questions. For Function Calling, the model does not generate audio and only returns the tool calling parameters. Qwen3.8-Omni-Flash-Realtime supports Function Calling and MCP tools in the same session. See client events for MCP configuration and MCP call limits for quotas. tools and enable_search are mutually exclusive, including MCP tools. The following fields describe Function Calling tools. Each tool is a Map that contains the following fields:
  • type (String, required): The value is fixed to "function".
  • function (Map, required): The definition of the tool function. This field contains the following subfields:
  • name (String, required): A custom name for the tool function. We recommend that you use the same name as the function, such as get_current_weather or get_current_time.
  • description (String, optional): A description of the tool function. The model uses this description to decide whether to call the function.
  • parameters (Map, optional): Input parameter descriptions. The model uses them to extract parameters. If the tool function does not require input parameters, you do not need to specify this field. This field contains the following subfields:
  • type (String, required): The value is fixed to "object".
  • properties (Map, optional): Describes the name, data type, and description of each input parameter. The Key is the parameter name, and the Value is a Map that contains the data type (type) and description (description).
  • required (List, optional): Specifies which input parameters are required.
> search_options using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions.
temperatureFloatSampling temperature that controls content diversity. Higher values produce more diverse content; lower values produce more deterministic content. Valid values: [0, 2). Because both temperature and top_p control content diversity, we recommend that you set only one of them. Default values of temperature:
  • qwen3.8-omni-flash-realtime: 0.6
  • Qwen3.5-Omni-Realtime series model: 0.7
  • qwen3-omni-flash-realtime-2025-09-15: 0.9
> temperature parameter using the parameters method of the OmniRealtimeConfig instance, in the same way you set the instructions parameter.
top_pFloatProbability threshold for nucleus sampling that controls content diversity. Higher values produce more diverse content; lower values produce more deterministic content. Valid values: (0, 1.0]. Because both temperature and top_p control content diversity, we recommend that you set only one of them. Default values of top_p:
  • qwen3.8-omni-flash-realtime: 0.95
  • Qwen3.5-Omni-Realtime series model: 0.8
  • qwen3-omni-flash-realtime-2025-09-15: 1.0
> top_p using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions.
top_kIntegerCandidate set size for sampling. For example, 50 means only the 50 highest-scoring tokens form the candidate set. Larger values increase randomness; smaller values increase determinism. Set to None or a value greater than 100 to disable top_k and use only top_p. The value must be greater than or equal to 0. Default values of top_k:
  • qwen3.8-omni-flash-realtime: 20
  • Qwen3.5-Omni-Realtime series model: 20
  • qwen3-omni-flash-realtime-2025-09-15: 50
> top_k using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions.
max_tokensIntegerMaximum number of tokens to return. > max_tokens setting does not affect the generation process of the model. If the number of tokens generated by the model exceeds max_tokens, the returned content is truncated. For qwen3.8-omni-flash-realtime, the valid range is [1, 65536]. For other models, default and maximum values equal the model's maximum output length. See Model list for details. Use max_tokens to limit output length for generating summaries, keywords, controlling costs, or reducing response time. > max_tokens using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions.
repetition_penaltyFloatControls repetition in generated sequences. Higher values reduce repetition. 1.0 means no penalty. qwen3.8-omni-flash-realtime also accepts 0. For other models, the value must be greater than 0. Default values of repetition_penalty:
  • qwen3.8-omni-flash-realtime: 0
  • Qwen3.5-Omni-Realtime series model: 1.0
  • Other models: 1.05
> repetition_penalty using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions.
presence_penaltyFloatControls content repetition in model output. Valid values: [-2.0, 2.0]. Positive values reduce repetition; negative values increase it. Default values of presence_penalty:
  • qwen3.8-omni-flash-realtime: 1
  • Qwen3.5-Omni-Realtime series model: 1.5
  • Other models: 0.0
Scenarios: Higher values suit scenarios requiring diversity or creativity, such as creative writing or brainstorming. Lower values suit scenarios requiring consistency or technical precision, such as technical documents. > presence_penalty using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions.
seedIntegerMakes generation more deterministic, ensuring consistent results across runs. Passing the same seed with identical parameters produces the same result as much as possible. Valid values: 0 to 231−1. Default value: -1. > seed using the OmniRealtimeConfig instance's parameters method. The usage is the same as for instructions.
Example 1 (Description):
conversation.updateSession(OmniRealtimeConfig.builder()
        .modalities(Arrays.asList(OmniRealtimeModality.AUDIO, OmniRealtimeModality.TEXT))
        .voice("Tina")
        .enableTurnDetection(true)
        .enableInputAudioTranscription(true)
        .parameters(Map.of(
                "instructions", "You are a personal assistant named Xiaoyun"
        ))
        .build()
);

Key interfaces

OmniRealtimeConversation class

Import with import com.alibaba.dashscope.audio.omni.OmniRealtimeConversation;.
Method signatureServer response event (sent via callback)Description
See Example 1 belowServer events > Session created session.updated > Session configuration updatedCreates a connection to the server.
See Example 2 belowsession.updated > Session configuration updatedUpdates session configuration. For parameter details, see Request parameters. After connecting, the server returns default session configurations. Call this method immediately after connecting to update settings. When the server receives the session.update event, it validates the parameters. Invalid parameters return an error; otherwise the server updates session configuration.
See Example 3 belowNoneAppends Base64-encoded audio to the cloud input buffer. The buffer is temporary storage that can be written to and committed later.
  • If "turn_detection" is enabled, the audio buffer is used to detect speech, and the server decides when to commit.
  • If "turn_detection" is disabled, the client can send up to 15 MiB of audio per event. Streaming smaller blocks makes VAD more responsive.
See Example 4 belowNoneAdds Base64-encoded image data to the cloud video buffer. Accepts local images or real-time video stream captures. Image input limits:
  • The image format must be JPG or JPEG. The recommended image resolution is 480p or 720p, with a maximum of 1080p.
  • A single image after Base64 encoding must not exceed 256 KB. We recommend keeping the raw image size below 190 KB before encoding.
  • The image data must be Base64-encoded.
  • We recommend that you send images to the server at a frequency of 1 image per second.
See Example 5 belowinput_audio_buffer.cleared > The audio received by the server is clearedDeletes the audio in the current cloud buffer.
See Example 6 belowinput_audio_buffer.committed > The server received the committed audioCommits audio and video from the cloud buffer. Returns an error if the buffer is empty.
  • If "turn_detection" is enabled, the client does not need to send this event. The server automatically commits the audio buffer.
  • If "turn_detection" is disabled, the client must commit the audio buffer to create a user message item.
Note: 1. If input_audio_transcription is configured for the session, the system transcribes the audio. 2. Committing the input audio buffer does not create a response from the model.
See Example 7 belowServer events > The server starts to generate a response response.output_item.added > New output content is available in the response Server events > A conversation item is created response.content_part.added > New output content is added to the assistant message item response.audio_transcript.delta > Incrementally generated transcribed text response.audio.delta > Incrementally generated audio from the model response.audio_transcript.done > Text transcription is complete response.audio.done > Audio generation is complete response.content_part.done > Streaming of text or audio content for the assistant message is complete response.output_item.done > Streaming of the entire output item for the assistant message is complete response.done > The response is completeInstructs the server to create a model response. When the session is configured in "turn_detection" mode, the server automatically creates a model response.
See Example 8 belowNoneCancels the in-progress response. If no response is available to cancel, the server returns an error.
See Example 9 belowNoneSends the conversation.item.create event to the server. For a Function Calling tool result, item is a JsonObject with these fields:
  • type: Must be function_call_output.
  • call_id: Corresponds to the call_id in the response.function_call_arguments.done event.
  • output: A string that represents the tool execution result.
For MCP approval, the same event uses item.type="mcp_approval_response". See MCP approval response for its fields.
See Example 10 belowNoneStops the task and closes the connection.
See Example 11 belowNoneGets the session ID of the current task.
See Example 12 belowNoneGets the response ID of the most recent response.
Example 1 (Method signature):
public void connect() throws NoApiKeyException, InterruptedException
Example 2 (Method signature):
public void updateSession(OmniRealtimeConfig config)
Example 3 (Method signature):
public void appendAudio(String audioBase64)
Example 4 (Method signature):
public void appendVideo(String videoBase64)
Example 5 (Method signature):
public void clearAppendedAudio()
Example 6 (Method signature):
public void commit()
Example 7 (Method signature):
public void createResponse(String instructions, List<OmniRealtimeModality> modalities)
Example 8 (Method signature):
public void cancelResponse()
Example 9 (Method signature):
public void createItem(JsonObject item)
Example 10 (Method signature):
public void close()
Example 11 (Method signature):
public String getSessionId()
Example 12 (Method signature):
public String getResponseId()

Callback interface (OmniRealtimeCallback)

The server returns response events and data via callbacks. Implement callback methods to process server responses. Import with import com.alibaba.dashscope.audio.omni.OmniRealtimeCallback;.
MethodParameterReturn valueDescription
See Example 1 belowNoneNoneCalled immediately after a connection is established with the server.
See Example 2 belowmessage: The server response event.NoneContains interface call responses and model-generated text and audio. See Server events.
See Example 3 belowcode: The status code for closing the WebSocket. reason: The reason for closing the WebSocket.NoneCalled after the connection to the server is closed.
Example 1 (Method):
public void onOpen()
Example 2 (Method):
public abstract void onEvent(JsonObject message)
Example 3 (Method):
public abstract void onClose(int code, String reason)

FAQ

Q: How are the input audio and images aligned?

The Qwen-Omni real-time model uses the audio stream as the input timeline. Images are inserted into the audio stream based on the time they are sent. You can add images at any point in the audio timeline. In real-time interaction scenarios, you can enable or disable video input at any time. In real-time interaction scenarios, we recommend that you send images at a frame rate of 1 or 2 fps and send audio in 100 ms packets.

Q: What are the differences between the two modes of the turn_detection switch?

When turn_detection is enabled, it supports two modes: server_vad and semantic_vad:
  • Enable "turn_detection":
    • Input state: The cloud-based VAD detects the end of a sentence in the input audio and immediately triggers Qwen-Omni inference to return the reply text and speech.
    • Response state: In this state, you can continue to send audio and video input without interruption while the model is responding. After the response is complete, the state returns to the input state to await the next speech input.
    • Interruption: If the user starts speaking while the model is responding, an interruption is triggered. The service immediately stops the current response and switches to the input state.
  • Disable "turn_detection":
    • You must manually determine the end of a round of audio and video input and trigger Qwen-Omni inference to obtain a response using commit and create_response.
    • While the model is responding, you must stop sending audio and video input. You can resume input for the next round only after the model has finished responding.
    • You must use the cancel_response method to interrupt the model's response.
Note that when turn_detection is enabled, you can still actively trigger a response using commit and create_response, and actively interrupt it using cancel_response.

Q: Why do I need to select another model for input_audio_transcription?

Qwen-Omni real-time is an end-to-end multimodal model. Its text output is a response to the input, not a direct transcription of the input audio. A separate ASR model is required for transcription. Currently, the model is determined by the built-in configuration and is not configurable.