Skip to main content
QwenCloud home page
Docs
Best Practices

Realtime call with qwen3.5-omni-plus-realtime via WebRTC

Use qwen3.5-omni-plus-realtime model via WebRTC protocol for real-time calls

Describes how to connect to the QwenCloud Realtime API in a browser using WebRTC and JavaScript to enable real-time audio/video calls with the qwen3.5-omni-plus-realtime model.
WebRTC is suitable for browser-based, low-latency voice scenarios. Audio is transmitted directly over UDP with built-in echo cancellation and noise reduction. WebRTC supports only server-side VAD mode (server_vad or semantic_vad). Manual mode is not supported.

Prerequisites and notes

  1. Configure an API key and set it as an environment variable.
  2. Use a modern browser that supports WebRTC (Chrome, Edge, Firefox, Safari, and so on).
  3. The browser must have microphone permission. For video calls, camera permission is also required.
  4. Browsers cannot directly send SDP exchange requests to the server due to CORS restrictions. In the demo, run the curl command in a terminal to complete the connection. In production, this restriction does not apply when the request is proxied through an AppServer.

Implement AI audio/video calls

The following sequence diagram shows the complete WebRTC audio/video call flow: WebRTC audio/video call sequence diagram
image

Create RTCPeerConnection

Call the browser's native RTCPeerConnection constructor to create a connection instance. No ICE servers need to be configured (the server handles NAT traversal). 
pc = new RTCPeerConnection({ iceServers: [ ] });
Register key callbacks: 
// Connection state listener
pc.onconnectionstatechange = () => {
  if (!pc) return;
  if (pc.connectionState === 'connected') {
    setStatus('Connected, please speak', 'connected');
  } else if (["failed", "closed", "disconnected"].includes(pc.connectionState)) {
    endSession(true);
  }
};
// Receive and play remote audio stream + start recording
pc.ontrack = async (e) => {
  const stream = e.streams[0];
  ensureHiddenAudioEl();
  hiddenRemoteAudioEl.srcObject = stream;
  try { await hiddenRemoteAudioEl.play(); } catch {}
  startRecordingRemoteStream(stream);
};

Get local media streams

Use a single getUserMedia call to request audio (required) and video (optional). Whether video is enabled is determined by the user's Enable video checkbox. 
const wantVideo = !!sendVideoCheckbox.checked;
const constraints = wantVideo
  ? {
      audio: true,
      video: {
        facingMode: { ideal: "user" },
        frameRate: { ideal: 30, max: 30 },
        width: { ideal: 640 },
        height: { ideal: 480 },
      }
    }
  : { audio: true };
localStream = await navigator.mediaDevices.getUserMedia(constraints);
Both audio and video are obtained in a single getUserMedia call, not separately. The video preview runs at 30 fps (for smooth local preview); the send frame rate is reduced to 2 fps via Canvas.

Add media tracks to PeerConnection

Add audio track: 
localStream.getAudioTracks().forEach(t => {
  pc.addTrack(t, localStream);
  gatedAudioTracks.push(t);
});
Add video track (optional, with Canvas frame reduction to 2 fps): Canvas dimensions are obtained dynamically from the actual camera resolution rather than hard-coded: 
const sendFps = 2;
const settings = localStream.getVideoTracks()[0].getSettings();
sendCanvas = document.createElement("canvas");
sendCanvas.width = settings.width || 640;   // Dynamically obtain actual width
sendCanvas.height = settings.height || 480;  // Dynamically obtain actual height
sendCanvasCtx = sendCanvas.getContext("2d", { alpha: false });
sendCanvasStream = sendCanvas.captureStream(sendFps); // 2fps
const lowFpsTrack = sendCanvasStream.getVideoTracks()[0];
pc.addTrack(lowFpsTrack, sendCanvasStream);
gatedVideoTracks.push(lowFpsTrack);
// requestAnimationFrame loop: draw camera frames to Canvas
const pump = () => {
  if (!sendCanvasCtx || !sendCanvas) return;
  try { sendCanvasCtx.drawImage(localVideo, 0, 0, sendCanvas.width, sendCanvas.height); } catch {}
  sendRafId = requestAnimationFrame(pump);
};
sendRafId = requestAnimationFrame(pump);
Media gating (critical): After adding tracks, block sending immediately to ensure no media is pushed before session.created is received: 
// 1. Disable all track.enabled
gateMedia(false);  // track.enabled = false
// 2. Replace sender tracks with null to fully block sending
audioSender = pc.getSenders().find(s => s.track?.kind === 'audio');
videoSender = pc.getSenders().find(s => s.track?.kind === 'video');
audioTrack = audioSender?.track;
videoTrack = videoSender?.track;
await audioSender?.replaceTrack(null);
await videoSender?.replaceTrack(videoTrack ? null : undefined);
This is equivalent to enableSendMediaStream(false) in other SDKs. Sending must not be restored until session.created is received.

Create DataChannel

Create a DataChannel named oai-events to exchange session control events with the AI server. 
const dc = pc.createDataChannel('oai-events');
dc.onopen = () => console.log("DC open");
dc.onmessage = (e) => {
  handleDcMessage(e.data, dc);
};
// Also listen for DataChannels created by the server
pc.ondatachannel = (event) => {
  const ch = event.channel;
  ch.onmessage = (e) => {
    handleDcMessage(e.data, ch);
  };
};

Generate Offer SDP

Call createOffer() and set the local description. Wait for ICE gathering to complete before using the full Offer SDP. 
pc.onicegatheringstatechange = () => {
  if (!pc) return;
  if (pc.iceGatheringState === "complete" && pc.localDescription?.sdp) {
    const sdp = pc.localDescription.sdp;
    // ICE gathering complete, Offer SDP ready
    // Auto-generate curl command for users
  }
};
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);
Wait for iceGatheringState === "complete" before using the SDP. At that point, the SDP contains all ICE candidate information.

Exchange SDP (via curl command or AppServer)

Send the Offer SDP to the QwenCloud server and obtain the Answer SDP. In the demo, this is done via a curl command: 
curl -X POST 'https://dashscope-intl.aliyuncs.com/api/v1/webrtc/realtime?model=qwen3.5-omni-plus-realtime' \
  -H 'Content-Type: application/sdp' \
  -H 'Authorization: Bearer $DASHSCOPE_API_KEY' \
  --data-binary '<Offer SDP content>'
In production, proxy this step through an AppServer to avoid exposing the API key in the front end.

Set Answer SDP to establish connection

Set the Answer SDP returned by the server as the remote description to establish the WebRTC connection. Normalize the SDP format before setting it: 
function normalizeSdpForSetRemote(sdp) {
  sdp = String(sdp).trim().replace(/\r?\n/g, "\r\n");
  if (!sdp.endsWith("\r\n")) sdp += "\r\n";
  return sdp;
}
const answerSdp = normalizeSdpForSetRemote(txt);
await pc.setRemoteDescription({ type: 'answer', sdp: answerSdp });
The SDP specification requires \r\n line endings. normalizeSdpForSetRemote handles line ending compatibility from different sources.

Configure AI session (session.update)

After the connection is established, the server sends a session.created event through the DataChannel. Upon receiving it:
  1. Release the media gate to restore audio/video sending
  2. Send session.update to configure session parameters
Release gate and restore media: 
function handleDcMessage(data, channel) {
  let obj;
  try { obj = JSON.parse(data); } catch (err) { return; }
  if (obj?.type === "session.created") {
    // Release gate: restore track.enabled
    gateMedia(true);
    // Restore actual track to sender
    if (audioSender) audioSender.replaceTrack(audioTrack);
    if (videoSender && videoTrack) videoSender.replaceTrack(videoTrack);
    // Send session configuration
    sendUpdate(channel);
  }
}
session.update message body: 
const update = {
  event_id: `event_${Date.now()}`,
  type: "session.update",
  session: {
    input_audio_format: "pcm",
    input_audio_transcription: { model: "qwen3-asr-flash-realtime" },
    instructions: "You are a helpful assistant.",
    modalities: ["text", "audio"],
    output_audio_format: "pcm",
    smooth_output: false,
    turn_detection: {
      prefix_padding_ms: 500,
      silence_duration_ms: 800,
      threshold: 0.5,
      type: "server_vad",
    },
  },
};
if (channel && channel.readyState === "open") channel.send(JSON.stringify(update));
turn_detection.type can be set to server_vad (volume-based detection) or semantic_vad (semantic detection). Manual VAD is not supported in WebRTC mode.

Real-time conversation

After the connection is established, audio/video is transmitted in real time over RTP. Remote AI voice responses are received through the ontrack callback and played back. MediaRecorder is used to record responses for download. Receive and record remote audio: 
pc.ontrack = async (e) => {
  const stream = e.streams[0];
  ensureHiddenAudioEl();
  hiddenRemoteAudioEl.srcObject = stream;
  try { await hiddenRemoteAudioEl.play(); } catch {}
  startRecordingRemoteStream(stream); // Start recording
};
function startRecordingRemoteStream(remoteStream) {
  const audioTracks = remoteStream.getAudioTracks();
  if (!audioTracks.length) return;
  const audioStream = new MediaStream(audioTracks);
  recordedChunks = [ ];
  mediaRecorder = new MediaRecorder(audioStream, { mimeType: 'audio/webm' });
  mediaRecorder.ondataavailable = (e) => {
    if (e.data && e.data.size > 0) recordedChunks.push(e.data);
  };
  mediaRecorder.onstop = () => {
    audioBlob = new Blob(recordedChunks, { type: 'audio/webm' });
    // Available for download after recording stops
  };
  mediaRecorder.start();
}
DataChannel event display: All events sent and received through the DataChannel (including session.created, response.audio_transcript.done, and so on) are displayed in the event panel with support for expanding to view the full JSON: 
function pushEventFromDataChannel(eventObj) {
  const ts = eventObj.timestamp || nowTs();
  events.unshift({ event: eventObj, timestamp: ts });
  renderEvents();
}

End session and release resources

Release all resources in order when ending a call. The order is important: 
function endSession(silent = false) {
  // 1. Stop Canvas frame-rate reduction loop
  if (sendRafId) cancelAnimationFrame(sendRafId);
  sendRafId = 0;
  if (sendCanvasStream) sendCanvasStream.getTracks().forEach(t => t.stop());
  sendCanvasStream = null; sendCanvasCtx = null; sendCanvas = null;
  // 2. Stop recording
  try { if (mediaRecorder && mediaRecorder.state !== "inactive") mediaRecorder.stop(); } catch {}
  mediaRecorder = null;
  // 3. Stop local media stream
  if (localStream) {
    localStream.getTracks().forEach(t => t.stop());
    localStream = null;
  }
  // 4. Close PeerConnection
  if (pc) { try { pc.close(); } catch {} pc = null; }
  // 5. Clean up remote audio element
  if (hiddenRemoteAudioEl) {
    try { hiddenRemoteAudioEl.pause(); } catch {}
    hiddenRemoteAudioEl.srcObject = null;
    hiddenRemoteAudioEl.remove();
    hiddenRemoteAudioEl = null;
  }
}
After the session ends, use the Download remote audio button to save the AI response recording in WebM format.

Important notes

  1. Media gate must be released after session.created: Media data sent before the server sends session.created will be discarded. Use replaceTrack(null) to fully block sending until then.
  2. Video frame reduction is implemented via Canvas: Local preview runs at 30 fps; only 2 fps is sent to the server, controlled by captureStream(2). This conserves bandwidth.
  3. SDP format normalization: Ensure line endings are \r\n before setting the Answer SDP. Otherwise, setRemoteDescription may fail.
  4. Video is optional: If the user does not enable video, only microphone permission is requested -- no camera permission prompt appears.
  5. Remote audio is recorded automatically: MediaRecorder records the AI response audio stream. A WebM file is available for download after the session ends.
  6. WebRTC supports only server-side VAD: The manual mode is not supported. Use server_vad (volume detection) or semantic_vad (semantic detection).

Download the complete demo

Download the complete sample code: webrtc_demo.html.