Skip to main content
QwenCloud home page
Docs
Quick Start

Connect to a model

Complete workflow for connecting to a model via AOQ, WebRTC, or WebSocket

Describes how to connect to Realtime API models or applications via AOQ, WebRTC, and WebSocket, including connection flows, sequence diagrams, and code samples for each protocol.

AOQ connection

AOQ is deeply customized on the QUIC protocol, optimized for native mobile apps. It supports mixed audio/video/data transport with exceptional poor-network resilience. The following example uses an iOS demo.

Overall sequence diagram

AOQ overall sequence diagram

Create engine and set callbacks

let config = AoqCreateConfig()
config.workDir = workDir
config.enableDumpAudio = false
engine = AoqClientEngine.createEngine(config, delegate: self)
Implement the AoqEngineDelegate protocol to listen for callbacks including onConnectionStatusChange, onDataMsg, and onError.

Start audio capture and playback

// Audio capture
let capCfg = AoqAudioCaptureConfig()
capCfg.channel = 1; capCfg.isExternal = false
engine.startAudioCapture(capCfg)
// Audio playback
let playCfg = AoqAudioPlaybackConfig()
playCfg.channel = 1; playCfg.isExternal = false
engine.startAudioPlayer(playCfg)
// Video capture (optional)
let vidCfg = AoqVideoCaptureConfig()
vidCfg.width = 720; vidCfg.height = 1280; vidCfg.fps = 15
engine.startVideoCapture(vidCfg)

Get connection credentials

Have your AppServer proxy the request to QwenCloud. See the Token authentication section.

Configure codecs and establish connection

Set codec parameters and call connect: 
// Audio codec configuration
let encCfg = AoqAudioCodecConfig()
encCfg.codecType = .audioPCM; encCfg.sampleRate = 16000; encCfg.channel = 1
engine.setAudioEncoderConfig(encCfg)
engine.setAudioDecoderConfig(encCfg)
// Disable media sending before connect; enable after receiving session.updated
engine.enableSendMediaStream(.audio, enable: false)
let config = AoqConnectConfig()
config.token = token
config.sid = sid
config.certFingerprint = certificate
config.relayEndpoints = relayEndpoints
config.workspaceIdHash = workspaceIdHash
config.publishTracks = [audioTrack, dataTrack]
config.subscribeTracks = [audioTrack, dataTrack]
engine.connect(config)
The AOQ SDK starts sending media data by default after connecting. This example demonstrates how to disable media sending when connecting to a model.

Configure AI session

Example of sending session.update after a successful connection. For details, see Model client events reference: 
func onConnectionStatusChange(_ status: AoqConnectionStatus) {
  if status == .connected { sendSessionUpdate() }
}
private func sendSessionUpdate() {
  let json = """
  {
    // Event ID, generated by the client
    "event_id": "event_ToPZqeobitzUJnt3QqtWg",
    // Event type, always session.update
    "type": "session.update",
    // Session configuration
    "session": {
        // Output modalities: ["text"] (text only) or ["text","audio"] (text and audio)
        "modalities": [
            "text",
            "audio"
        ],
        // Output audio voice
        "voice": "Ethan",
        // Input audio format; currently only "pcm" is supported. Input audio is 16 kHz PCM.
        "input_audio_format": "pcm",
        // Output audio format; currently only "pcm" is supported. Output audio is 24 kHz PCM.
        "output_audio_format": "pcm",
        // System message for setting the model's objective or role.
        "instructions": "You are an AI customer service agent for a five-star hotel. Accurately and helpfully answer customer inquiries about room types, facilities, pricing, and reservation policies. Always respond professionally and helpfully. Do not provide unverified information or information outside the scope of hotel services.",
        // Whether to enable voice activity detection. If enabled, pass a config object and the server will automatically detect voice start and end.
        // Set to null to let the client control when to trigger a model response.
        "turn_detection": {
            // VAD type: server_vad or semantic_vad. Recommended: semantic_vad for qwen3.5-omni-realtime.
            "type": "semantic_vad",
            // VAD detection threshold. Increase in noisy environments; decrease in quiet ones.
            "threshold": 0.5,
            // Duration of silence after which a model response is triggered
            "silence_duration_ms": 800
        }
    }
  }
  """
  let msg = AoqDataMsg()
  msg.data = json.data(using: .utf8)!
  engine.send(msg)
}

Enable media sending after receiving session.updated

Example of handling the session.updated event from the model. For details, see Model server events reference: 
func onDataMsg(_ msg: AoqDataMsg) {
  guard let obj = try? JSONSerialization.jsonObject(with: msg.data) as? [String: Any],
        let type = obj["type"] as? String else { return }
  if type == "session.updated" {
    engine.enableSendMediaStream(.audio, enable: true)
    engine.enableSendMediaStream(.video, enable: true)
  }
}
  1. Start sending media streams only after receiving session.updated. The AI may not be ready to receive data before this event.
  2. The audio and video tracks added during connection (AOQ media channels) automatically deliver data to the server.
    • Audio: transmitted directly through the audio track -- no need to send input_audio_buffer.append events.
    • Video: frames are sent through the video track -- no need to send input_image_buffer.append events.

Disconnect and destroy engine

engine.disconnect()
AoqClientEngine.destroy()

WebRTC connection

WebRTC does not provide an SDK. On the web, connect using JavaScript. On other platforms, use an open-source library or a third-party RTC provider that supports the standard WebRTC protocol. The following example uses JavaScript on the web.

Overall flow diagram

WebRTC overall flow diagram

Establish connection

# pip install aiortc aiohttp certifi
import asyncio, aiohttp, ssl, certifi
from aiortc import RTCPeerConnection, RTCConfiguration, RTCSessionDescription
from aiortc.mediastreams import AudioStreamTrack
API_KEY = "your-api-key"
MODEL = "{model_name}"
SIGNALING_URL = f"https://dashscope-intl.aliyuncs.com/api/v1/webrtc/realtime?model={MODEL}"
async def connect():
  pc = RTCPeerConnection(RTCConfiguration(iceServers=[]))
  # Add audio track to ensure the Offer SDP includes m=audio (required by server)
  pc.addTrack(AudioStreamTrack())
  # Create a DataChannel to trigger SDP negotiation (name can be customized; server pushes events via the "txt" channel)
  pc.createDataChannel("oai-events")
  # SDP exchange: create an Offer and send it to the server
  offer = await pc.createOffer()
  await pc.setLocalDescription(offer)
  async with aiohttp.ClientSession() as session:
    async with session.post(
      SIGNALING_URL,
      ssl=ssl.create_default_context(cafile=certifi.where()),
      data=offer.sdp.encode("utf-8"),
      headers={
        "Content-Type": "application/sdp",
        "Authorization": f"Bearer {API_KEY}",
      },
    ) as resp:
      if not resp.ok:
        raise Exception(f"SDP exchange failed: {resp.status} {await resp.text()}")
      answer_sdp = await resp.text()
  print("=== Offer SDP ===")
  print(offer.sdp)
  print("=== Answer SDP ===")
  print(answer_sdp)
  # ICE connection completes automatically
  await pc.setRemoteDescription(RTCSessionDescription(sdp=answer_sdp, type="answer"))
  print("WebRTC connection established")
  return pc

Configure model parameters

Listen for DataChannel messages from the model to ensure correct interaction ordering: 
pc.ondatachannel = (event) => {
  const ch = event.channel;
  ch.onmessage = (e) => {
    let obj;
    try { obj = JSON.parse(e.data); }
    catch (err) {
      return;
    }
    if (obj?.type === "session.created") {
      sendUpdate(event.channel);
      // Start sending audio/video
      audioSender?.replaceTrack(audioTrack);
      videoSender?.replaceTrack(videoTrack);
    }
  };
};

Send and receive media data

The audio and video tracks added during connection (RTP media channels) automatically deliver data to the server.
  • Audio: transmitted directly through the audio track (RTP) -- no need to send input_audio_buffer.append events.
  • Image: frames are sent through the video track (RTP). The input_image_buffer.append event is not supported.
WebRTC supports only server-side VAD mode (server_vad or semantic_vad). Manual mode is not supported.

Demo source code

Prerequisites

  • Use a modern browser that supports WebRTC (Chrome, Edge, Firefox, Safari, and so on).
  • The browser must have microphone permission.
  • Browsers cannot directly send connection requests to the server due to cross-origin security restrictions. Use a terminal to run the curl command to establish the connection.

Run the demo

Create an HTML file named webrtc_demo.html and paste the following code into it: webrtc_demo.html. Open the file in a browser and follow these steps:
  1. Click Start session. The page generates an Offer SDP and the corresponding curl command.
  2. Click Copy curl command and run it in a terminal. The output is the Answer SDP.
  3. Paste the Answer SDP into the Answer SDP text box on the page and click Set Answer to establish the connection and start the voice conversation.

WebSocket connection

You can connect via the DashScope SDK or the model API. For details, see: