The AOQ Client SDK provides comprehensive video capabilities, covering video capture, rendering, codec configuration, frame data callbacks, and external video input. This document introduces common video features across Android (Java), iOS (Objective-C), and HarmonyOS (ArkTS).
1. Video capture
1.1 Overview
Video capture opens the device camera and feeds real-time video frames into the SDK encoding pipeline. The SDK supports two capture modes:
- Internal capture (default): The SDK automatically manages the camera -- opening it, capturing frames, and closing it. Supports switching between front and rear cameras.
- External capture: The application manages the camera or other video source. Captured frames are fed into the SDK via
pushExternalVideoCapturedFrame.
1.2 Capture configuration parameters
| Parameter | Type | Default | Description |
|---|
| width | int | 1280 | Capture width in pixels. Not used in external capture mode. |
| height | int | 720 | Capture height in pixels. Not used in external capture mode. |
| fps | int | 15 | Capture frame rate. In external capture mode, the frame rate is determined by how fast frames are pushed. |
| isExternal | bool | false | Whether to use external capture mode. |
| cameraDirection | AoqCameraDirection | Front(0) | Camera direction. Not used in external capture mode. |
1.3 Camera direction enum
| Enum value | Number | Description |
|---|
| AoqCameraDirectionFront | 0 | Front-facing camera |
| AoqCameraDirectionBack | 1 | Rear-facing camera |
1.4 API reference
| Function | Android | iOS | HarmonyOS |
|---|
| Start capture | startVideoCapture(config) | startVideoCapture:config: | startVideoCapture(config) |
| Stop capture | stopVideoCapture() | stopVideoCapture | stopVideoCapture() |
| Switch camera | switchCamera(direction) | switchCamera: | switchCamera(direction) |
1.5 Example
AoqVideoCaptureConfig config = new AoqVideoCaptureConfig();
config.width = 1280;
config.height = 720;
config.fps = 15;
config.cameraDirection = AoqCameraDirection.AoqCameraDirectionFront;
engine.startVideoCapture(config);
AoqVideoCaptureConfig *config = [[AoqVideoCaptureConfig alloc] init];
config.width = 1280;
config.height = 720;
config.fps = 15;
config.cameraDirection = AoqCameraDirectionFront;
[engine startVideoCapture:config];
let config: AoqVideoCaptureConfig = {
width: 1280,
height: 720,
fps: 15,
cameraDirection: AoqCameraDirection.AoqCameraDirectionFront
};
engine.startVideoCapture(config);
2. Video rendering
2.1 Overview
Video rendering displays locally captured or remotely received video frames on screen. The SDK supports setting a local preview window and a remote rendering window. Use trackType to distinguish between the video stream (Video) and the screen share stream (Screen).
2.2 Render modes
| Enum value | Number | Description |
|---|
| AoqRenderModeAuto | 0 | Automatic mode |
| AoqRenderModeStretch | 1 | Stretch to fill. The image may be distorted. |
| AoqRenderModeFill | 2 | Fit with letterboxing. The full image is shown. |
| AoqRenderModeCrop | 3 | Crop mode. Parts of the image may be cut off. |
2.3 Canvas configuration
| Parameter | Type | Default | Description |
|---|
| view | Platform view | null | Rendering view. Android: SurfaceView or TextureView. iOS: UIView. HarmonyOS: XComponent. |
| renderMode | AoqRenderMode | Auto(0) | Render display mode |
2.4 API reference
| Function | Android | iOS | HarmonyOS |
|---|
| Set local preview | setLocalView(trackType, canvas) | setLocalView:trackType:canvas: | setLocalView(trackType, canvas) |
| Set remote view | setRemoteView(trackType, canvas) | setRemoteView:trackType:canvas: | setRemoteView(trackType, canvas) |
Platform differences: Android uses SurfaceView or TextureView as the rendering container. iOS uses UIView (wrapped internally by AoqRenderView with Metal acceleration). HarmonyOS uses XComponent (managed by AoqXComponentController for native rendering).
3. Video codec configuration
3.1 Overview
Configure video encoding parameters including encoding format, resolution, frame rate, bitrate, keyframe interval, mirroring, and orientation. Use trackType to apply different encoding configurations to the video track and the screen share track.
3.2 Encoding configuration parameters
| Parameter | Type | Default | Description |
|---|
| trackType | AoqTrackType | Video(1) | Track type: Video |
| codecType | AoqEncoderType | VideoH264(3) | Encoding format |
| width | int | 720 | Encoded width |
| height | int | 1280 | Encoded height |
| fps | int | 5 | Encoding frame rate |
| bitrate | int | 500000 | Target bitrate (bps) |
| minBitrate | int | 128000 | Minimum bitrate (bps) |
| keyframeInterval | int | 2 | Keyframe interval (seconds) |
| mirrorMode | AoqMirrorMode | Disabled(0) | Mirror mode |
| orientationMode | AoqOrientationMode | Auto(0) | Orientation mode |
| isExternal | bool | false | External encoding mode. When true, the application pushes pre-encoded frames. |
| Enum value | Number | Description |
|---|
| AoqEncoderTypeVideoH264 | 3 | H.264 encoding |
| AoqEncoderTypeVideoJpeg | 4 | JPEG encoding (for external encoded frames) |
3.4 Mirror mode
| Enum value | Number | Description |
|---|
| AoqMirrorModeDisabled | 0 | Mirroring disabled |
| AoqMirrorModeEnabled | 1 | Mirroring enabled |
3.5 Orientation mode
| Enum value | Number | Description |
|---|
| AoqOrientationModeAuto | 0 | Automatic orientation |
| AoqOrientationModePortrait | 1 | Portrait orientation |
| AoqOrientationModeLandscape | 2 | Landscape orientation |
3.6 API reference
| Function | Android | iOS | HarmonyOS |
|---|
| Set encoding config | setVideoEncoderConfig(config) | setVideoEncoderConfig: | setVideoEncoderConfig(config) |
4.1 Overview
External video frame input lets your application push custom video frame data into the SDK for external capture or external encoding scenarios. Two push methods are supported:
- Push raw frames: Push unencoded pixel data (I420, NV12, NV21, BGRA, RGBA, and similar formats) to the SDK. The SDK encodes the data.
- Push encoded frames: Push already-encoded data (for example, JPEG) directly to the SDK. The SDK packages and sends it without re-encoding.
Routing is controlled by trackType: AoqTrackTypeVideo routes to the video capture track; AoqTrackTypeScreen routes to the screen share track.
| Enum value | Number | Description | Platform support |
|---|
| AoqVideoPixelFormatI420 | 1 | I420 tri-planar | All platforms |
| AoqVideoPixelFormatNV12 | 2 | NV12 bi-planar | All platforms |
| AoqVideoPixelFormatNV21 | 3 | NV21 bi-planar | All platforms |
| AoqVideoPixelFormatBGRA | 4 | BGRA packed | All platforms |
| AoqVideoPixelFormatRGBA | 5 | RGBA packed | All platforms |
| AoqVideoPixelFormatCVPixelBuffer | 6 | Apple zero-copy | iOS only |
| AoqVideoPixelFormatTextureOES | 7 | OES texture | Android only |
| AoqVideoPixelFormatTexture2D | 8 | 2D texture | Android only |
4.3 Raw video frame data structure (AoqVideoFrame)
| Field | Type | Description |
|---|
| format | AoqVideoPixelFormat | Pixel format |
| width | int | Width in pixels |
| height | int | Height in pixels |
| data | byte[] / ArrayBuffer | Packed format data (NV12/NV21/BGRA/RGBA) |
| dataY / dataU / dataV | byte[] / ArrayBuffer | I420 tri-planar data |
| strideY / strideU / strideV | int | I420 tri-planar strides |
| textureId | int | Texture ID (valid for Android TextureOES/Texture2D) |
| transformMatrix | float[16] | 4x4 texture transform matrix (Android) |
| eglContext | EGLContext | Shared EGL context (Android) |
| pixelBuffer | CVPixelBufferRef | Apple zero-copy (iOS) |
| timeStamp | long | Timestamp in ms. If 0, the SDK uses the local clock. |
4.4 Encoded video frame data structure (AoqVideoEncodedFrame)
| Field | Type | Default | Description |
|---|
| codec | AoqVideoCodecType | JPEG(0) | Encoding format |
| data | byte[] / ArrayBuffer | - | Encoded data |
| width | int | - | Width in pixels |
| height | int | - | Height in pixels |
| timeStamp | long | 0 | Timestamp in ms |
4.5 API reference
| Function | Android | iOS | HarmonyOS |
|---|
| Push raw frame | pushExternalVideoCapturedFrame(trackType, frame) | pushExternalVideoCapturedFrame:frame: | pushExternalVideoCapturedFrame(trackType, frame) |
| Push encoded frame | pushExternalVideoEncodedFrame(trackType, frame) | pushExternalVideoEncodedFrame:frame: | pushExternalVideoEncodedFrame(trackType, frame) |
5. Video frame callbacks
5.1 Overview
Video frame callbacks let you obtain raw frame data at different points in the video pipeline, for use in video analysis, custom processing, recording, and similar scenarios. Both read-only and read-write modes are supported. In read-write mode, you can modify the frame data and write it back to the SDK.
5.2 Supported data source positions
| Data source | Enum value | Description |
|---|
| Captured | 0 | Video data after capture, before pre-processing |
| PreEncode | 1 | Video data before encoding, after pre-processing |
| Remote | 2 | Remote video data after decoding, before rendering |
5.3 Callback configuration parameters
| Parameter | Type | Default | Description |
|---|
| format | AoqVideoPixelFormat | I420(1) | Pixel format for the callback data |
| alignment | AoqVideoObserverAlignment | Default(0) | Width alignment policy |
| mode | AoqVideoObserverMode | ReadOnly(0) | Read-only (0) or read-write (1) mode |
| mirrorApplied | bool | false | Whether to apply mirroring to the callback data |
5.4 Width alignment enum
| Enum value | Number | Description |
|---|
| AoqVideoObserverAlignmentDefault | 0 | Default alignment |
| AoqVideoObserverAlignmentEven | 1 | 2-byte alignment |
| AoqVideoObserverAlignment4 | 2 | 4-byte alignment |
| AoqVideoObserverAlignment8 | 3 | 8-byte alignment |
| AoqVideoObserverAlignment16 | 4 | 16-byte alignment |
5.5 Usage steps
- Register the observer: Call
setVideoFrameObserver to set the video frame callback listener.
- Enable the data source: Call
enableVideoFrameObserver to select the data source position and start callbacks.
- Handle callback data: Obtain frame data inside the callback. Frame data is only valid during the callback. Copy it if you need it asynchronously.
5.6 API reference
| Function | Android | iOS | HarmonyOS |
|---|
| Register observer | setVideoFrameObserver(listener) | setVideoFrameObserver: | setVideoFrameObserver(observer) |
| Enable callbacks | enableVideoFrameObserver(enabled, source, config) | enableVideoFrameObserver:videoSource:config: | enableVideoFrameObserver(enabled, source, config) |
5.7 Callback methods
| Callback | Android | iOS | HarmonyOS |
|---|
| Captured data | onCapturedVideoFrame(frame) | onCapturedVideoFrame: | onCapturedVideoFrame(frame) |
| Pre-encode data | onPreEncodeVideoFrame(trackType, frame) | onPreEncodeVideoFrame:frame: | onPreEncodeVideoFrame(trackType, frame) |
| Remote data | onRemoteVideoFrame(trackType, frame) | onRemoteVideoFrame:frame: | onRemoteVideoFrame(trackType, frame) |
Returning true/YES from a callback indicates that the data has been modified and should be written back to the SDK. This only takes effect in ReadWrite mode with I420 format.
7.1 Overview
Control whether local media streams are sent. Use trackType to target a specific track (Audio, Video, or Screen). When sending is disabled, capture and encoding continue, but data is not sent to the remote end.
7.2 API reference
| Function | Android | iOS | HarmonyOS |
|---|
| Control stream sending | enableSendMediaStream(trackType, enable) | enableSendMediaStream:enable: | enableSendMediaStream(trackType, enable) |
7.3 Track type enum
| Enum value | Number | Description |
|---|
| AoqTrackTypeAudio | 0 | Audio track |
| AoqTrackTypeVideo | 1 | Video track |
| AoqTrackTypeData | 2 | Data track |
8. Video device state monitoring
8.1 Overview
The SDK automatically monitors camera state changes and notifies the application layer through the onVideoDeviceStateChanged callback.
8.2 Device state codes
| State code | Value | Description |
|---|
| AoqVideoDeviceNone | 0 | Initial state |
| AoqVideoDeviceCaptureStarting | 1 | Capture starting |
| AoqVideoDeviceCaptureStarted | 2 | Capture started |
| AoqVideoDeviceCaptureStopping | 3 | Capture stopping |
| AoqVideoDeviceCaptureStopped | 4 | Capture stopped |
| AoqVideoDeviceCaptureFail | 5 | Capture failed |
8.3 Callback reference
| Callback | Android | iOS | HarmonyOS |
|---|
| Device state change | onVideoDeviceStateChanged(state) | onVideoDeviceStateChanged: | onVideoDeviceStateChanged(state) |
9. Video error and warning codes
9.1 Video error codes
| Error code | Value | Description |
|---|
| AoqErrorCodeVideo | 200 | General video error |
| VideoExternalBufferFull | 210 | Video external buffer full |
| VideoDevice | 220 | General video device error |
| CameraOpenFail | 221 | Failed to open camera |
| CameraAuthFailed | 222 | Camera permission denied |
| CameraOccupied | 223 | Camera in use by another process |
| CameraRunningError | 224 | Camera running error |
| VideoCodec | 230 | General video codec error |
| EncoderInitFail | 231 | Encoder initialization failed |
| VideoRender | 240 | General video rendering error |
| RenderCreateFail | 241 | Renderer creation failed |
| RenderDrawError | 242 | Rendering draw error |
| Screen | 300 | General screen share error |
Additional Android error codes: ScreenPermissionDenied(310) -- screen share permission denied; ScreenForegroundServiceFailed(311) -- foreground service failed to start.
9.2 Video warning codes
| Warning code | Value | Description |
|---|
| AoqWCVideo | 200 | General video warning |
| CameraEnumerateError | 201 | Camera enumeration error |
| EncoderSwitched | 202 | Encoder switched warning |
| RenderDowngrade | 203 | Render downgrade warning |
Appendix: Complete video API method list
| Category | Method | Description |
|---|
| Capture control | startVideoCapture | Open video capture device |
| Capture control | stopVideoCapture | Close video capture device |
| Capture control | switchCamera | Switch between front and rear camera |
| Rendering control | setLocalView | Set local preview window |
| Rendering control | setRemoteView | Set remote rendering window |
| Codec | setVideoEncoderConfig | Set video encoding parameters |
| External input | pushExternalVideoCapturedFrame | Push raw video frame |
| External input | pushExternalVideoEncodedFrame | Push encoded video frame |
| Screen share | startScreenCapture | Start screen capture |
| Screen share | stopScreenCapture | Stop screen capture |
| Stream control | enableSendMediaStream | Control media stream sending |
| Frame callbacks | setVideoFrameObserver | Register video frame observer |
| Frame callbacks | enableVideoFrameObserver | Enable or disable video frame callbacks |
