Custom video input

Describes the two custom video input modes supported by the AOQ Client SDK -- raw frame mode and encoded frame mode -- including configuration and code examples for each.

Overview

The AOQ Client SDK's built-in video module covers basic video needs, but in some scenarios the built-in capture module may not be sufficient. Custom video capture is useful when you need to:

Work around camera device conflicts or compatibility issues.
Feed video data from a custom capture system or video file into the SDK for transmission.
Publish AI-generated frames, screen recordings, or virtual camera content through the SDK.

The AOQ Client SDK supports two custom video input modes:

Raw frame mode: Capture raw video frames in formats such as BGRA, I420, NV12, or NV21, then push them to the SDK via pushExternalVideoCapturedFrame. The SDK handles encoding and transmission internally.
Encoded frame mode: Encode video frames yourself (currently JPEG only), then push the encoded data directly to the SDK via pushExternalVideoEncodedFrame, bypassing the SDK's internal encoder.

Sample code

Coming soon.

Prerequisites

An engine instance has been created by calling createEngine.
A connection to the server has been established (the onConnectionStatusChange callback has reported AoqConnectionStatusConnected).

Implementation

Choose one of the two modes based on your use case. The two modes cannot be used simultaneously: only one push interface can be active at a time.

Mode 1: Raw frame mode

Capture raw video frames in formats such as BGRA, I420, NV12, or NV21 and push them to the SDK for encoding and transmission. The SDK handles the complete encoding and sending pipeline internally.

1. Configure video encoding parameters

The SDK's internal encoder encodes the raw frames you push. Adjust encoding parameters to suit your use case.

AoqClientEngine.AoqVideoCodecConfig config = new AoqClientEngine.AoqVideoCodecConfig();
config.width            = 1280;
config.height           = 720;
config.fps              = 2;
config.bitrate          = 500000;   // Starting bitrate: 500 kbps
config.minBitrate       = 128000;   // Minimum bitrate: 128 kbps
config.keyframeInterval = 2;
// Leave isExternal at the default false — the SDK handles encoding internally
engine.setVideoEncoderConfig(config);

Parameters:

Parameter	Type	Default	Description
trackType	AoqTrackType	AoqTrackTypeVideo	Video track type
codecType	AoqEncoderType	AoqEncoderTypeVideoH264	Encoder type
width	int	720	Encoding width (pixels)
height	int	1280	Encoding height (pixels)
fps	int	5	Frame rate
bitrate	int	500000	Starting bitrate (bps)
minBitrate	int	128000	Minimum bitrate (bps)
keyframeInterval	int	2	Keyframe interval (seconds)
isExternal	boolean	false	Keep false for raw frame mode
mirrorMode	AoqMirrorMode	AoqMirrorModeDisabled	Mirror mode
orientationMode	AoqOrientationMode	AoqOrientationModeAuto	Orientation mode

2. Start video capture in external capture mode

Call startVideoCapture with isExternal=true to tell the SDK not to open the camera and to expect frames from an external source. This is required for raw frame mode -- if you skip this call, the SDK will not consume any frames you push.

AoqClientEngine.AoqVideoCaptureConfig config = new AoqClientEngine.AoqVideoCaptureConfig();
config.isExternal = true;  // Do not open the camera; external source will push frames
// When isExternal=true, width/height/fps have no effect — the actual resolution
// and frame rate are determined by the pushed data
int ret = engine.startVideoCapture(config);

Parameters:

Parameter	Type	Default	Description
width	int	1280	Capture width (ignored when `isExternal=true`)
height	int	720	Capture height (ignored when `isExternal=true`)
fps	int	15	Capture frame rate (ignored when `isExternal=true`)
isExternal	boolean	false	true: do not open the camera; external source provides frames
cameraDirection	AoqCameraDirection	AoqCameraDirectionFront	Camera direction (ignored when `isExternal=true`)

3. Push raw video frames

Call pushExternalVideoCapturedFrame to feed captured raw frames into the SDK. The SDK handles encoding and transmission. Supported formats: BGRA, I420, NV12, NV21, RGBA. On Apple platforms, CVPixelBuffer zero-copy is also supported.

3.1 BGRA format

BGRA is a packed format with 4 bytes per pixel (Blue, Green, Red, Alpha). Frame size = width x height x 4 bytes.

// Build a BGRA video frame
AoqClientEngine.AoqVideoFrame frame = new AoqClientEngine.AoqVideoFrame();
frame.format    = AoqClientEngine.AoqVideoPixelFormat.AoqVideoPixelFormatBGRA;
frame.width     = 1280;
frame.height    = 720;
frame.data      = bgraBytes; // byte[], length = width * height * 4
frame.timeStamp = System.currentTimeMillis();
int ret = engine.pushExternalVideoCapturedFrame(
    AoqClientEngine.AoqTrackType.AoqTrackTypeVideo, frame);

3.2 I420 format

I420 is a planar format with three separate planes (Y, U, V). Y plane size = width x height; U and V planes are each (width/2) x (height/2).

// Build an I420 video frame
AoqClientEngine.AoqVideoFrame frame = new AoqClientEngine.AoqVideoFrame();
frame.format    = AoqClientEngine.AoqVideoPixelFormat.AoqVideoPixelFormatI420;
frame.width     = 1280;
frame.height    = 720;
frame.dataY     = yPlane;  // byte[], length = width * height
frame.dataU     = uPlane;  // byte[], length = (width/2) * (height/2)
frame.dataV     = vPlane;  // byte[], length = (width/2) * (height/2)
frame.strideY   = 1280;    // Y plane row stride (bytes)
frame.strideU   = 640;     // U plane row stride (bytes)
frame.strideV   = 640;     // V plane row stride (bytes)
frame.timeStamp = System.currentTimeMillis();
int ret = engine.pushExternalVideoCapturedFrame(
    AoqClientEngine.AoqTrackType.AoqTrackTypeVideo, frame);

3.3 NV12 / NV21 format

NV12 and NV21 are semi-planar formats consisting of a Y plane and an interleaved UV plane. NV12 interleaves UV in that order; NV21 interleaves VU. Frame size = width x height x 3 / 2 bytes, packed into the data field.

// Build an NV12 video frame (NV21 is identical — just change the format field)
AoqClientEngine.AoqVideoFrame frame = new AoqClientEngine.AoqVideoFrame();
frame.format    = AoqClientEngine.AoqVideoPixelFormat.AoqVideoPixelFormatNV12;
frame.width     = 1280;
frame.height    = 720;
frame.data      = nv12Bytes; // byte[], length = width * height * 3 / 2
frame.timeStamp = System.currentTimeMillis();
int ret = engine.pushExternalVideoCapturedFrame(
    AoqClientEngine.AoqTrackType.AoqTrackTypeVideo, frame);

3.4 CVPixelBuffer format (Apple platforms)

On iOS and macOS, you can pass a CVPixelBufferRef directly for zero-copy transfer, avoiding the performance overhead of memory copies.

// iOS / macOS
let frame = AoqVideoFrame()
frame.format      = .cvPixelBuffer
frame.width       = 1280
frame.height      = 720
frame.pixelBuffer = pixelBuffer  // CVPixelBufferRef
frame.timeStamp   = Int64(Date().timeIntervalSince1970 * 1000)
// The SDK holds pixelBuffer asynchronously — retain it with +1 ref count.
// The SDK releases it when done.
let _ = Unmanaged.passRetained(pixelBuffer)
engine.pushExternalVideoCapturedFrame(.video, frame: frame)

4. Stop raw frame capture

When you no longer need to push frames, stop the push timer first, then call stopVideoCapture to shut down video capture.

// 1. Stop the frame push timer
stopExternalFramePush();
// 2. Stop video capture
engine.stopVideoCapture();

Mode 2: Encoded frame mode

Encode video frames yourself (currently JPEG only) and push the encoded data directly to the SDK, bypassing the internal encoder. This mode does not require calling startVideoCapture or any other capture-related APIs.

1. Configure video encoding parameters and enable external encoding

Call setVideoEncoderConfig with isExternal=true to tell the SDK to skip the internal encoder and expect pre-encoded data from outside.

AoqClientEngine.AoqVideoCodecConfig config = new AoqClientEngine.AoqVideoCodecConfig();
config.width      = 1280;
config.height     = 720;
config.fps        = 2;
config.isExternal = true;  // Skip internal encoding; external source provides encoded frames
engine.setVideoEncoderConfig(config);

Once configured, you can push encoded frames immediately -- there is no need to call startVideoCapture.

2. Push encoded video frames

Call pushExternalVideoEncodedFrame to pass pre-encoded video data directly to the SDK. Only JPEG encoding is currently supported.

// Generate JPEG data from a Bitmap
android.graphics.Bitmap bmp = android.graphics.Bitmap.createBitmap(
    width, height, android.graphics.Bitmap.Config.ARGB_8888);
// ... fill in Bitmap content ...
java.io.ByteArrayOutputStream baos = new java.io.ByteArrayOutputStream();
bmp.compress(android.graphics.Bitmap.CompressFormat.JPEG, 85, baos);
bmp.recycle();
// Build the encoded frame and push it
AoqClientEngine.AoqVideoEncodedFrame frame = new AoqClientEngine.AoqVideoEncodedFrame();
frame.codec     = AoqClientEngine.AoqVideoCodecType.AoqVideoCodecTypeJPEG;
frame.data      = baos.toByteArray();
frame.width     = width;
frame.height    = height;
frame.timeStamp = System.currentTimeMillis();
int ret = engine.pushExternalVideoEncodedFrame(
    AoqClientEngine.AoqTrackType.AoqTrackTypeVideo, frame);

AoqVideoEncodedFrame parameters:

Parameter	Type	Default	Description
codec	AoqVideoCodecType	AoqVideoCodecTypeJPEG	Encoding format; currently only JPEG is supported
data	byte[]	null	Encoded frame data
width	int	0	Frame width (pixels)
height	int	0	Frame height (pixels)
timeStamp	long	0	Timestamp (milliseconds); when 0, the SDK uses the local clock

3. Stop pushing encoded frames

Encoded frame mode does not involve any capture device, so stopping is simply a matter of halting your push timer.

stopExternalFramePush();

Reference

AoqVideoFrame (raw frame mode)

Field	Type	Description
format	AoqVideoPixelFormat	Pixel format
width	int	Frame width (pixels)
height	int	Frame height (pixels)
data	byte[]	Packed format data (NV12/NV21/BGRA/RGBA)
dataY	byte[]	I420 Y plane data
dataU	byte[]	I420 U plane data
dataV	byte[]	I420 V plane data
strideY	int	I420 Y plane row stride (bytes)
strideU	int	I420 U plane row stride (bytes)
strideV	int	I420 V plane row stride (bytes)
textureId	int	Android texture ID (TextureOES/Texture2D)
transformMatrix	float[]	Texture transform matrix (4x4, row-major)
eglContext	EGLContext	Android shared EGL context (for texture mode)
pixelBuffer	CVPixelBufferRef	Apple zero-copy CVPixelBuffer (iOS/macOS only)
timeStamp	long	Timestamp (milliseconds); when 0, the SDK uses the local clock

AoqVideoPixelFormat enum values

Enum value	Numeric value	Description
AoqVideoPixelFormatUnknown	0	Unknown format
AoqVideoPixelFormatI420	1	I420 planar format
AoqVideoPixelFormatNV12	2	NV12 semi-planar format (UV interleaved)
AoqVideoPixelFormatNV21	3	NV21 semi-planar format (VU interleaved)
AoqVideoPixelFormatBGRA	4	BGRA packed format
AoqVideoPixelFormatRGBA	5	RGBA packed format
AoqVideoPixelFormatCVPixelBuffer	6	Apple CVPixelBuffer (iOS/macOS only)
AoqVideoPixelFormatTextureOES	7	Android OES external texture
AoqVideoPixelFormatTexture2D	8	Android 2D texture

AoqVideoEncodedFrame (encoded frame mode)

Field	Type	Description
codec	AoqVideoCodecType	Encoding format
data	byte[]	Encoded frame data
width	int	Frame width (pixels)
height	int	Frame height (pixels)
timeStamp	long	Timestamp (milliseconds); when 0, the SDK uses the local clock

AoqVideoCodecType enum values

Enum value	Numeric value	Description
AoqVideoCodecTypeJPEG	0	JPEG encoding format

Important notes

Raw frame mode: you must call startVideoCapture(isExternal=true) before pushing frames. If you skip this call, the SDK returns a parameter error.
Encoded frame mode: call setVideoEncoderConfig(isExternal=true) and push immediately -- calling startVideoCapture is not required.
Raw frame mode and encoded frame mode cannot be used simultaneously: only one push interface can be active at a time.
Encoded frame mode currently supports JPEG only.
After you push a frame, the SDK manages its lifecycle internally. You do not need to retain the data after the push call returns.

​Overview

​Sample code

​Prerequisites

​Implementation

​Mode 1: Raw frame mode

​1. Configure video encoding parameters

​2. Start video capture in external capture mode

​3. Push raw video frames

​3.1 BGRA format

​3.2 I420 format

​3.3 NV12 / NV21 format

​3.4 CVPixelBuffer format (Apple platforms)

​4. Stop raw frame capture

​Mode 2: Encoded frame mode

​1. Configure video encoding parameters and enable external encoding

​2. Push encoded video frames

​3. Stop pushing encoded frames

​Reference

​AoqVideoFrame (raw frame mode)

​AoqVideoPixelFormat enum values

​AoqVideoEncodedFrame (encoded frame mode)

​AoqVideoCodecType enum values

​Important notes