AI - Qualcomm Intelligent Multimedia SDK

This section covers QIM SDK AI sample applications that demonstrate vision, audio, and multi-model inference on Qualcomm platforms using LiteRT models accelerated on Qualcomm AI hardware. The following tables list all available GStreamer C/C++ AI applications and their platform support. Select the appropriate configuration tab for your setup.

Config #1
Config #2

Application	Source code	Description	Input sources	QCS6490	IQ-8275	IQ-9075	IQ-615
Image classification	`gst-ai-classification`	Classification on streams from a file source or RTSP.	Camera, file, RTSP, USB	✓	✓	✓	✓
Object detection	`gst-ai-object-detection`	Object detection on streams from a file source or RTSP.	Camera, file, RTSP, USB	✓	✓	✓	×
Pose detection	`gst-ai-pose-detection`	Pose detection on streams from a file source or RTSP.	File, RTSP, USB	✓	✓	✓	✓
Image segmentation	`gst-ai-segmentation`	Image segmentation on streams from a file source or RTSP.	File, RTSP	✓	✓	✓	×
Daisy chain detection + classification	`gst-ai-daisychain-detection-classification`	Cascaded object detection and classification.	File, RTSP, USB	✓	✓	✓	×
Daisy chain detection + pose	`gst-ai-daisychain-detection-pose`	Cascaded object detection and pose detection.	File, RTSP, USB	✓	✓	✓	×
Monodepth	`gst-ai-monodepth`	Monocular depth estimation from file or RTSP.	File, RTSP	✓	✓	✓	×
Face detection	`gst-ai-face-detection`	Face detection from file or RTSP.	File, RTSP	✓	✓	✓	✓
Audio classification	`gst-ai-audio-classification`	Audio event classification from microphone or file.	Audio, file	✓	✓	✓	✓
Metadata parsing	`gst-ai-metadata-parser-example`	Parse ML metadata and count people from file or RTSP.	File, RTSP	✓	✓	✓	×
AI USB camera	`gst-ai-usb-camera-app`	USB camera streaming with optional object detection.	USB	✓	✓	✓	×
AI event encoder	`gst-ai-event-encoder`	Encode video only when a person is detected.	File, RTSP	✓	✓	✓	×

Application	Source code	Description	Input sources	QCS6490	IQ-8275	IQ-9075	IQ-615
Image classification	`gst-ai-classification`	Classification on streams from a camera, file, or RTSP.	Camera, file, RTSP, USB	✓	✓	✓	✓
Object detection	`gst-ai-object-detection`	Object detection on streams from a camera, file, or RTSP.	Camera, file, RTSP, USB	✓	✓	✓	×
Pose detection	`gst-ai-pose-detection`	Pose detection on streams from camera, file, or RTSP.	Camera, file, RTSP, USB	✓	✓	✓	✓
Image segmentation	`gst-ai-segmentation`	Image segmentation on streams from camera, file, or RTSP.	Camera, file, RTSP	✓	✓	✓	×
Parallel inferencing	`gst-ai-parallel-inference`	Run multiple AI models simultaneously on one stream.	Camera, file, RTSP	✓	✓	✓	×
Multi input/output object detection	`gst-ai-multi-input-output-object-detection`	Object detection across multiple I/O stream combinations.	Camera, file, RTSP	✓	✓	✓	×
Daisy chain detection + classification	`gst-ai-daisychain-detection-classification`	Cascaded object detection and classification.	Camera, file, RTSP, USB	✓	✓	✓	×
Daisy chain detection + pose	`gst-ai-daisychain-detection-pose`	Cascaded object detection and pose detection.	Camera, file, RTSP, USB	✓	✓	✓	×
Monodepth	`gst-ai-monodepth`	Monocular depth estimation from camera, file, or RTSP.	Camera, file, RTSP	✓	✓	✓	×
Video super-resolution	`gst-ai-superresolution`	Upscale low-resolution video to high resolution.	File	✓	✓	✓	×
Multistream inference	`gst-ai-multistream-inference`	AI inference across multiple simultaneous streams.	Camera, file, RTSP	✓	✓	✓	×
Multistream batch inference	`gst-ai-multistream-batch-inference`	Batched AI inference on up to 24 file streams.	File	✓	✓	✓	×
Face detection	`gst-ai-face-detection`	Face detection from camera, file, or RTSP.	Camera, file, RTSP	✓	✓	✓	✓
Face recognition	`gst-ai-face-recognition`	Face recognition from camera or RTSP.	Camera, RTSP	✓	✓	✓	×
Audio classification	`gst-ai-audio-classification`	Audio event classification from microphone or file.	Audio, file	✓	✓	✓	✓
Metadata parsing	`gst-ai-metadata-parser-example`	Parse ML metadata and count people.	Camera, file, RTSP	✓	✓	✓	×
AI USB camera	`gst-ai-usb-camera-app`	USB camera streaming with optional object detection.	USB	✓	✓	✓	×
Image segmentation using Python with container	`snpe_segmentation_app.py`	Segmentation using python bindings from a Docker.	Image	✓	✓	✓	×
Hardware benchmarking Application	`QCS6490-Vision-AI-Demo`	A GUI application to monitor hardware utilization with sample applications	Camera and USB	✓	✓	✓	×
AI event encoder	`gst-ai-event-encoder`	Encode video only when a person is detected.	Camera, file, RTSP	✓	✓	✓	×

Prerequisites

Try me

Some of the steps in the pre-requisties will be removed from future releases once the necessary fixes are mainlined.

Set up Wi-Fi

Connect to the Wireless Access Point (Wi-Fi Router):

nmcli dev wifi connect <WiFi-SSID> password <WiFi-password>

Check the connection and device status:

nmcli -p device

Login to the target deviceLocate the IP address of the device according to the type of network connection, using the UART console on the Linux host:For Ethernet:

ip address show eth2

For Wi-Fi:

ip address show wlp1s0

Use the IP address from the Linux host to establish an SSH connection to the device:

ssh root@<ip-address>

Example:

ssh root@192.168.0.222

Connect to the SSH shell using the following password:

oelinux123

Download Models and Artifacts

On the target device, obtain the download_artifacts.sh script, set executable permissions, and run it to download the model, media, and label files:

cd /tmp/
curl -L -O https://raw.githubusercontent.com/qualcomm/sample-apps-for-qualcomm-linux/refs/heads/main/qualcomm-linux/scripts/download_artifacts.sh
chmod +x download_artifacts.sh
./download_artifacts.sh

Enable qticamsrc

In the terminal of the target device, run the following command to enable the qticamsrc on Config #2:

echo -n "camx" > /var/data
efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -w -f /var/data
efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -p
sync
reboot

Enable Audio and GPU Delegate

In the terminal of the target device, run the following command to enable audio:

systemctl stop pipewire wireplumber pipewire.socket pipewire-manager.socket
chmod 777 /dev/dma_heap/system
adsprpcd audiopd &
systemctl start pipewire wireplumber
wpctl status

To set the default devices for sink and source, get the device numbers from wpctl status and run the following command:

wpctl set-default <device ID>

In the terminal of the target device, run the following command to enable the GPU delegate and backend:

mount -o rw,remount /

export OCL_ICD_FILENAMES=/usr/lib/libOpenCL_adreno.so.1

AI Vision Applications

Object Detection

The gst-ai-object-detection application allows you to detect objects within images and videos. The use cases show the execution of YOLOv5, YOLOv8 and YOLOX on Qualcomm AI HW accelerator. The following figure shows the pipeline, which receives the input from a live camera feed, file, USB source, or an RTSP stream, preprocesses it, runs inferences on AI hardware. The results are either displayed on the screen, saved as an encoded MP4 file, or streamed over the RTSP server. For information about the plugins used in the pipeline flow, see Pipeline flow

Application: gst-ai-object-detection

Try me

When the software image includes the qticamsrc plugin, the camera framework uses it by default. If absent, the framework switches to libcamera instead. Since Config #1 lacks support for qticamsrc, the system defaults to libcamera.

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	IMX577 camera	File output	Display	RTSP output
Config #1	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes
Config #2	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model file	Label file
Qualcomm Neural Processing SDK	`yolonas.dlc`	`yolonas.json`
LiteRT	`yolov8_det_w8a8.tflite` / `yolox_quantized.tflite`	`yolox.json`
Qualcomm AI Engine Direct	`yolov8_det_quantized.bin`	`yolov8.json`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-object-detection --config-file=/etc/configs/config_detection.json

The sample application uses the /etc/configs/config_detection.json file to read the input parameters.To display all available options:

gst-ai-object-detection -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-object-detection application uses the /etc/configs/config_detection.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<path-to-input-video>",
  "ml-framework": "<snpe or tflite or qnn framework>",
  "yolo-model-type": "<yolov8 or yolonas or yolov5 or yolox>",
  "model": "<path-to-model-file>",
  "labels": "<path-to-label-file>",
  "threshold": "<post processing threshold, integer value from 1 to 100>",
  "runtime": "<dsp, gpu, cpu runtime>",
  "output-type": "waylandsink or filesink or rtspsink",
  "snpe-tensors": "<model output tensor name>"
}

For USB camera input, set the video-format, resolution, and framerate parameters in the config file to match the camera capabilities, see Configure USB camera.

The snpe-tensors field applies only to the SNPE runtime. To retrieve the output tensor names for a DLC model, open the model in Netron.

When using DLC models from the AI Hub, the snpe-tensors field is optional.

Available configurations

Config #2
Config #1

Camera source, LiteRT model, DSP runtime

{
  "camera": 0,
  "ml-framework": "tflite",
  "yolo-model-type": "yolox",
  "model": "/etc/models/yolox_quantized.tflite",
  "labels": "/etc/labels/yolox.json",
  "threshold": 40,
  "runtime": "dsp",
  "output-type": "waylandsink",
  "snpe-tensors": "<Model output tensor name>"
}

Camera source, LiteRT model, CPU runtime

{
  "ml-framework": "tflite",
  "yolo-model-type": "yolox",
  "model": "/etc/models/yolox_quantized.tflite",
  "labels": "/etc/labels/yolox.json",
  "threshold": 40,
  "runtime": "cpu",
  "output-type": "waylandsink",
  "snpe-tensors": "<Model output tensor name>"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

Detected objects with bounding boxes and labels are overlaid on the video and displayed on the local display.

Pipeline Flow

The following table lists the plugins used in the object detection pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux`, which demultiplexes the stream. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`v4l2src`	• Captures the live stream from USB camera. • Uses `tee` to split the stream for inferencing.
`h264parse`	Parses the H.264 video bitstream.
`v4l2h264dec`	Hardware-decodes H.264 video to raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimlsnpe` `qtimltflite` `qtimlqnn`	1. After the inference runtime receives the tensor stream on its sink pad, it runs inference using the provided model. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	Handles inference results from any object detection model. 1. Applies a threshold to the chosen number of results. 2. Loads the YOLO (YOLOv5, YOLOv8, or YOLO-NAS) module. 3. Produces video frames with only bounding boxes that can be overlaid on objects. 4. Sends these processed frames to the sink pad of `qtivcomposer`.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.
`filesink`	Receives the video stream on sink pad and saves it as an H.264-encoded MP4 file.
`qtirtspbin`	1. Serves as a network sink. 2. Transmits UDP packets to the network.

Config JSON Field Description

Field	Values / Description
`ml-framework`	Supported ML frameworks: • `snpe` (Qualcomm Neural Processing SDK) • `tflite` (LiteRT) • `qnn` (Qualcomm AI Engine Direct)
`yolo-model-type`	Supported YOLO architectures: • `yolov8` • `yolonas` • `yolov5` • `yolox`
`runtime`	Hardware runtimes: • `cpu` • `gpu` • `dsp`
`Input source`	Supported input sources: • `camera` (0=primary, 1=secondary) • `file-path` • `rtsp-ip-port` • `usb-camera` (set `enable-usb-camera` to `TRUE`)
`output-ip-address`	Output RTSP server IP address
`port`	Output RTSP server port
`output-type`	Supported output sinks: • `waylandsink` (display) • `filesink` (MP4 file) • `rtspsink` (RTSP stream)
`snpe-tensors`	`["output-tensor-name", "output-tensor-name"]`
`USB camera video-format and resolution`	1. Use one of the following `video-format` options: • `nv12` • `yuy2` • `mjpeg` 2. Use the following resolution fields: • `width` • `height` • `framerate`
`output-file`	Output filename. The default output file is `output_object_detection.mp4`.

Image Classification

The gst-ai-classification application is designed to identify the subject in an image. The use cases are implemented using the Qualcomm Neural Processing SDK, LiteRT, or Qualcomm AI Engine Direct models. The pipeline receives a video stream from a camera, file source, USB source, or RTSP, preprocesses it, and runs the inference on AI hardware. The results are either displayed on the screen, saved as an encoded MP4 file, or streamed over the RTSP server. For information about the plugins used in the pipeline flow, see Pipeline flow

Application: gst-ai-classification

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	IMX577 camera	File output	Display	RTSP output
Config #1	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes
Config #2	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model file	Label file
Qualcomm Neural Processing SDK	`inceptionv3.dlc`	`classification.json`
LiteRT	`inception_v3_quantized.tflite`	`classification.json`
Qualcomm AI Engine Direct	`inception_v3_quantized.bin`	`classification.json`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-classification --config-file=/etc/configs/config_classification.json

The sample application uses the /etc/configs/config_classification.json file to read the input parameters.To display all available options:

gst-ai-classification -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-classification application uses the /etc/configs/config_classification.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<path-to-input-video>",
  "ml-framework": "<snpe or tflite or qnn framework>",
  "model": "<path-to-model-file>",
  "labels": "<path-to-label-file>",
  "threshold": <post processsing threshold, integer value from 1 to 100>,
  "runtime": "<dsp, gpu, cpu runtime>",
  "output-type": "waylandsink or filesink or rtspsink"
}

For USB camera input, set the video-format, resolution, and framerate parameters in the config file to match the camera capabilities, see Configure USB camera.

Available configurations

Config #2
Config #1

Camera source, LiteRT model, DSP runtime

{
  "camera": 0,
  "ml-framework": "tflite",
  "model": "/etc/models/inception_v3_quantized.tflite",
  "labels": "/etc/labels/classification.json",
  "threshold": 40,
  "runtime": "dsp",
  "output-type": "waylandsink"
}

Camera source, LiteRT model, CPU runtime

{
  "ml-framework": "tflite",
  "model": "/etc/models/inception_v3_quantized.tflite",
  "labels": "/etc/labels/classification.json",
  "threshold": 40,
  "runtime": "cpu",
  "output-type": "waylandsink"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The classified object label and confidence score are overlaid on the video and displayed on the local display

Pipeline Flow

The following table lists the plugins used in the classification pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux`, which demultiplexes the stream. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`v4l2src`	• Captures the live stream from USB camera. • Uses `tee` to split the stream for inferencing.
`h264parse`	Parses the H.264 video bitstream.
`v4l2h264dec`	Hardware-decodes H.264 video to raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream on its source pad. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimlsnpe` `qtimltflite` `qtimlqnn`	1. After the inference runtime receives the tensor stream on its sink pad, it runs inference using the provided model. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	Handles inference results from any classification model. 1. Applies a threshold to the chosen number of results. 2. Loads the MobileNet-softmax postprocessing module. 3. Produces results as video frames with classification labels. 4. Sends these processed frames to the sink pad of `qtivcomposer`.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.
`filesink`	Receives the video stream on sink pad and saves it as an H.264-encoded MP4 file.
`qtirtspbin`	1. Serves as a network sink. 2. Transmits UDP packets to the network.

Config JSON Field Description

Field	Values / Description
`ml-framework`	Supported ML frameworks: • `snpe` (Qualcomm Neural Processing SDK) • `tflite` (LiteRT) • `qnn` (Qualcomm AI Engine Direct)
`runtime`	Hardware runtimes: • `cpu` • `gpu` • `dsp`
`Input source`	Supported input sources: • `camera` (0=primary, 1=secondary) • `file-path` • `rtsp-ip-port` • `usb-camera` (set `enable-usb-camera` to `TRUE`)
`output-ip-address`	Output RTSP server IP address.
`port`	Output RTSP server port.
`output-type`	Supported output sinks: • `waylandsink`(display) • `filesink` (MP4 file) • `rtspsink` (RTSP stream)
`USB camera video-format and resolution`	1. Use one of the following `video-format` options: • `nv12` • `yuy2` • `mjpeg` 2. Use the following resolution fields: • `width` • `height` • `framerate`
`output-file`	Output filename. The default output file is `output_classification.mp4`.

Face Detection

The gst-ai-face-detection application collects the live video input from a camera, file, or an RTSP stream and uses the Qualcomm AI Engine direct and LiteRT face detection models to produce a preview with the overlaid AI model output on the HDMI display. The following figure shows the pipeline, which receives the input, preprocesses it, runs inferences on AI hardware, and displays the results on the screen. For information about the plugins used in the pipeline flow, see Pipeline flow.

Application: gst-ai-face-detection

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	No	No	No	Yes	No
Config #2	Yes	Yes	No	Yes	No	Yes	No

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	`face_det_lite_quantized.tflite`	`face_detection.json`
Qualcomm AI Engine Direct	`face_det_lite_quantized.bin`	`face_detection.json`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-face-detection --config-file=/etc/configs/config_face_detection.json

The sample application uses the /etc/configs/config_face_detection.json file to read the input parameters.To display all available options:

gst-ai-face-detection -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-face-detection application uses the /etc/configs/config_face_detection.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<input-video-path>",
  "ml-framework": "<tflite or qnn framework>",
  "model": "<path-to-model-file",
  "labels": "<path-to-label-file",
  "threshold": <post-processing threshold, integer value from 1-100>,
  "runtime": ""<cpu, gpu or dsp runtime>"
}

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
  "file-path": "/etc/media/video.mp4",
  "ml-framework": "tflite",
  "model":"/etc/models/face_det_lite_quantized.tflite",
  "labels": "/etc/labels/face_detection.json",
  "threshold": 51,
  "runtime": "dsp"
}

File source, LiteRT model, CPU runtime

{
  "file-path": "/etc/media/video.mp4",
  "ml-framework": "tflite",
  "model":"/etc/models/face_det_lite_quantized.tflite",
  "labels": "/etc/labels/face_detection.json",
  "threshold": 51,
  "runtime": "cpu"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

Pipeline Flow

The following table lists the plugins used in the face detection pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`h264parse`	• Parses the H.264 video bitstream.
`v4l2h264dec`	• Hardware-decodes H.264 video to raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite` `qtimlqnn`	1. After the inference runtime receives the tensor stream on its sink pad, it runs inference using the provided model. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	1. Handles inference results from any face detection model. 2. Applies a threshold to the chosen number of results.
`qtimetamux`	1. Receives string-based postprocessing output text with video frame and multiplexes it.
`qtivoverlay`	1. Receives the multiplexed stream. 2. Overlays the bounding boxes on the stream.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Description
`ml-framework`	Supported ML frameworks: • `tflite` (LiteRT) • `qnn` (Qualcomm AI Engine Direct)
`runtime`	Supported hardware runtimes: • `cpu` • `gpu` • `dsp`
`Input source`	Supported input sources: • `file-path` • `rtsp-ip-port` • `camera`

Semantic Segmentation

The gst-ai-segmentation application allows you to divide an image into different and meaningful parts or segments and assign a label to each homogeneous segment based on the similarity of the attributes. The application uses Qualcomm Neural Processing SDK runtime, Qualcomm AI Engine direct runtime, and LiteRT for image segmentation. The following figure shows the pipeline, which receives the input from a live camera feed, file, or an RTSP stream, preprocesses the video data, runs inferences using AI hardware, and displays the segmented data on the screen. For information about the plugins used in the pipeline flow, see Pipeline flow.

Application: gst-ai-segmentation

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	No	No	No	Yes	No
Config #2	Yes	Yes	No	Yes	No	Yes	No

Sample Model and Label Files

Runtime	Model file	Label file
Qualcomm Neural Processing SDK	`deeplabv3_plus_mobilenet.dlc`	`deeplabv3_resnet50.json`
LiteRT	`deeplabv3_plus_mobilenet_quantized.tflite`	`deeplabv3_resnet50.json`
Qualcomm AI Engine Direct	`deeplabv3_plus_mobilenet_quantized.bin`	`deeplabv3_resnet50.json`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-segmentation --config-file=/etc/configs/config_segmentation.json

The sample application uses the /etc/configs/config_segmentation.json file to read the input parameters.To display all available options:

gst-ai-segmentation -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-segmentation application uses the /etc/configs/config_segmentation.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<input-video-path>",
  "ml-framework": "<snpe, tflite, or qnn framework>",
  "model": "<path-to-model-file>",
  "labels": "<path-to-label-file>",
  "runtime": "<dsp, gpu, or cpu runtime>"
}

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
 "file-path": "/etc/media/video.mp4",
 "ml-framework": "tflite",
 "model": "/etc/models/deeplabv3_plus_mobilenet_quantized.tflite",
 "labels": "/etc/labels/deeplabv3_resnet50.json",
 "runtime": "dsp"
}

File source, LiteRT model, CPU runtime

{
"file-path": "/etc/media/video.mp4",
"ml-framework": "tflite",
"model": "/etc/models/deeplabv3_plus_mobilenet_quantized.tflite",
"labels": "/etc/labels/deeplabv3_resnet50.json",
"runtime": "cpu"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The segmented data is displayed on the local display.

Pipeline Flow

The following table lists the plugins used in the segmentation pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for concurrent display and ML inference.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux`, which demultiplexes the stream. • Uses `tee` to split the stream for processing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for processing.
`h264parse`	• Parses the H.264 video bitstream to ensure downstream elements can handle the payload.
`v4l2h264dec`	• Hardware-accelerated decoder that converts H.264 video into raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data. This preprocessing is done when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream on its source pad. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimlsnpe` `qtimltflite` `qtimlqnn`	1. After the inference runtime receives the tensor stream on its sink pad, it runs inference using the provided model. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	1. Converts the inference tensors received on its sink pad into video formats that multimedia plugins use for further processing.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Description
`ml-framework`	Supported ML frameworks: • `snpe` (Qualcomm Neural Processing SDK) • `tflite` (LiteRT) • `qnn` (Qualcomm AI Engine Direct)
`runtime`	Supported hardware runtimes: • `cpu` • `gpu` • `dsp`
`Input source`	Supported input sources: • `file-path` • `rtsp-ip-port` • `camera`

Pose Detection

The gst-ai-pose-detection application allows you to detect the body pose of the subject in an image or video. The use case processes input streams from a camera, file, or an RTSP source and uses LiteRT and Qualcomm AI Engine direct models for pose detection. The results are either displayed on the screen, saved as an encoded MP4 file, or streamed over the RTSP server. The following figure shows the pipeline, which receives the input from a live camera feed, file, USB source, or an RTSP stream, preprocesses it, conducts inference on AI hardware, and generates the output. This process allows for real-time pose detection and visualization of human poses. For information about the plugins used in the pipeline flow, see Pipeline flow.

Application: gst-ai-pose-detection

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	Yes	No	Yes	Yes	Yes
Config #2	Yes	Yes	Yes	Yes	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model file	Label files
LiteRT	`hrnet_pose_quantized.tflite`	`hrnet_pose.json`, `hrnet_settings.json`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-pose-detection --config-file=/etc/configs/config_pose.json

The sample application uses the /etc/configs/config_pose.json file to read the input parameters.To display all available options:

gst-ai-pose-detection -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-pose-detection application uses the /etc/configs/config_pose.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<input-video-path>",
  "ml-framework": "<tflite or qnn framework>",
  "model": "<path-to-model-file>",
  "labels": "<path-to-label-file>",
  "pose-settings-path": "<path-to-hrnet_settings-file>",
  "output-type": "waylandsink or filesink or rtspsink",
  "runtime": "<cpu, gpu or dsp runtime>"
}

For USB camera input, set the video-format, resolution, and framerate parameters in the config file to match the camera capabilities, see Configure USB camera.

To change the threshold, you must configure the confidence value in the hrnet_settings.json file.

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
  "file-path": "/etc/media/video.mp4",
  "ml-framework": "tflite",
  "model": "/etc/models/hrnet_pose_quantized.tflite",
  "labels": "/etc/labels/hrnet_pose.json",
  "pose-settings-path":"/etc/labels/hrnet_settings.json",
  "runtime": "dsp",
  "output-type": "waylandsink"
}

File source, LiteRT model, CPU runtime

{
  "file-path": "/etc/media/video.mp4",
  "ml-framework": "tflite",
  "model": "/etc/models/hrnet_pose_quantized.tflite",
  "labels": "/etc/labels/hrnet_pose.json",
  "pose-settings-path":"/etc/labels/hrnet_settings.json",
  "runtime": "cpu",
  "output-type": "waylandsink"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The displayed output shows the detected pose of the objects.

Pipeline Flow

The following table lists the plugins used in the pose detection pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for concurrent display and ML inference.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for processing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for processing.
`v4l2src`	• Captures the live stream from USB camera. • Uses `tee` to split the stream for processing.
`h264parse`	• Parses the H.264 video bitstream to ensure downstream elements can handle the payload.
`v4l2h264dec`	• Hardware-accelerated decoder that converts H.264 video into raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite` `qtimlqnn`	1. Uses the HRNet model for pose detection. 2. The application runs on the external delegate to execute the model using the Hexagon Tensor Processor. 3. After the inference runtime receives the tensor stream on its sink pad, it does the following: • Runs the inference. • Produces a tensor stream containing the inference results on its source pad. • Manages the inference results from the pose detection model.
`qtimlpostprocess`	1. Applies a threshold to the chosen number of results. 2. Loads corresponding modules for various pose detection models. 3. In this specific use case, `qtimlpostprocess` does the following: • Loads the HRNet module. • Produces results in the form of video frames with drawn poses. • Sends the results to the sink pad of `qtivcomposer` for further processing or display.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.
`filesink`	1. Receives the video stream on its sink pad and saves it as an H.264-encoded MP4 file.
`qtirtspbin`	1. Serves as a network sink. 2. Transmits UDP packets to the network.

Config JSON Field Description

Field	Values / Description
`ml-framework`	Supported ML frameworks: • `tflite` (LiteRT) • `qnn` (Qualcomm AI Engine Direct)
`runtime`	Hardware runtimes: • `cpu` • `gpu` • `dsp`
`Input source`	Supported input sources: • `camera` (0=primary, 1=secondary) • `file-path` • `rtsp-ip-port` • `usb-camera` (set `enable-usb-camera` to `TRUE`)
`output-ip-address`	Output RTSP server IP address.
`port`	Output RTSP server port.
`output-type`	Supported output sinks: • `waylandsink` (display) • `filesink` (MP4 file) • `rtspsink` (RTSP stream)
`USB camera video-format and resolution`	1. Use one of the following `video-format` options: • `nv12` • `yuy2` • `mjpeg` 2. Use the following resolution fields: • `width` • `height` • `framerate`
`enable-usb-camera`	Set to `TRUE` or `FALSE`.
`output-file`	Output filename. Default: `output_pose`

For better accuracy and detection results, use the gst-ai-daisychain-detection-pose application.

Mono Depth

The gst-ai-monodepth application allows you to infer depth of a source feed from a live camera stream, file, or an RTSP stream. The following figure shows the pipeline, which captures feed from the source, preprocesses it, and runs inferences using the AI hardware. For information about the plugins used in the pipeline, see Pipeline flow.

Application: gst-ai-monodepth

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	No	No	No	Yes	No
Config #2	Yes	Yes	No	Yes	No	Yes	No

Sample Model and Label Files

Runtime	Model file	Label file
Qualcomm Neural Processing SDK	`midasv2.dlc`	`monodepth.json`
LiteRT	`midas_quantized.tflite`	`monodepth.json`
Qualcomm AI Engine Direct	`midas_quantized.bin`	`monodepth.json`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-monodepth --config-file=/etc/configs/config_monodepth.json

The sample application uses the /etc/configs/config_monodepth.json file to read the input parameters.To display all available options:

gst-ai-monodepth -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-monodepth application uses the /etc/configs/config_monodepth.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<input video path>",
  "ml-framework": "<snpe, tflite, or qnn framework>",
  "model": "<path-to-model-file>",
  "labels": "<path-to-label-file>",
  "runtime": "<dsp, gpu, or cpu runtime>"
}

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
  "file-path": "/etc/media/video.mp4",
  "ml-framework": "tflite",
  "model": "/etc/models/midas_quantized.tflite",
  "labels": "/etc/labels/monodepth.json",
  "runtime": "dsp"
}

File source, LiteRT model, CPU runtime

{
  "file-path": "/etc/media/video.mp4",
  "ml-framework": "tflite",
  "model": "/etc/models/midas_quantized.tflite",
  "labels": "/etc/labels/monodepth.json",
  "runtime": "cpu"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The overlaid model output stream is shown side by side with the live feed.

Pipeline Flow

The following table lists the plugins used in the monodepth pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for concurrent display and ML inference.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for processing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for processing.
`h264parse`	• Parses the H.264 video bitstream to ensure downstream elements can handle the payload.
`v4l2h264dec`	• Hardware-accelerated decoder that converts H.264 video into raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data. This preprocessing is done when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream on its source pad. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimlsnpe` `qtimltflite` `qtimlqnn`	Uses the Midasv2 model for monodepth calculation. 1. The inference runtime receives the tensor stream on its sink pad. 2. The runtime executes the inference. 3. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	1. Converts the inference tensors received on its sink pad into video formats that multimedia plugins use for further processing.
`qtivtransform`	1. Converts the buffers on its source pad to formats compatible with composition on `waylandsink`.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Description
`ml-framework`	Supported ML frameworks: • `snpe` (Qualcomm Neural Processing SDK) • `tflite` (LiteRT) • `qnn` (Qualcomm AI Engine Direct)
`runtime`	Supported hardware runtimes: • `cpu` • `gpu` • `dsp`
`Input source`	Supported input sources: • `file-path` • `rtsp-ip-port` • `camera` (0=primary, 1=secondary)

Super Resolution

The gst-ai-superresolution application allows you to generate high resolution video frames from low-resolution input. The following figures shows the pipeline, which receives a video stream from a file source as input, processes it through the super resolution module using LiteRT, and displays the output.

This application isn’t supported in Config #1 for the QLI 2.0 GA release because CPU runtime is not supported.

For information about the plugins used in the pipeline, see Pipeline flow.

Application: gst-ai-superresolution

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #2	Yes	No	No	No	Yes	Yes	No

Sample Model Files

Runtime	Model file
LiteRT	`quicksrnetsmall_quantized.tflite`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-superresolution --config-file=/etc/configs/config-superresolution.json

The sample application uses the /etc/configs/config-superresolution.json file to read the input parameters.To display all available options:

gst-ai-superresolution -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-superresolution application uses the /etc/configs/config-superresolution.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "input-file-path": "<input-video-path>",
  "model": "<path-to-model-file>",
  "output-file-path": "<path-to-output-video>"
}

The video super‑resolution application requires an input video resolution of 128 × 128.

Available configurations

Config #2

File source, LiteRT model, DSP runtime

{
  "input-file-path": "/etc/media/video.mp4",
  "model": "/etc/models/quicksrnetsmall_quantized.tflite"
}

Expected Output

The output is displayed on an HDMI monitor.

Pipeline Flow

The following table lists the plugins used in the superresolution pipeline:

Plugin	Description
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for processing.
`h264parse`	• Parses the H.264 video bitstream to ensure downstream elements can handle the payload.
`v4l2h264dec`	• Hardware-accelerated decoder that converts H.264 video into raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data. This preprocessing is done when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream on its source pad. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	• Runs on LiteRT and uses the `quicksrnetsmall_quantized` model for super resolution. 1. The inference runtime receives the tensor stream on its sink pad. 2. The runtime executes the inference. 3. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	• Handles inference results from any super resolution model. 1. Loads the SRNet module. 2. Produces results as high-resolution video frames. 3. Sends the processed frames to the sink pad of `qtivcomposer`.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Description
`Input source`	• `input-file-path`: The directory path of the input video.
`model`	• `model`: The path to the super resolution model.
`Output source`	Configuration for the output destination: • `output-file-path`: The directory path of the output video. If the `output-file-path` is not provided, the display output is automatically enabled.

AI Event Encoder

The gst-ai-event-encoder application receives the live video stream input from camera, file, or RTSP source. When a human enters the video frame the application preprocesses the video, runs inferences on the AI hardware, and encodes the video. The encoding stops 5 seconds after the human moves away from the frame and restarts when anyone enters the frame. The following figures show the event detection and recording pipelines for event encoder application. For information about the plugins used in the pipeline flow, see Pipeline flow.

Application: gst-ai-event-encoder

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	No	No	No	Yes	No
Config #2	Yes	Yes	No	Yes	No	Yes	No

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	`yolox_quantized.tflite`	`yolox.json`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-event-encoder --config-file=/etc/configs/config-event-encoder.json

The sample application uses the /etc/configs/config-event-encoder.json file to read the input parameters.To display all available options:

gst-ai-event-encoder -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-event-encoder application uses the /etc/configs/config-event-encoder.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<path to video file>",
  "model": "<path to model file>",
  "labels": "<path to label file>",
  "threshold": <integer between 1 and 100>,
  "runtime": "<cpu, gpu, or dsp runtime>"
}

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
  "file-path": "/etc/media/video.mp4",
  "model": "/etc/models/yolox_quantized.tflite",
  "labels": "/etc/labels/yolox.json",
  "threshold": 40,
  "runtime": "dsp"
}

File source, LiteRT model, CPU runtime

{
  "file-path": "/etc/media/video.mp4",
  "model": "/etc/models/yolox_quantized.tflite",
  "labels": "/etc/labels/yolox.json",
  "threshold": 40,
  "runtime": "cpu"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The output is saved as an MP4 file within /etc/media folder as output-1.mp4, output-2.mp4, and so on.

Pipeline Flow

The following table lists the plugins used in the event encoder pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for concurrent display and ML inference.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for processing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for processing.
`h264parse`	• Parses the H.264 video bitstream to ensure downstream elements can handle the payload.
`v4l2h264dec`	• Hardware-accelerated decoder that converts H.264 video into raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	1. After the inference runtime receives the tensor stream on its sink pad, it executes the inference. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	Instance 1 (Detection Overlay): • Applies a threshold to the chosen number of results. • Loads the YOLOv8 module. • Produces video frames with only bounding boxes for object overlay. • Sends processed frames to the sink pad of `qtivcomposer`. Instance 2 (Metadata Generation): • Produces output in text format (bounding box coordinates and labels). • Connects to an `appsink` plugin where metadata is read, parsed, and logged. • Uses bounding box information to count the number of humans in each frame.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Description
`runtime`	Supported hardware runtimes: • `cpu` • `gpu` • `dsp`
`Input source`	Supported input sources: • `file-path` • `rtsp-ip-port` • `camera` (0=primary, 1=secondary)

Metadata Parser

The gst-ai-metadata-parser-example application receives the live video stream input from camera, file, or RTSP source, and passes the stream to the YOLO models for object detection and preview. The overlaid AI model output, including labels and bounding boxes, is displayed on an HDMI display. The extracted metadata is logged to the console and used to count the number of humans in the frame. The following figure shows the pipeline for metadata parsing. For information about the plugins used in the pipeline flow, see Pipeline flow.

Application: gst-ai-metadata-parser-example

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	No	No	No	Yes	No
Config #2	Yes	Yes	No	Yes	No	Yes	No

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	`yolox_quantized.tflite`	`yolox.json`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-metadata-parser-example --config-file=/etc/configs/config-metadata-parser.json

To view the bounding box information along with the human count, run the following command before running the application:

export GST_DEBUG=4

The sample application uses the /etc/configs/config-metadata-parser-example.json file to read the input parameters.To display all available options:

gst-ai-metadata-parser-example -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-metadata-parser-example application uses the /etc/configs/config-metadata-parser-example.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<path to video file>",
  "model": "<path to model file>",
  "labels": "<path to label file>",
  "threshold": <integer between 1 and 100>,
  "runtime": "<cpu, gpu, or dsp runtime>"
}

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
  "file-path": "/etc/media/video.mp4",
  "model": "/etc/models/yolox_quantized.tflite",
  "labels": "/etc/labels/yolox.json",
  "threshold": 40,
  "runtime": "dsp"
}

File source, LiteRT model, CPU runtime

{
  "file-path": "/etc/media/video.mp4",
  "model": "/etc/models/yolox_quantized.tflite",
  "labels": "/etc/labels/yolox.json",
  "threshold": 40,
  "runtime": "cpu"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

Pipeline Flow

The following table lists the plugins used in the metadata parser pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream into two for inferencing and composing.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream into two for inferencing and composing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream into two for inferencing and composing.
`h264parse`	• Parses the H.264 video.
`v4l2h264dec`	• Decodes the video.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream on its source pad. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	• After the inference runtime receives the tensor stream on its sink pad, it runs the inference. • Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	Instance 1 (Object Detection): • Applies a threshold to the chosen number of results. • Loads the YOLOv8 module. • Produces video frames with only bounding boxes that can be overlaid on objects. • Sends these processed frames to the sink pad of `qtivcomposer`. Instance 2 (Human Counting): • Produces the output in a text format (bounding box coordinates and labels). • This output is connected to `appsink` plugin where the metadata is read, parsed, and logged. • The bounding box information is used to count the number of humans in each frame.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. `waylandsink` submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values/Description
`runtime`	Use one of the following runtimes: `cpu` `gpu` `dsp`
`Input source`	Use one of the following input sources: `camera` – Primary (`0`) or secondary (`1`). `file-path` – The directory path to the video file. `rtsp-ip-port` – The address of the RTSP stream: `rtsp://<ip>:<port>/<stream>`

AI USB Camera

The gst-ai-usb-camera-app streams video from a USB webcam connected to the Qualcomm EVK. This webcam should be accessible as a /dev/videoX device. Additionally, you can perform object detection and preview the results. You can choose to preview the output on Wayland, or encode to a video file, or live stream through the RTSP. Alternatively, you can set enable-object-detection as True to perform object detection. The following figures show a pipeline, which processes the input from the USB camera to generate various outputs. For information about the plugins used in this pipeline, see Pipeline flow.

Application: gst-ai-usb-camera-app

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	No	No	Yes	No	Yes	Yes	Yes
Config #2	No	No	Yes	No	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model Files	Label Files
`Qualcomm Neural Processing SDK`	`yolonas.dlc` `yolov5.dlc` `yolov8.dlc`	`yolonas.json` `yolov5.json` `yolov8.json`
`LiteRT`	`yolov8_det_w8a8.tflite` `yolonas_quantized.tflite` `yolov5.tflite` `yolox_quantized.tflite`	`yolov8.json` `yolonas.json` `yolov5.json` `yolox.json`
`Qualcomm AI Engine direct`	`yolov8_det_quantized.bin`	`yolov8.json`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-usb-camera-app --config-file=/etc/configs/config-usb-camera-app.json

The sample application uses the /etc/configs/config-usb-camera-app.json file to read the input parameters.To display all available options:

gst-ai-usb-camera-app -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-usb-camera-app application uses the /etc/configs/config-usb-camera-app.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "width": "<Supported USB camera width>",
  "height": "<Supported USB camera Height>",
  "framerate": "<Supported USB camera fps>",
  "video-format": "<Supported USB camera video format like yuy2, mjpeg, nv12>",
  "output": "<output type like waylandsink, filesink or rtspsink>",
  "ip-address": "<Device IP address in case of rtsp streaming>",
  "port": "<Device port num in case of rtsp streaming>",
  "enable-object-detection": "<to enable the object-detection>",
  "file-path": "<input video path>",
  "ml-framework": "<snpe, or tflite or qnn framework>",
  "yolo-model-type": "<yolov8 or yolonas or yolov5 or yolox>",
  "model": "<Model Path>",
  "labels": "<Label Path>",
  "threshold": "<Post-processing threshold, integer value from 1-100>",
  "runtime": "<dsp, cpu or gpu runtime>",
  "snpe-tensors": "<Model output tensor name>"
}

For USB camera input, set the video-format, resolution, and framerate parameters in the config file to match the camera capabilities.

You can run the Yolo-NAS-Quantized.tflite model using the YOLOv8 value in the yolo-model-type field.

The snpe-tensors field applies only to the SNPE runtime. To retrieve the output tensor names for a DLC model, open the model in Netron.

When using DLC models from the AI Hub, the snpe-tensors field is optional.

If the USB camera isn’t detected on the target device, Download the required firmware. See Download PCIe to USB controller firmware.

Available configurations

Config #2
Config #1

RTSP input, LiteRT, YOLOx model and DSP runtime

{
  "width": 1920,
  "height": 1080,
  "framerate": 30,
  "output":"waylandsink",
  "video-format":"yuy2",
  "model":"/etc/models/yolox_quantized.tflite",
  "labels":"/etc/labels/yolox.json",
  "output-file":"/etc/media/output.mp4",
  "ip-address":"127.0.0.1",
  "port":"8900",
  "enable-object-detection": "TRUE",
  "ml-framework": "tflite",
  "yolo-model-type": "yolox",
  "threshold": 75,
  "runtime": "dsp"
}

RTSP input, LiteRT, YOLOx model and CPU runtime

{
  "width": 1920,
  "height": 1080,
  "framerate": 30,
  "output":"waylandsink",
  "video-format":"yuy2",
  "model":"/etc/models/yolox_quantized.tflite",
  "labels":"/etc/labels/yolox.json",
  "output-file":"/etc/media/output.mp4",
  "ip-address":"127.0.0.1",
  "port":"8900",
  "enable-object-detection": "TRUE",
  "ml-framework": "tflite",
  "yolo-model-type": "yolox",
  "threshold": 75,
  "runtime": "cpu"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

Pipeline flow

The following table lists the plugins used in AI USB camera pipelines:

Pipeline	Description
Dump the USB camera to a filesink	• USB camera captures the live camera stream. • `qtivtransform` transforms the stream data. • `capsfilter` is applied to enforce constraints on the raw video data. • `filesink` is used to dump the data into a file.
Video encoding	• USB camera captures the live camera stream. • `qtivtransform` transforms the stream data. • `capsfilter` is applied to enforce constraints on the raw video data. • `v4l2h264enc` is used to encode the video using the H.264 format. • `h264parse` is used to parse the video. • `mp4mux` is used to multiplex the video into an MP4 container. • `filesink` is used to write the video to a file.
RTSP streaming	• USB camera captures the live camera stream. • `qtivtransform` transforms the stream data. • `capsfilter` is applied to enforce constraints on the raw video data. • `v4l2h264enc` is used to encode the video using the H.264 format. • `h264parse` is used to parse the video. • `qtirtspbin` is used to load the stream to RTSP.
USB camera and object detection on RTSP	• USB camera captures the live camera stream. • `capsfilter` is applied to enforce constraints on the raw video data. • `tee` is used to split the stream for inferencing. • `qtivtransform` transforms the stream data. • `qtimlvconverter` performs preprocessing and converts the video stream to a tensor stream, which is used for inferencing. • `qtimlsnpe`, `qtimltflite`, or `qtimlqnn` run the inference on the stream. • `qtimlpostprocess` handles the inference results from any object detection model and produces video frames. • `qtivcomposer` composes the video frames and shares them with `qtirtspbin`. • `qtirtspbin` submits the composed video stream to Weston, which renders it on the local display.
USB camera and object detection on wayland	• USB camera captures the live camera stream. • `capsfilter` is applied to enforce constraints on the raw video data. • `tee` is used to split the stream for inferencing. • `qtivtransform` transforms the stream data. • `qtimlvconverter` performs preprocessing and converts the video stream to a tensor stream, which is used for inferencing. • `qtimlsnpe`, `qtimltflite`, or `qtimlqnn` run the inference on the stream. • `qtimlpostprocess` handles the inference results from any object detection model and produces video frames. • `qtivcomposer` composes the video frames and shares them with `waylandsink`. • `waylandsink` submits the composed video stream to Weston, which renders it on the local display.
Object detection using USB camera and file encode	• USB camera captures the live camera stream. • `capsfilter` is applied to enforce constraints on the raw video data. • `tee` is used to split the stream for inferencing. • `qtivtransform` transforms the stream data. • `qtimlvconverter` performs preprocessing and converts the video stream to a tensor stream, which is used for inferencing. • `qtimlsnpe`, `qtimltflite`, or `qtimlqnn` run the inference on the stream. • `qtimlpostprocess` handles the inference results from any object detection model and produces video frames. • `qtivcomposer` composes the video frames and shares them with `filesink`. • `filesink` writes the composed video stream to a file.

Config JSON field description

Field	Values/Description
`ml-framework`	Use one of the following models: • `snpe` — Qualcomm Neural Processing SDK • `tflite` — LiteRT • `qnn` — Qualcomm AI Engine Direct
`yolo-model-type`	Run one of the following models respectively: • `yolov5` • `yolov8` • `yolonas`
`runtime`	Use one of the following runtimes: • `cpu` • `gpu` • `dsp`
`output`	Use one of the following output types: • `filesink` • `waylandsink` • `rtspsink`
`enable-object-detection`	• `TRUE` • `FALSE`
`snpe-tensors`	`["output-tensor-name","output-tensor-name"]`
`USB camera video-format and resolution`	Use one of the following video formats: • `nv12` • `yuy2` • `mjpeg` Use the following resolution parameters: • `width`: Input USB camera source resolution width. • `height`: Input USB camera source resolution height. • `framerate`: Input USB camera source framerate.

Face Recognition

The gst-ai-face-recognition application collects the live video input from a camera or an RTSP stream and shares this input for face detection, facial landmarking, and face recognition. It uses the face_det_quantized models for face detection, facemap_3dmm_quantized model for facial landmarking, and face_attrib_net_quantized model for face recognition labels. The result is a preview of the overlaid AI model on the HDMI display.

This application isn’t supported in Config #1 for the QLI 2.0 GA release because CPU runtime is not supported.

The following figure shows the pipeline, which receives the input, preprocesses it, runs inferences on AI hardware, and displays the results on the screen.

Application: gst-ai-face-recognition For information about the plugins used in the pipeline flow, see Pipeline flow.

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #2	No	Yes	No	Yes	No	Yes	No

Sample Model and Label Files

Runtime	Model files	Label files
Qualcomm AI Engine Direct & LiteRT	`face_det_lite_quantized.tflite`, `facemap_3dmm_quantized.tflite`, `face_attrib_net_quantized.tflite`, `face_det_lite_quantized.bin`, `facemap_3dmm_quantized.bin`, `face_attrib_net_quantized.bin`	`face_detection.json`, `face_recognition_settings.json`, `face_recognition.json`, `facemap_3dmm_settings.json`

Register a face for facial recognition

Before running the gst-ai-face-recognition application, you can register a face for secure verification and authentication:

Ensure that you complete the Prerequisites.

To register a face, use the following gst-pipeline on the target device shell:

gst-pipeline-app -e \
qtimlvconverter name=stage_01_preproc mode=image-batch-non-cumulative \
qtimltflite name=stage_01_inference model=/etc/models/face_det_lite_quantized.tflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
external-delegate-options="QNNExternalDelegate,backend_type=htp;" \
qtimlpostprocess name=stage_01_postproc settings="{\"confidence\": 40.0}" results=4 module=qfd labels=/etc/labels/face_detection.json \
qtimlvconverter name=stage_03_preproc mode=roi-batch-cumulative \
qtimltflite name=stage_03_inference model=/etc/models/face_attrib_net_quantized.tflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
external-delegate-options="QNNExternalDelegate,backend_type=htp;" \
qticamsrc video_0::type=video name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080 ! queue ! waylandsink fullscreen=true sync=false \
camsrc.image_1 ! video/x-raw,width=1920,height=1080 ! qtivtransform ! video/x-raw,format=NV12 ! tee name=t_split_1 \
t_split_1. ! queue ! metamux_1. \
t_split_1. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! stage_01_inference. stage_01_inference. ! queue ! \
stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! metamux_1. \
qtimetamux name=metamux_1 ! queue ! tee name=t_split_3 \
t_split_3. ! queue ! stage_03_preproc. stage_03_preproc. ! queue ! stage_03_inference. stage_03_inference. ! queue ! \
multifilesink location=/etc/data/tensor_%d.bin sync=true async=false enable-last-sample=false

To prepare for capturing a facial image, do the following:

Select the following options from the list. Choose the number corresponding to the option:

PLAYING: Move the pipeline to the Playing state.
Plugin Mode ➔ camsrc ➔ capture-image: Capture the image using a camera source.

Using the live preview on the display, face the camera and ensure that the camera is pointed straight and there is only one person in the frame.

In the terminal, enter 1 for the following values:

GstImageCaptureMode for arg0.
guint for arg1.

To capture all the sides of your face, select capture-image do the following for each side:

Left and right: Turn your head left by 40° while keeping the landmarks visible, then repeat steps 3 and 4. Turn your head right (by 40°) and repeat.

Up and down: Raise your head by 30° while keeping the landmarks visible, then repeat steps 3 and 4. Lower your head (by 30°) and repeat.

To stop the pipeline, use (b)Back and (q)Quit.
After running the pipeline, five individual tensor bins are created (tensor_0.bin to tensor_4.bin) with facial properties recorded for each side of the face.

On the target device, go to /etc/data/, find the tensor bins. To pull the bins from the target device to the Linux host computer, run the following commands:

scp root@<IP-Address>:/etc/data/tensor_0.bin .

scp root@<IP-Address>:/etc/data/tensor_1.bin .

scp root@<IP-Address>:/etc/data/tensor_2.bin .

scp root@<IP-Address>:/etc/data/tensor_3.bin .

scp root@<IP-Address>:/etc/data/tensor_4.bin .

To merge the tensor bins with all the facial properties into a cohesive image, download and run the facedb.py script in the same directory as the tensor bins on the Linux host computer.

Download the facedb.py script:

curl -L -O https://raw.githubusercontent.com/quic/sample-apps-for-qualcomm-linux/refs/heads/main/qualcomm-linux/scripts/facedb.py

Run the script. Note that <Name of the person> is case and style sensitive. Ensure that you use the same name consistently.

python3 ./facedb.py "<Name of the person>" 512 32 tensor_0.bin tensor_1.bin tensor_2.bin tensor_3.bin tensor_4.bin

Push the face.bin binary to /etc/data directory and rename it to face0.bin.

scp face.bin root@<ip addressof target device>:/etc/data/face0.bin

To generate the face_recognition.json file and register the new person into the database, use the following reference label file for two-person registered face:

[
  {"id": 0, "color": "0x00FF00FF", "label": "<Name of Person>"},
  {"id": 1, "color": "0xFFFF00FF", "label": "<Name of Person>"}
]

Update the ID field according to the number in the list. If more faces are registered, add the structure in a new line within face_recognition.json.

To generate the face_recognition_settings.json file use the following reference label file:

{
  "confidence": 51.0,
    "databases":[
      {"id":  0, "database": "/etc/data/face0.bin"},
      {"id":  1, "database": "/etc/data/face1.bin"}
    ]
}

To push the updated face_recognition.json and face_recognition_settings.json files to the /etc/labels directory on the target device.

scp face_recognition.json root@<ip address of target device>:/etc/labels

scp face_recognition_settings.json root@<ip address of target device>:/etc/labels

Run the application on the target device

The following commands provide the default model and label paths. If you have a different folder structure, replace the default paths in the command-line parameters. See Sample model and label files.

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-face-recognition --config-file=/etc/configs/config-face-recognition.json

The sample application uses the /etc/configs/config-face-recognition file to read the input parameters.To display all available options:

gst-ai-face-recognition -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-face-recognition application uses the /etc/configs/config-face-recognition.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "ml-framework": "<qnn or tflite>",
  "face-detection-model": "<path for face detection model file>",
  "face-landmark-model":"<path for face landmark model file>",
  “face-recognition-model”:”<path for face recognition model file>”,
  "face-detection-labels":”<path for face detection labels file>”,
  "face-recognition-labels":”<path for face recognition labels file>”,
  "face-recognition-settings": "<Path of face recognition settings>",
  "facemap-3dmm-settings": "<Path of facemap-3dmm settings>"
}

Available configurations

Config #2

Camera source, LiteRT, and DSP runtime

{
  "ml-framework":"tflite",
  "face-detection-model":"/etc/models/face_det_lite_quantized.tflite",
  "face-landmark-model":"/etc/models/facemap_3dmm_quantized.tflite",
  "face-recognition-model":"/etc/models/face_attrib_net_quantized.tflite",
  "face-detection-labels": "/etc/labels/face_detection.json",
  "face-recognition-labels": "/etc/labels/face_recognition.json",
  "face-recognition-settings": "/etc/labels/face_recognition_settings.json",
  "facemap-3dmm-settings": "/etc/labels/facemap_3dmm_settings.json"
}

Expected output

Pipeline Flow

The following table lists the plugins used in the daisychain detection and classification pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`h264parse`	• Parses the H.264 video bitstream.
`v4l2h264dec`	• Hardware-decodes H.264 video to raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	1. After the inference runtime receives the tensor stream on its sink pad, it runs the inference. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess` (Pose-estimation)	• Uses the `lite-3dmm` module to perform facial pose recognition.
`qtimlpostprocess` (Classification)	• Uses the `qfr` module to receive the stream from `qtimetamux` and classifies the face.
`qtimetamux`	1. Receives the output of the face detection models from `qtimlpostprocess` and multiplexes it. 2. Receives the output of facial pose from `qtimlpostprocess` and multiplexes it.
`tee`	• Splits the stream for inferencing.
`qtivoverlay`	1. Receives the multiplexed stream. 2. Overlays the bounding boxes on the stream.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Description
ml-framework	Use one of the following models: - tflite – LiteRT - qnn – Qualcomm AI Engine direct
Models and labels	See Sample model and label files
face-detection-model	The path to the face detection model
face-landmark-model	The path to the face landmark model
face-recognition-model	The path to the face recognition model
face-detection-labels	The path to the face detection labels
face-recognition-labels	The path to the face recognition labels
face-recognition-settings	The path of face recognition setting labels
facemap-3dmm-settings	The path of facemap-3dmm setting labels

AI Audio Applications

Audio Classification

The gst-ai-audio-classification application shows audio classification using input from either a file source or a microphone. It displays both the classification results and a video preview. The following figure shows the pipeline, which gets the input from a file or a microphone, preprocesses it, and runs inferences on AI hardware. The results are displayed on the screen. For information about the plugins used in the pipeline flow, see Pipeline flow.

Application: gst-ai-audio-classification

Try me

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	`yamnet.tflite`	`yamnet.json`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-audio-classification --config-file=/etc/configs/config-audio-classification.json

The sample application uses the /etc/configs/config-audio-classification.json file to read the input parameters.To display all available options:

gst-ai-audio-classification -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-audio-classification application uses the /etc/configs/config-audio-classification.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<path to video file>",
  "model": "<path to model file>",
  "labels": "<path to label file>",
  "threshold": <integer between 1 and 100>,
  "runtime": "<cpu or gpu>",
  "codec": "<mp3 or flac>"
}

Available configurations

Config #2
Config #1

File source, LiteRT model, CPU runtime

{
  "file-path": "/etc/media/video-mp3.mp4",
  "model": "/etc/models/yamnet.tflite",
  "labels": "/etc/labels/yamnet.json",
  "runtime": "cpu",
  "threshold": 20,
  "codec": "mp3"
}

File source, LiteRT model, CPU runtime

{
  "file-path": "/etc/media/video-mp3.mp4",
  "model": "/etc/models/yamnet.tflite",
  "labels": "/etc/labels/yamnet.json",
  "runtime": "cpu",
  "threshold": 20,
  "codec": "mp3"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The output video and classified audio are played on the screen.

Pipeline Flow

The following table lists the plugins used in the audio classification pipeline:

Plugin	Description
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for processing.
`h264parse`	• Parses the H.264 video.
`v4l2h264dec`	• Decodes the video bitstream into raw frames.
`mpegaudioparse` / `flacparse`	• Parses the audio bitstream (MP3 or FLAC) to ensure downstream elements can handle the payload.
`mpg123audiodec` / `flacdec`	• Decodes the compressed audio (MP3 or FLAC) into raw audio buffers.
`audioconvert`	• Converts raw audio buffers between various possible formats to ensure compatibility.
`audioresample`	• Resamples the audio buffers to different sample rates as required by the model.
`pulsesrc`	• Reads the live audio stream from the microphone.
`audiobuffersplit`	• Splits the incoming audio buffers into equal-sized chunks for consistent processing.
`qtimlaconverter`	1. Receives the audio stream on its sink pad. 2. Performs preprocessing on the audio stream data. 3. Converts the stream to a tensor stream for inferencing in the later stages of the pipeline.
`qtimltflite`	1. Receives the tensor stream on its sink pad. 2. Performs inferencing using the YAMNet model. 3. Produces a tensor stream with the results on its source pad.
`qtimlpostprocess`	• Uses the `yamnet` module to handle audio classification inference results: • Applies a threshold to the chosen number of results. • Creates a text overlay for the identified audio classes.
`qtivcomposer`	• Combines the text overlay for classification results and the video preview into a single composed frame.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Description
`runtime`	Use one of the following runtimes: • `cpu` • `gpu`
`Input source`	Use one of the following input sources: • `file-path`: The directory path to the video file. • `Microphone`
`threshold=<integer>`	Use any integer between 1 and 100.
`codec`	The audio codec of input video: • `MP3` (default) • `FLAC`

AI Multi-Model Applications

Daisychain Detection + Classification

The gst-ai-daisychain-detection-classification application allows you to perform cascaded object detection and classification with a camera, file source, or RTSP stream. The use case involves detecting objects and classifying the detected objects. The following figures show the pipeline workflow, which captures the video stream from the source, preprocesses it, and runs inferences using AI hardware. The results are either displayed on the screen, saved as an encoded MP4 file, or streamed over the RTSP server. For information about the plugins used in this pipeline, see Pipeline flow.

Application: gst-ai-daisychain-detection-classification

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	Yes	No	Yes	Yes	Yes
Config #2	Yes	Yes	Yes	Yes	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model files	Label files
LiteRT	detection: `yolox_quantized.tflite`, classification: `inception_v3_quantized.tflite`	`yolox.json`, `classification.json`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-daisychain-detection-classification --config-file=/etc/configs/config_daisychain_detection_classification.json

The sample application uses the /etc/configs/config_daisychain_detection_classification.json file to read the input parameters.To display all available options:

gst-ai-daisychain-detection-classification -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-daisychain-detection-classification application uses the /etc/configs/config_daisychain_detection_classification.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "input-file": "<input-video-path>",
  "detection-model": "<path-to-detection model>",
  "detection-labels": "<path-to-detection-labels>",
  "classification-model": "<path-to-classification-model>",
  "classification-labels": "<path-to-classification-labels>",
  "detection-runtime": "<Can be dsp or cpu or gpu>",
  "classification-runtime": "<Can be dsp or cpu or gpu>"
}

For USB camera input, set the video-format, resolution, and framerate parameters in the config file to match the camera capabilities, see Configure USB camera.

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
 "input-file": "/etc/media/video.mp4",
 "detection-model": "/etc/models/yolox_quantized.tflite",
 "detection-labels": "/etc/labels/yolox.json",
 "classification-model": "/etc/models/inception_v3_quantized.tflite",
 "classification-labels": "/etc/labels/classification.json",
 "detection-runtime": "dsp",
 "classification-runtime": "dsp"
}

File source, LiteRT model, CPU runtime

{
  "input-file": "/etc/media/video.mp4",
  "detection-model": "/etc/models/yolox_quantized.tflite",
  "detection-labels": "/etc/labels/yolox.json",
  "classification-model": "/etc/models/inception_v3_quantized.tflite",
  "classification-labels": "/etc/labels/classification.json",
  "detection-runtime": "cpu",
  "classification-runtime": "cpu"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The cropped video frame is overlaid on the frame and displayed on a local device.

The classification models trained on the Imagenet dataset don’t contain the person class.

Pipeline Flow

The following table lists the plugins used in the daisychain detection and classification pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`v4l2src`	• Captures the live stream from USB camera. • Uses `tee` to split the stream for inferencing.
`h264parse`	• Parses the H.264 video bitstream.
`v4l2h264dec`	• Hardware-decodes H.264 video to raw frames.
`qtimetamux`	• Multiplexes the stream.
`qtivsplit`	• Crops the full frame into smaller frames based on the detected bounding boxes (maximum 4).
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	1. After the inference runtime receives the tensor stream on its sink pad, it runs the inference. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess` (Detection)	• Handles inference results from any object detection model: • Applies a threshold to the chosen number of results. • Loads the YOLOv8 module. • Produces video frames with only bounding boxes that can be overlaid on objects. • Produces video frames with only bounding boxes that can be cropped.
`qtimlpostprocess` (Classification)	• Processes results on the cropped frame: • Applies the threshold to the chosen number of results on the cropped frame. • Loads the MobileNet-softmax module. • Produces results as video frames with classification labels. • Sends them to the sink pad of `qtivcomposer`.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`filesink`	• Receives the video stream on its sink pad and saves it as an H.264-encoded MP4 file.
`qtirtspbin`	1. Serves as a network sink. 2. Transmits UDP packets to the network.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Description
`Input source`	Supported input sources: • `input-file`: The directory path to the video file. • `rtsp-ip-port`: The address of the RTSP stream in `rtsp://<ip>:<port>/<stream>` format.
`Models and labels`	Supported model and label paths: • `detection-model`: The path to the detection model file. • `detection-labels`: The path to the detection label file. • `classification-model`: The path to the classification model file. • `classification-labels`: The path to the classification label file.
`output-type`	Use one of the following output sinks: • `waylandsink`: To display output via the Weston compositor. • `filesink`: To store output in a local file. • `rtspsink`: To stream output to a network server.
`USB camera video-format and resolution`	1. Use one of the following `video-format` options: • `nv12` • `yuy2` • `mjpeg` 2. Use the following resolution parameters: • `width`: Input USB camera source resolution width. • `height`: Input USB camera source resolution height. • `framerate`: Input USB camera source framerate.
`output-file`	• Output filename. The default output file is `output_detection.mp4`.
`output-ip-address and port`	Network configuration for RTSP output: • `output-ip-address`: Output server IP address. • `port`: Output server port.

Daisychain Detection + Pose

The gst-ai-daisychain-detection-pose application allows you to perform cascaded object detection and pose detection with a camera, file source, or an RTSP stream. The use cases involve detecting objects and estimating the body poses of the subject in an image or a video. The following figure show the application workflow, which receives the source, postprocesses it, and runs inferences on AI hardware. The results are either displayed on the screen, saved as an encoded MP4 file, or streamed over the RTSP server. For information about the plugins used in the pipeline flow, see Pipeline flow.

Application: gst-ai-daisychain-detection-pose

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	Yes	No	Yes	Yes	Yes
Config #2	Yes	Yes	Yes	Yes	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model files	Label files
LiteRT	detection: `yolox_quantized.tflite`, pose: `hrnet_pose_quantized.tflite`	detection: `yolox.json`, pose: `hrnet_pose.json`, pose: `hrnet_settings.json`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-daisychain-detection-pose --config-file=/etc/configs/config-daisychain-detection-pose.json

The sample application uses the /etc/configs/config-daisychain-detection-pose.json file to read the input parameters.To display all available options:

gst-ai-daisychain-detection-pose -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-daisychain-detection-pose application uses the /etc/configs/config-daisychain-detection-pose.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "input-file": "<input-video-path>",
  "rtsp-ip-port": "<RTSP-IP-port-address>",
  "detection-model": "<path-to-detection model>",
  "detection-labels": "<path-to-detection-labels>",
  "detection-runtime": "<It can take dsp, gpu or cpu as runtime. Runs detection model on given runtime>",
  "pose-runtime": "<It can take dsp, gpu or cpu as runtime. Runs pose model on given runtime>",
  "output-file": "<output-file-path>"
}

For QCS6490, if file-path and rtsp-ip-port are not present in the configuration file, then the camera input is selected.

For USB camera input, set the video-format, resolution, and framerate parameters in the config file to match the camera capabilities, see Configure USB camera.

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
 "input-file": "/etc/media/video.mp4",
 "pose-runtime":"dsp",
 "detection-runtime":"dsp",
 "detection-model": "/etc/models/yolox_quantized.tflite",
 "detection-labels": "/etc/labels/yolox.json",
 "pose-model": "/etc/models/hrnet_pose_quantized.tflite",
 "pose-labels": "/etc/labels/hrnet_pose.json",
 "pose-settings-path":"/etc/labels/hrnet_settings.json"
}

File source, LiteRT model, CPU runtime

{
 "input-file": "/etc/media/video.mp4",
 "pose-runtime":"cpu",
 "detection-runtime":"cpu",
 "detection-model": "/etc/models/yolox_quantized.tflite",
 "detection-labels": "/etc/labels/yolox.json",
 "pose-model": "/etc/models/hrnet_pose_quantized.tflite",
 "pose-labels": "/etc/labels/hrnet_pose.json",
 "pose-settings-path":"/etc/labels/hrnet_settings.json"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The cropped video frame is overlaid on the frame and displayed on a local device.

Pipeline Flow

The following table lists the plugins used in the daisychain detection and pose pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`v4l2src`	• Captures the live stream from USB camera. • Uses `tee` to split the stream for inferencing.
`h264parse`	• Parses the H.264 video bitstream.
`v4l2h264dec`	• Hardware-decodes H.264 video to raw frames.
`qtimetamux`	• Multiplexes the stream.
`qtivsplit`	• Crops the full frame into smaller frames based on the detected bounding boxes (maximum 4).
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	1. After the inference runtime receives the tensor stream on its sink pad, it runs the inference. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess` (Detection)	• Handles inference results from any object detection model: • Applies a threshold to the chosen number of results. • Loads the YOLOv8 module. • Produces video frames with only bounding boxes that can be overlaid on objects. • Produces video frames with only bounding boxes that can be cropped.
`qtimlpostprocess` (Pose)	• Applies a threshold to the chosen number of results. • Loads corresponding modules for various pose detection models. In this specific use case, `qtimlpostprocess` does the following: 1. Loads the HRNet module. 2. Produces results in the form of video frames with drawn poses. 3. Sends the results to the sink pad of `qtivcomposer` for further processing or display.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`filesink`	• Receives the video stream on sink pad and saves it as an H.264-encoded MP4 file.
`qtirtspbin`	1. Serves as a network sink. 2. Transmits UDP packets to the network.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Description
`Input source`	Use one of the following input sources: • `input-file`: The directory path to the video file. • `rtsp-ip-port`: The address of the RTSP stream in `rtsp://<ip>:<port>/<stream>` format. • `enable-usb-camera`: Set to `TRUE` or `FALSE` to enable/disable USB camera input.
`Models and labels`	Supported model and label paths: • `detection-model`: The path to the detection model file. • `detection-labels`: The path to the detection label file. • `pose-model`: The path to the pose model file. • `pose-labels`: The path to the pose label file.
`Output source`	• `output-file`: The directory path to save the output file. Note: The display is not enabled if this field is left empty.
`USB camera video-format and resolution`	1. Use one of the following `video-format` options: • `nv12` • `yuy2` • `mjpeg` 2. Use the following resolution fields: • `width`: Input USB camera source resolution width. • `height`: Input USB camera source resolution height. • `framerate`: Input USB camera source framerate.
`detection-runtime and classification-runtime`	Hardware runtime configuration: • Takes `CPU`, `GPU`, or `DSP` as input. • Executes the respective use case model in the specified runtime for optimized inference.

Multistream Inference

The gst-ai-multistream-inference application shows AI inference (object detection and classification) on up to 32 input streams coming from camera, file, or RTSP stream. The following figure shows the pipeline, which receives several input streams, preprocesses them, runs AI inferences, combines the streams, and merges them all into a single video output. The maximum number of input streams supported on each SoC as verified on 1080P and 720P are follows:

QCS6490–8
Dragonwing IQ-8275–16
Dragonwing IQ-9075–32
This application isn’t supported in Config #1 for the QLI 2.0 GA release because CPU runtime is not supported.
The output is displayed on an HDMI display, saved as an H.264 encoded MP4 file, or converted into an RTSP stream.

For information about the plugins used in this pipeline, see Pipeline flow.

Application: gst-ai-multistream-inference

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #2	Yes	Yes	No	Yes	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	detection: `yolox_quantized.tflite` classification: `inception_v3_quantized.tflite`	detection: `yolox.json` classification: `classification.json`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-multistream-inference --config-file=/etc/configs/config-multistream-inference.json

The sample application uses the /etc/configs/config-multistream-inference.json file to read the input parameters.To display all available options:

gst-ai-multistream-inference -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-multistream-inference application uses the /etc/configs/config-multistream-inference.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "input-file-path": ["<input file 1>", "<input file 2>"],
  "input-rtsp-path": ["<rtsp stream 1>", "<rtsp stream 2 >"],
  "input-type": "<h264 or h265>",
  "model": "<path to model file>",
  "labels": "<path to label file>",
  "output-display": "<0 or 1>",
  "output-file-path": "<output file path>",
  "output-ip-address": "<ip address of test kit>",
  "output-port-number": "<port number over which rtsp stream can be listened>",
  "use-case": "<0 or 1>"
}

Available configurations

Config #2

Object Detection on 8 H.264 file inputs, LiteRT model, DSP runtime

{
  "input-file-path":
    [
      "/etc/media/video.mp4",
      "/etc/media/video.mp4",
      "/etc/media/video.mp4",
      "/etc/media/video.mp4",
      "/etc/media/video.mp4",
      "/etc/media/video.mp4",
      "/etc/media/video.mp4",
      "/etc/media/video.mp4"
    ],
  "model": "/etc/models/yolox_quantized.tflite",
  "labels": "/etc/labels/yolox.json",
  "input-type": "h264",
  "output-display": 1,
  "use-case": 0
}

Expected Output

Pipeline Flow

The following table lists the plugins used in the metadata parser pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux`, which demultiplexes the stream. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`h264parse`	• Parses the H.264 video.
`v4l2h264dec`	• Decodes the video.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data. This preprocessing is done when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream on its source pad. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	1. After the inference runtime receives the tensor stream on its sink pad, it runs the inference. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	Handles the inference results from any object detection, classification, pose detection, and segmentation model. Detection use case: • Applies a threshold to the chosen number of results. • Loads the YOLOv8 module. • Produces video frames with only bounding boxes that can be overlaid on objects, sending them to the sink pad of `qtivcomposer`. Classification use case: • Applies the threshold to the chosen number of results. • Loads the MobileNet-softmax module. • Produces results as video frames with classification labels, sending them to the sink pad of `qtivcomposer`.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. `waylandsink` submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.
`filesink`	1. Receives the video stream on its sink pad. 2. Saves the stream as a H.264-encoded MP4 file.
`qtirtspbin`	1. Serves as a network sink. 2. Transmits UDP packets to the network.

Config JSON Field Description

Field	Values / Description
`Input source`	Use one of the following input sources: • `num-camera`: The number of inputs from the camera. • `camera`: The input camera if `num-camera=1`. • `input-file-path`: The directory path to the video file. • `input-rtsp-path`: The address of the RTSP stream: `rtsp://<ip>:<port>/<stream>`
`input-type`	The video encoding type for file and RTSP input: • `H.264` • `H.265`
`Output`	Use one of the following outputs: • `output-file-path`: The directory path to save the output file. • `output-ip-address`: The IP address of the device on which the RTSP stream can be played. • `output-port-number`: The port number of the device on which the RTSP stream can be played. • `output-display`: The connected display device for preview. Select `1` to enable this option.

Multi-Stream Batch Inference

The gst-ai-multistream-batch-inference application shows batched AI inference (object detection and segmentation) on up to 24 input streams from video files. The following figure shows the pipeline, which receives several input streams, preprocesses them, runs AI inferences, combines the streams with inference, and merges them into a single video output. The maximum number of input streams supported on each SoC are follows: QCS6490–8 Dragonwing IQ-8275–4 Dragonwing IQ-9075–4 The output is displayed either on an HDMI display or saved as an H.264 encoded MP4 file. For information about the plugins used in this pipeline, see Pipeline flow.

This application isn’t supported in Config #1 for the QLI 2.0 GA release because CPU runtime is not supported.

Application: gst-ai-multistream-batch-inference

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #2	Yes	No	No	No	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	segmentation: `deeplabv3_plus_mobilenet_quantized.tflite` detection: `yolov8_det_w8a8.tflite`	segmentation:`deeplabv3_resnet50.json` detection: `yolov8.json`
Qualcomm AI Engine direct	segmentation: `deeplabv3_plus_mobilenet_quantized.bin` detection: `yolov8_det_quantized.bin`	segmentation:`deeplabv3_resnet50.json` detection: `yolov8.json`
Qualcomm Neural Processing SDK	segmentation: `deeplabv3_plus_mobilenet_quantized.dlc` detection: `yolov8_det_quantized.dlc`	segmentation:`deeplabv3_resnet50.json` detection: `yolov8.json`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-multistream-batch-inference --config-file=/etc/configs/config-multistream-batch-inference.json

The sample application uses the /etc/configs/config-multistream-batch-inference.json file to read the input parameters.To display all available options:

gst-ai-multistream-batch-inference -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-multistream-batch-inference application uses the /etc/configs/config-multistream-batch-inference.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "output-type": "wayland or filesink",
  "out-file":"<Path to output file if output-type is filesink>",
  "pipeline-info":[
    {
      "id": "<Batch id takes value from 0 to 5>",
      "Input type": "<Input source type like file>",
      "input-file-path": [
        {
        "<stream[i]: Comma separated input file/RTSP path>"
        }
      ],
      "mlframework": "<tflite or snpe or qnn>",
      "model-path": "<path-to-model-file>",
      "labels": "<path-to-label-file>",
      "post processing plugin": "qtimlpostprocess"
    }
  ]
}

For 16 and 24 streams, add the required elements in the pipeline-info parameter. The id parameter takes the values from 0 to 5 for each added batch.

Available configurations

Config #2

File source, LiteRT model, DSP runtime

{
  "output-type":"wayland",
  "pipeline-info":[
    {
      "id":0,
      "input-type":"file",
      "input-file-path":[
        {
            "stream-0":"/etc/media/video.mp4",
            "stream-1":"/etc/media/video.mp4",
            "stream-2":"/etc/media/video.mp4",
            "stream-3":"/etc/media/video.mp4"
        }
      ],
      "mlframework":"tflite",
      "model-path":"/etc/models/yolov8_det_w8a8.tflite",
      "labels-path":"/etc/labels/yolov8.json",
      "post-process-plugin": "qtimlpostprocess"
    }
  ]
}

Expected Output

Pipeline Flow

The following table lists the plugins used in the metadata parser pipeline:

Plugin	Description
`filesrc`	1. Captures the video stream using `filesrc`. 2. `qtdemux` demultiplexes the stream. 3. Uses `tee` to split the stream for inferencing.
`h264parse`	Parses the H.264 video.
`v4l2h264dec`	Decodes the video.
`qtibatch`	1. Reads input from the streams on its sink pad. 2. Batches the streams for preprocessing.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization 3. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	1. After the inference runtime receives the tensor stream on its sink pad, it runs the inference. 2. Produces a tensor stream with the inference results on its source pad.
`qtimldemux`	1. Demultiplexes the batched output. 2. Splits the output corresponding to the input streams.
`qtimlpostprocess`	Converts the inference tensors received on the sink pad into video formats that the multimedia plugins can use for further processing.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.
`filesink`	Takes the video stream received on its sink pad and saves it as an H.264-encoded MP4 file.

Config JSON Field Description

Field	Values/Description
`output-type`	Use one of the following output types: • `Wayland`: Displays output on Weston. • `filesink`: Encodes the output in a video file.
`out-file`	The file path to save the output file.
`pipeline-info`	Provides the pipeline information: 1. `Stream id`: Ranges from 0 to 5. 2. `Input-type`: The input source file. 3. `Input-file-path`: The array of the input file path.
`mlframework`	Use one of the following frameworks: • `tflite` • `qnn` • `snpe`
`model-path`	The path to the model file.
`labels-path`	The path to the labels file.

Multi input/output object detection

The gst-ai-multi-input-output-object-detection application allows you to perform object detection, object classification, pose detection, and image segmentation on an input stream from different sources such as a camera, a file, or an RTSP network. The use cases implement the LiteRT models for object detection, image segmentation, classification, and pose detection. The following figure shows the pipeline workflow, which captures video streams for inferencing from different sources such as camera, file, or RTSP. For information about the plugins used in the pipeline, see Pipeline flow.

This application isn’t supported in Config #1 for the QLI 2.0 GA release because CPU runtime is not supported.

Application: gst-ai-multi-input-output-object-detection

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #2	Yes	Yes	No	Yes	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	`yolov5.tflite`	`yolov5.json`

Prerequisites

Update the following commands according to the Python version in your Linux host computer.

Create the Python 3.8 virtual environment:

sudo apt-get install python3.8

sudo apt-get install python3.8-venv

python3.8 -m venv py3.8

source py3.8/bin/activate

Generate the yolov5.tflite model:

git clone https://github.com/ultralytics/yolov5.git

cd yolov5

python -m pip install -r requirements.txt tensorflow-cpu

python export.py --weights yolov5m.pt --img 320 --include tflite --int8 --data data/coco128.yaml

In the terminal of the host computer, run the following command to push the model to the target device:

scp yolov5m-int8.tflite root@<ip address of the device>:/etc/models/yolov5.tflite

If any model isn’t available after downloading the script file, you can download the model from IoT– Qualcomm AI Hub.

In the terminal of the host computer, run the following command to push the model files to the target device:

scp <model_filename> root@< address of target device>:/etc/models

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

Enter SSH shell and copy the YOLOX label files to YOLOv5:

cp /etc/labels/yolox.json /etc/labels/yolov5.json

Run the application:

gst-ai-multi-input-output-object-detection --config-file=/etc/configs/config-multi-input-output-object-detection.json

The sample application uses the /etc/configs/config-multi-input-output-object-detection.json file to read the input parameters.To display all available options:

gst-ai-multi-input-output-object-detection -h

To stop the use case, press CTRL + C.

Pull the files from the target device, once you are done running the application:

scp root@<ip address of target device>:/etc/media/out.mp4 <destination directory>

Configurations

The gst-ai-multi-input-output-object-detection application uses the /etc/configs/config-multi-input-output-object-detection file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "num-camera": "<number-of-cameras>",
  "camera-id": "<specific-camera-id>",
  "input-file-path": "<JSON-array-of-input-file>",
  "input-rtsp-path": "<JSON-array-of-RTSP-URLs>",
  "model": "<path-to-model-file>",
  "labels": "<path-to-label-file>",
  "output-file-path": "<path-to-output-file>",
  "output-ip-address": "<ip-address-of-output>",
  "output-port-number": "<port number over which rtsp stream can be listened>",
  "output-display": "<true or false>"
}

Ensure that the total number of input streams from the camera, RTSP, and file source doesn’t exceed 6.

For QCS6490, if file-path and rtsp-ip-port are not present in the configuration file, then the camera input is selected.

Available configurations

Config #2

File source, LiteRT model, DSP runtime

{
  "input-file-path":
      [
          "/etc/media/video1.mp4",
          "/etc/media/video2.mp4"
      ],
  "model": "/etc/models/yolov5.tflite",
  "labels": "/etc/labels/yolov5.json",
  "output-display": true,
  "output-file-path": "/etc/media/output.mp4",
  "output-ip-address": "127.0.0.1",
  "output-port-number": "8554"
}

Expected Output

Based on the use case, the results are either displayed on an HDMI screen, saved as an H.264 encoded MP4 file, or streamed over the RTSP server.

Pipeline Flow

The following table lists the plugins used in the metadata parser pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from the camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`. • Followed by `qtdemux`, which demultiplexes the stream. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`. • Followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`h264parse`	Parses the H.264 video.
`v4l2h264dec`	Decodes the video.
`qtimlvconverter`	• Receives the video stream on its sink pad. • Performs the following preprocessing on the stream data when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization • The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	• Runs on LiteRT and uses the `yolov5.tflite` model for object detection. • After the inference runtime receives the tensor stream on its sink pad, it runs the inference. • Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	• Converts the inference tensors that it receives on its sink pad into video formats that the multimedia plugins can process later.
`qtivcomposer`	• Composes frames with contents from its sink pads. • Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	• `waylandsink` submits the video stream received on its sink pad to the Wayland compositor. • Renders the video stream on a local display.
`filesink`	Takes the video stream that it receives on its sink pad and saves it as an H.264-encoded MP4 file.
`qtirtspbin`	• Serves as a network sink. • Transmits UDP packets to the network.

Config JSON Field Description

Field	Values/Description
`Input source`	Use one of the following input sources: • `num-camera`: The number of inputs from the camera. Select either `1` or `2`. • `camera-id`: The id of the test camera. Select either `0` or `1`. • `input-file-path`: The directory path to the video file. • `input-rtsp-path`: The address of the RTSP stream: `rtsp://<ip>:<port>/<stream>`
`Models and labels`	• `model`: The path to the model file. • `labels`: The path to the label file.
`Output`	Use one of the following outputs: • `output-file-path`: The directory path to save the output file. • `output-ip-address`: The IP address of the device on which the RTSP stream can be played. • `output-port-number`: The port number of the device on which the RTSP stream can be played. • `output-display`: The connected display device for preview.

Parallel Inferencing

The gst-ai-parallel-inference application allows you to perform object detection, object classification, pose detection, and image segmentation on an input stream from different sources such as a camera, a file, or an RTSP network. The use cases implement the LiteRT models for object detection, image segmentation, classification, and pose detection. The following figure shows the pipeline, which receives input streams from a camera, file, or an RTSP stream, performs the parallel inferencing for the four use cases, and displays the results side by side on the screen.

This application isn’t supported in Config #1 for the QLI 2.0 GA release because CPU runtime is not supported.

For information about the plugins used in this pipeline, see Pipeline flow.

Application: gst-ai-parallel-inference

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #2	Yes	Yes	No	Yes	No	Yes	No

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	detection: `yolox_quantized.tflite` classification: `inception_v3_quantized.tflite` segmentation: `deeplabv3_plus_mobilenet_quantized.tflite` pose: `hrnet_pose_quantized.tflite`	detection: `yolox.json` classification: `classification.json` segmentation: `deeplabv3_resnet50.json` pose: `hrnet_pose.json`, `hrnet_settings.json`

Run the application on the target device

Download artifacts

Ensure that you complete the Prerequisites. This downloads all required artifacts to the target device.

Run the application

gst-ai-parallel-inference --config-file=/etc/configs/config-parallel-inference.json

The sample application uses the /etc/configs/config-parallel-inference.json file to read the input parameters.To display all available options:

gst-ai-parallel-inference -h

To stop the use case, press CTRL + C.

Configurations

The gst-ai-parallel-inference application uses the /etc/configs/config-parallel-inference.json file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "camera": "<camera-id>",
  "file-path": "<input-video-path>",
  "rtsp-ip-port": "<RTSP-IP-Port-address>",
  "detection-model": "<path-to-detection model>",
  "detection-labels": "<path-to-detection-labels>",
  "pose-model": "<path-to-pose-model>",
  "pose-labels": "<path-to-pose-label>",
  "pose-settings-path": "<path-to-pose-settings-file>",
  "segmentation-model": "<path-to-segmentation-model>",
  "segmentation-labels": "<path-to-segmentation-labels>",
  "classification-model": "<path-to-classification-model>",
  "classification-labels": "<path-to-classification-labels>"
}

For QCS6490, if file-path and rtsp-ip-port are not present in the configuration file, then the camera input is selected.

Available configurations

Config #2

File source, LiteRT model, DSP runtime

{
    "file-path": "/etc/media/video.mp4",
    "detection-model": "/etc/models/yolox_quantized.tflite",
    "detection-labels": "/etc/labels/yolox.json",
    "pose-model": "/etc/models/hrnet_pose_quantized.tflite",
    "pose-labels": "/etc/labels/hrnet_pose.json",
    "pose-settings-path": "/etc/labels/hrnet_settings.json",
    "segmentation-model": "/etc/models/deeplabv3_plus_mobilenet_quantized.tflite",
    "segmentation-labels": "/etc/labels/deeplabv3_resnet50.json",
    "classification-model": "/etc/models/inception_v3_quantized.tflite",
    "classification-labels": "/etc/labels/classification.json"
}

Expected Output

After performing the four parallel inferences, the results are displayed side by side on the screen.

Pipeline Flow

The following table lists the plugins used in the metadata parser pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from the camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`. • Followed by `qtdemux`, which demultiplexes the stream. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`. • Followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`h264parse`	Parses the H.264 video.
`v4l2h264dec`	Decodes the video.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization 3. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	• After the inference runtime receives the tensor stream on its sink pad, it runs the inference. • Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess` — detection	a. Receives the inference tensors from the object detection model. b. Converts the inference tensors on its sink pad into formats such as video or text that the multimedia plugins can process later. c. Applies the threshold to the chosen number of results. d. Loads the corresponding modules for detection models. In this use case, `qtimlpostprocess` does the following: • Loads the YOLOv8 submodule. • Produces results as structures of text. • Sends them to the sink pad of `qtimetamux`.
`qtimlpostprocess` — classification	a. Receives the inference tensors from the classification model. b. Converts the inference tensors on its sink pad into formats such as video or text that the multimedia plugins can process later. c. Applies the threshold to the chosen number of results. d. Loads the corresponding modules for classification models. In this use case, `qtimlpostprocess` does the following: • Loads the submodule of the model. • Produces results as video frames with classification labels. • Sends them to the sink pad of `qtivcomposer`.
`qtimlpostprocess` — segmentation	a. Receives the inference tensors on its sink pad. b. Converts the inference tensors into video formats that the multimedia plugins can process later. c. Produces the semantic segmentations for the frame. d. Loads the corresponding modules for the segmentation models. In this use case, `qtimlpostprocess` does the following: • Loads the deeplab-argmax submodule. • Produces video frames with segmentation masks. • Sends them to the sink pad of `qtivcomposer`.
`qtimlpostprocess` — pose	a. Receives the inference tensors on its sink pad. b. Converts the inference tensors into video formats that the multimedia plugins can process later. c. Applies the threshold to the chosen number of results. d. Loads the corresponding modules for various pose estimation models. In this use case, `qtimlpostprocess` does the following: • Loads the HRNet module. • Produces results as video frames with poses drawn. • Sends them to the sink pad of `qtivcomposer`.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values/Description
`Input source`	Use one of the following input sources: • `camera`: Primary (`0`) or secondary (`1`). • `file-path`: The directory path to the video file. • `rtsp-ip-port`: The address of the RTSP stream: `rtsp://<ip>:<port>/<stream>`
`Models and labels`	• `detection-model`: The path to the detection model. • `detection-labels`: The path to the detection label. • `pose-model`: The path to the pose model. • `pose-labels`: The path to the pose labels. • `segmentation-model`: The path to the segmentation model. • `segmentation-labels`: The path to the segmentation labels. • `classification-model`: The path to the classification model. • `classification-labels`: The path to the classification labels.

Hardware benchmarking application

The hardware benchmarking application monitors the device hardware usage for a defined set of sample applications to capture metrics such as CPU/GPU/NPU usage and device thermals. These metrics explain the resource usage and throttling, which help to tune your AI use cases according to the requirements. The following figure shows the pipeline, which processes the input from a set of USB cameras to generate various outputs.

This application isn’t supported in Config #1 for the QLI 2.0 GA release because CPU runtime is not supported.

For information about the plugins used in this pipeline, see Pipeline flow.

Try me

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	• inception_v3_quantized.tflite • deeplabv3_plus_mobilenet_quantized.tflite • hrnet_pose_quantized.tflite • midas_quantized.tflite • yolox_quantized.tflite	• classification.json • deeplabv3_resnet50.json • hrnet_pose.json • monodepth.json • yolox.json

Setup the target device

To access the target device from your Linux host computer, set up SSH. For instructions, see Sign in using SSH.

If SSH is already set up, you can skip this step.

Use the HDMI port to connect the display to the device. For instructions, see Set up HDMI display.If you face issues with display, see Troubleshoot display issues.

Connect two USB cameras and a mouse to the target device.

Install the Qualcomm® Profiler on the Linux host computer. For installation instructions see Qualcomm Profiler.After connecting the device to the PC, run InstallerLE from the following locations:

For Linux:

cd /opt/qcom/Shared/QualcommProfiler/API/target-le
./InstallerLE

For Windows:

cd “C:\Program Files (x86)\Qualcomm\Shared\QualcommProfiler\API\target-le”
.\InstallerLE.exe

Run the application on the target device

Clone the repository for the demo application and push it to the target device:

git clone https://github.com/Avnet/QCS6490-Vision-AI-Demo.git

cd QCS6490-Vision-AI-Demo

git checkout QLI_2.0

scp -r ../QCS6490-Vision-AI-Demo root@<ip address of target device>:/opt

cd /opt/QCS6490-Vision-AI-Demo && bash install.sh

Start the application:

bash launch_visionai_with_env.sh

Select the preferred sample applications from the Camera 1 and Camera 2 drop-downs. The system thermal and hardware usage details appear at the bottom of the screen.You may run different sample applications to check the output and understand the hardware utilization.

Example 1: Choose the Camera option from Camera 1 and Camera 2 drop-down lists to observe the preview streams on the screen.
Example 2: Choose any sample application from Camera 1 and Camera 2 drop-down lists to observe the AI inferencing camera streams on the screen.

For more information and features of the application, select the Info icon.

Select the Exit icon to close the application.

Pipeline Flow

The following table lists the plugins used in the metadata parser pipeline:

Plugin	Description
`v4l2src`	• Captures the live stream from the USB camera. • Uses `tee` to split the stream for inferencing.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization 3. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite` `qtimltsnpe` `qtimlqnn`	• After the inference runtime receives the tensor stream on its sink pad, it runs the inference. • Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess` — detection	a. Receives the inference tensors from the object detection model. b. Converts the inference tensors on its sink pad into formats such as video or text that the multimedia plugins can process later. c. Applies the threshold to the chosen number of results. d. Loads the corresponding modules for detection models. In this use case, `qtimlpostprocess` does the following: • Loads the YOLO (`YOLOv5`, `YOLOv8`, `YOLOX`, or `YOLO-NAS`) submodule. • Produces results as structures of text. • Sends them to the sink pad of `qtimetamux`.
`qtimlpostprocess` — classification	a. Receives the inference tensors from the classification model. b. Converts the inference tensors on its sink pad into formats such as video or text that the multimedia plugins can process later. c. Applies the threshold to the chosen number of results. d. Loads the corresponding modules for classification models. In this use case, `qtimlpostprocess` does the following: • Loads the submodule of the model. • Produces results as video frames with classification labels. • Sends them to the sink pad of `qtivcomposer`.
`qtimlpostprocess` — segmentation	a. Receives the inference tensors on its sink pad. b. Converts the inference tensors into video formats that the multimedia plugins can process later. c. Produces the semantic segmentations for the frame. d. Loads the corresponding modules for the segmentation models. In this use case, `qtimlpostprocess` does the following: • Loads the `deeplab-argmax` submodule. • Produces video frames with segmentation masks. • Sends them to the sink pad of `qtivcomposer`.
`qtimlpostprocess` — pose	a. Receives the inference tensors on its sink pad. b. Converts the inference tensors into video formats that the multimedia plugins can process later. c. Applies the threshold to the chosen number of results. d. Loads the corresponding modules for various pose estimation models. In this use case, `qtimlpostprocess` does the following: • Loads the `HRNet` module. • Produces results as video frames with poses drawn. • Sends them to the sink pad of `qtivcomposer`.

Troubleshooting

Model not available after running the download script?

If any model isn’t available after downloading the script file, you can download the model manually from IoT — Qualcomm AI Hub and push it to the target device:

scp <model filename> root@<ip addr of the target device>:/etc/models

For example:

scp mobilenet_v2_quantized.tflite root@<ip addr of the target device>:/etc/models

Running sample applications from the UART shell

Remount the file system with read/write permissions:For Qualcomm Linux:

mount -o remount,rw /

Cannot locate the qticamsrc plugin?

If you cannot locate the qticamsrc plugin, ensure that the camera server is running and clear the GStreamer cache using the following commands:

ps -ef | grep cam-server

rm ~/.cache/gstreamer-1.0/registry.aarch64.bin

GStreamer logging and debugging

To enable basic GStreamer logging, run the following before launching the application:

export GST_DEBUG=2

To increase verbosity for specific plugins, use a comma-separated list with log levels (1–7):

export GST_DEBUG=3,qticamsrc:5,qtimlvconverter:5,qtimltflite:5

To redirect logs to a file for offline analysis:

export GST_DEBUG=3 
export GST_DEBUG_FILE=/tmp/gst_classification_debug.log 

​Prerequisites

​AI Vision Applications

​Object Detection

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

​Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​Image Classification

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

​Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​Face Detection

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

​Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​Semantic Segmentation

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

​Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​Pose Detection

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

​Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​Mono Depth

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

​Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​Super Resolution

​Input and Output Capabilities

​Sample Model Files

​Run the application on the target device

​Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​AI Event Encoder

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

​Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​Metadata Parser

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

​Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​AI USB Camera

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

​Configurations

​Expected Output

Prerequisites

AI Vision Applications

Object Detection

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Configurations

Expected Output

Image Classification

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Configurations

Expected Output

Face Detection

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Configurations

Expected Output

Semantic Segmentation

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Configurations

Expected Output

Pose Detection

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Configurations

Expected Output

Mono Depth

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Configurations

Expected Output

Super Resolution

Input and Output Capabilities

Sample Model Files

Run the application on the target device

Configurations

Expected Output

AI Event Encoder

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Configurations

Expected Output

Metadata Parser

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Configurations

Expected Output

AI USB Camera

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Configurations

Expected Output

Face Recognition

Input and Output Capabilities

Sample Model and Label Files

Register a face for facial recognition

Run the application on the target device

Configurations

Expected output

AI Audio Applications

Audio Classification

Sample Model and Label Files

Run the application on the target device

Configurations

Expected Output

AI Multi-Model Applications

Daisychain Detection + Classification

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device