AI - Qualcomm Intelligent Multimedia SDK

This section covers QIM SDK AI sample applications that demonstrate vision, audio, and multi-model inference on Qualcomm platforms using LiteRT models accelerated on Qualcomm AI hardware. The following tables list all available GStreamer C/C++ AI applications and their platform support. Select the appropriate configuration tab for your setup.

Config #1
Config #2

Application	Source code	Description	Input sources	Runtime	QCS6490 / IQ-8275 / IQ-9075	IQ-X7181/IQ-X5121	IQ-615 / IQ-2390
Image classification	`gst-ai-classification`	Classification on streams from a file source or RTSP.	Camera, file, RTSP, USB	cpu	✓	✓	✓
Object detection	`gst-ai-object-detection`	Object detection on streams from a file source or RTSP.	Camera, file, RTSP, USB	cpu	✓	✓	×
Pose detection	`gst-ai-pose-detection`	Pose detection on streams from a file source or RTSP.	File, RTSP, USB	cpu	✓	✓	✓
Image segmentation	`gst-ai-segmentation`	Image segmentation on streams from a file source or RTSP.	File, RTSP	cpu	✓	✓	×
Daisy chain detection + classification	`gst-ai-daisychain-detection-classification`	Cascaded object detection and classification.	File, RTSP, USB	cpu	✓	✓	×
Daisy chain detection + pose	`gst-ai-daisychain-detection-pose`	Cascaded object detection and pose detection.	File, RTSP, USB	cpu	✓	✓	×
Monodepth	`gst-ai-monodepth`	Monocular depth estimation from file or RTSP.	File, RTSP	cpu	✓	✓	×
Face detection	`gst-ai-face-detection`	Face detection from file or RTSP.	File, RTSP	cpu	✓	✓	✓
Audio classification	`gst-ai-audio-classification`	Audio event classification from microphone or file.	Audio, file	cpu	✓	✓	✓
Metadata parsing	`gst-ai-metadata-parser-example`	Parse ML metadata and count people from file or RTSP.	File, RTSP	cpu	✓	✓	×
AI USB camera	`gst-ai-usb-camera-app`	USB camera streaming with optional object detection.	USB	cpu	✓	✓	×
AI event encoder	`gst-ai-event-encoder`	Encode video only when a person is detected.	File, RTSP	cpu	✓	✓	×

Application	Source code	Description	Input sources	Runtime	QCS6490 / IQ-8275 / IQ-9075	IQ-X7181/IQ-X5121	IQ-615 / IQ-2390
Image classification	`gst-ai-classification`	Classification on streams from a camera, file, or RTSP.	Camera, file, RTSP, USB	dsp, gpu, cpu	✓	✓	✓
Object detection	`gst-ai-object-detection`	Object detection on streams from a camera, file, or RTSP.	Camera, file, RTSP, USB	dsp, gpu, cpu	✓	✓	×
Pose detection	`gst-ai-pose-detection`	Pose detection on streams from camera, file, or RTSP.	Camera, file, RTSP, USB	dsp, gpu, cpu	✓	✓	✓
Image segmentation	`gst-ai-segmentation`	Image segmentation on streams from camera, file, or RTSP.	Camera, file, RTSP	dsp, gpu, cpu	✓	✓	×
Parallel inferencing	`gst-ai-parallel-inference`	Run multiple AI models simultaneously on one stream.	Camera, file, RTSP	dsp	✓	✓	×
Multi input/output object detection	`gst-ai-multi-input-output-object-detection`	Object detection across multiple I/O stream combinations.	Camera, file, RTSP	dsp	✓	✓	×
Daisy chain detection + classification	`gst-ai-daisychain-detection-classification`	Cascaded object detection and classification.	Camera, file, RTSP, USB	dsp, gpu, cpu	✓	✓	×
Daisy chain detection + pose	`gst-ai-daisychain-detection-pose`	Cascaded object detection and pose detection.	Camera, file, RTSP, USB	dsp, gpu, cpu	✓	✓	×
Monodepth	`gst-ai-monodepth`	Monocular depth estimation from camera, file, or RTSP.	Camera, file, RTSP	dsp, gpu, cpu	✓	✓	×
Video super-resolution	`gst-ai-superresolution`	Upscale low-resolution video to high resolution.	File	dsp	✓	✓	×
Multistream inference	`gst-ai-multistream-inference`	AI inference across multiple simultaneous streams.	Camera, file, RTSP	dsp	✓	✓	×
Multistream batch inference	`gst-ai-multistream-batch-inference`	Batched AI inference on up to 24 file streams.	File	dsp	✓	✓	×
Face detection	`gst-ai-face-detection`	Face detection from camera, file, or RTSP.	Camera, file, RTSP	dsp, gpu, cpu	✓	✓	✓
Face recognition	`gst-ai-face-recognition`	Face recognition from camera or RTSP.	Camera, RTSP	dsp	✓	✓	✓
Audio classification	`gst-ai-audio-classification`	Audio event classification from microphone or file.	Audio, file	gpu, cpu	✓	✓	✓
Metadata parsing	`gst-ai-metadata-parser-example`	Parse ML metadata and count people.	Camera, file, RTSP	dsp, gpu, cpu	✓	✓	×
AI USB camera	`gst-ai-usb-camera-app`	USB camera streaming with optional object detection.	USB	dsp, gpu, cpu	✓	✓	×
Gesture Recognition	`gst-gesture-recognition`	Hand Gesture Recognition.	Camera, file, RTSP, USB	gpu	✓	✓	✓
Hardware benchmarking Application	`QCS6490-Vision-AI-Demo`	A GUI application to monitor hardware utilization with sample applications	Camera and USB	dsp	✓	✓	×
AI event encoder	`gst-ai-event-encoder`	Encode video only when a person is detected.	Camera, file, RTSP	dsp, gpu, cpu	✓	✓	×

Prerequisites

Try me

Some of the steps in the pre-requisties will be removed from future releases once the necessary fixes are mainlined.

Set up Wi-Fi

Connect to the Wireless Access Point (Wi-Fi Router):

nmcli dev wifi connect <WiFi-SSID> password <WiFi-password>

Check the connection and device status:

nmcli -p device

Login to the target deviceLocate the IP address of the device according to the type of network connection, using the device shell on the Linux host:

ip address show

Use the IP address from the Linux host to establish an SSH connection to the device:

ssh root@<ip-address>

Example:

ssh root@192.168.0.222

Connect to the SSH shell using the following password:

oelinux123

Download Models and Artifacts

On the target device, obtain the download_artifacts_2.x.sh script, set executable permissions, and run it to download the model, media, and label files:

cd /tmp/
curl -L -O https://raw.githubusercontent.com/qualcomm/sample-apps-for-qualcomm-linux/refs/heads/main/qualcomm-linux/scripts/download_artifacts_2.x.sh
chmod +x download_artifacts_2.x.sh
./download_artifacts_2.x.sh

On Qualcomm Ubuntu, run the above command with sudo.

Enable qticamsrc

In the terminal of the target device, run the following command to enable the qticamsrc on Config #2:

echo -n "camx" > /var/data
efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -w -f /var/data
efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -p
sync
reboot

Enable Audio and GPU Delegate

In the terminal of the target device, run the following command to enable audio:

systemctl stop pipewire wireplumber pipewire.socket pipewire-manager.socket
chmod 777 /dev/dma_heap/system
adsprpcd audiopd &
systemctl start pipewire wireplumber
wpctl status

To set the default devices for sink and source, get the device numbers from wpctl status and run the following command:

wpctl set-default <device ID>

In the terminal of the target device, run the following command to enable the GPU delegate and backend:

mount -o rw,remount /usr

export OCL_ICD_FILENAMES=/usr/lib/libOpenCL_adreno.so.1

(Optional) Download the YOLOv8 model

Use one of the following methods:

Download with a script
Export with AI Hub APIs
Generate a batch model

Create a Qualcomm AI Hub account

Create a Qualcomm AI Hub account.

Copy your API key

Go to Settings in the upper right corner and copy your API key.

Run the export script on the host computer

curl -L -O https://raw.githubusercontent.com/qualcomm/sample-apps-for-qualcomm-linux/refs/heads/main/qualcomm-linux/scripts/export_model.sh
chmod +x export_model.sh

Replace <API_KEY> with your API key:

./export_model.sh --api-key=<API_KEY>

The models are downloaded to the export_assets directory.

Copy the model to the ${HOME}/Downloads/qimsdk_samples/models/ directory on the device

scp <working-directory>/export_assets/yolov8_det-tflite-w8a8/yolov8_det.tflite $USER@<IP_ADDRESS_OF_TARGET_DEVICE>:~/Downloads/qimsdk_samples/models/

Use the YOLOv8-Detection-Quantized model from AI Hub.Current releases use the following SDK versions:

Qualcomm Linux (GA 2.1): Qualcomm AI Runtime SDK v2.50.0.260828

For example, to export the YOLOv8 QNN model, run:

qai-hub-models export yolov8_det --quantize w8a8 --target-runtime=qnn_context_binary --device="Dragonwing RB3 Gen 2 Vision Kit" --compile-options="--qairt_version 2.50" --profile-options "--qairt_version 2.50"

Update --qairt_version to match the version installed on your target device. To check the installed version:

qairt-net-run --version

To list the QAIRT SDK versions supported by AI Hub for the export command, run this command on the host device:

qai-hub list-frameworks

To list supported devices, run this command on the host device:

qai-hub list-devices

For example, to export the YOLOv8 LiteRT model, run:

qai-hub-models export yolov8_det --quantize w8a8 --target-runtime=tflite --device="Dragonwing RB3 Gen 2 Vision Kit"

To change the batch size, update <N> in the following export command:

qai-hub-models export <Model_Name> --batch-size <N> --device="Dragonwing RB3 Gen 2 Vision Kit"

For example, to export the YOLOv8 LiteRT model with batch size 4:

qai-hub-models export yolov8_det --quantize w8a8 --target-runtime=tflite --device="Dragonwing RB3 Gen 2 Vision Kit" --batch-size 4

AI Vision Applications

Object Detection

The gst-ai-object-detection application allows you to detect objects within images and videos. The use cases show the execution of YOLOv5, YOLOv8 and YOLOX on Qualcomm AI HW accelerator. The following figure shows the pipeline, which receives the input from a live camera feed, file, USB source, or an RTSP stream, preprocesses it, runs inferences on AI hardware. The results are either displayed on the screen, saved as an encoded MP4 file, or streamed over the RTSP server. For information about the plugins used in the pipeline flow, see Pipeline flow

The onnx runtime is not supported for this sample application in the QLI GA 2.1 release.

Application: gst-ai-object-detection

Try me

When the software image includes the qticamsrc plugin, the camera framework uses it by default. If absent, the framework switches to libcamera instead. Since Config #1 lacks support for qticamsrc, the system defaults to libcamera.

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	IMX577 camera	File output	Display	RTSP output
Config #1	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes
Config #2	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model file	Label file
Qualcomm Neural Processing SDK	`yolox.dlc`	`yolox.json`
LiteRT	`yolov8_det_w8a8.tflite` / `yolox_quantized.tflite`	`yolox.json`
Qualcomm AI Engine Direct	`yolov8_det_quantized.bin`	`yolov8.json`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-object-detection --config-file=/etc/configs/config_detection.json

This command runs the application using the sample configuration in /etc/configs/config_detection.json.To view the available command-line options, run:

gst-ai-object-detection -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-object-detection --config-file=/path/to/config_detection.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config_detection.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config_detection.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config_detection.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config_detection.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config_detection.json "$XDG_CONFIG_HOME/qimsdk_samples/config_detection.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-object-detection

Configurations

The gst-ai-object-detection application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<path-to-input-video-file>",
  "ml-framework": "<snpe or tflite or qnn framework>",
  "yolo-model-type": "<yolov8 or yolonas or yolov5 or yolox>",
  "model": "<path-to-model-file>",
  "labels": "<path-to-label-file>",
  "threshold": "<post processing threshold, integer value from 1 to 100>",
  "runtime": "<dsp, gpu, cpu runtime>",
  "output-type": "waylandsink or filesink or rtspsink",
  "snpe-tensors": "<model output tensor name>"
}

The application uses $HOME/Downloads/qimsdk_samples as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

Configuration field	Resolution of a relative path
`file-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`
`output-file`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`

For USB camera input, set the video-format, resolution, and framerate parameters in the config file to match the camera capabilities, see Configure USB camera.

The snpe-tensors field applies only to the SNPE runtime. To retrieve the output tensor names for a DLC model, open the model in Netron.

When using DLC models from the AI Hub, the snpe-tensors field is optional.

Available configurations

Config #2
Config #1

Camera source, LiteRT model, DSP runtime

{
  "camera": 0,
  "ml-framework": "tflite",
  "yolo-model-type": "yolox",
  "model": "yolox_quantized.tflite",
  "labels": "yolox.json",
  "threshold": 40,
  "runtime": "dsp",
  "output-type": "waylandsink"
}

Camera source, LiteRT model, CPU runtime

{
  "ml-framework": "tflite",
  "yolo-model-type": "yolox",
  "model": "yolox_quantized.tflite",
  "labels": "yolox.json",
  "threshold": 40,
  "runtime": "cpu",
  "output-type": "waylandsink"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

Detected objects with bounding boxes and labels are overlaid on the video and displayed on the local display.

Pipeline Flow

The following table lists the plugins used in the object detection pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux`, which demultiplexes the stream. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`v4l2src`	• Captures the live stream from USB camera. • Uses `tee` to split the stream for inferencing.
`h264parse`	Parses the H.264 video bitstream.
`v4l2h264dec`	Hardware-decodes H.264 video to raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimlsnpe` `qtimltflite` `qtimlqnn`	1. After the inference runtime receives the tensor stream on its sink pad, it runs inference using the provided model. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	Handles inference results from any object detection model. 1. Applies a threshold to the chosen number of results. 2. Loads the YOLO (YOLOv5, YOLOv8, or YOLO-NAS) module. 3. Produces video frames with only bounding boxes that can be overlaid on objects. 4. Sends these processed frames to the sink pad of `qtivcomposer`.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.
`filesink`	Receives the video stream on sink pad and saves it as an H.264-encoded MP4 file.
`qtirtspbin`	1. Serves as a network sink. 2. Transmits UDP packets to the network.

Config JSON Field Description

Field	Values / Description
`camera`	Camera selection: - `0` — Primary camera - `1` — Secondary camera
`file-path`	Path to the input media file. The media file should be placed in `$HOME/Downloads/qimsdk_samples/media` or provided using an absolute file path.
`rtsp-ip-port`	RTSP input URL.
`enable-usb-camera`	Enables or disables the USB camera. Supported values: `TRUE` or `FALSE`
`yolo-model-type`	YOLO model version: - `yolov5` - `yolov8` - `yolox` - `yolonas` Default: `yolox`
`ml-framework`	Supported ML frameworks: - `snpe` - `tflite` - `qnn` Default: `tflite`
`model`	Path to the model file. Default model file: `yolox_quantized.tflite` Other supported models: - YOLOv5 SNPE: `yolov5.dlc` - YOLOv8 SNPE: `yolov8.dlc` - YOLO NAS SNPE: `yolonas.dlc` - YOLOX SNPE: `yolox.dlc` - YOLOv5 TFLite: `yolov5.tflite` - YOLOv8 TFLite: `yolov8_det_quantized.tflite` - YOLOX TFLite: `yolox_quantized.tflite` - YOLO NAS TFLite: `yolonas_quantized.tflite` - YOLOv7 TFLite: `Yolo-v7-Quantized.tflite` - YOLOv8 QNN: `yolov8_det_quantized.bin` - YOLOX QNN: `yolox_quantized.bin` Default model directory: `$HOME/Downloads/qimsdk_samples/models`
`labels`	Path to the label file. Default label file: `yolox.json` Other supported label files: - YOLOv5: `yolov5.json` - YOLOv8: `yolov8.json` - YOLOX: `yolox.json` - YOLO NAS: `yolonas.json` - YOLOv7: `yolov7.json` Default labels directory: `$HOME/Downloads/qimsdk_samples/labels`
`threshold`	Detection threshold from `0` to `100`. Default: `40`
`output-type`	Supported output sinks: - `waylandsink` — Display output - `filesink` — MP4 file output - `rtspsink` — RTSP stream output
`output-file`	Output filename or path. Default: `$HOME/Downloads/qimsdk_samples/media/output_object_detection.mp4`
`video-format`	USB camera video format. Supported formats: - `nv12` - `yuy2` - `mjpeg` Applicable only to the USB camera source.
`width`	USB camera video width.
`height`	USB camera video height.
`framerate`	USB camera frame rate.
`runtime`	Supported hardware runtimes: - `cpu` - `gpu` - `dsp` Default: `dsp`
`snpe-tensors`	JSON array containing the output tensors for the SNPE model.
`output-ip-address`	RTSP output server IP address. Default: `127.0.0.1`
`port`	RTSP output server port. Default: `8900`

Image Classification

The gst-ai-classification application is designed to identify the subject in an image. The use cases are implemented using the Qualcomm Neural Processing SDK, LiteRT or Qualcomm AI Engine Direct models. The pipeline receives a video stream from a camera, file source, USB source, or RTSP, preprocesses it, and runs the inference on AI hardware. The results are either displayed on the screen, saved as an encoded MP4 file, or streamed over the RTSP server. For information about the plugins used in the pipeline flow, see Pipeline flow

The onnx runtime is not supported for this sample application in the QLI GA 2.1 release.

Application: gst-ai-classification

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	IMX577 camera	File output	Display	RTSP output
Config #1	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes
Config #2	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes

camera use-case is not supported on IQ-2390 and CQ-2390.

Sample Model and Label Files

Runtime	Model file	Label file
Qualcomm Neural Processing SDK	`inceptionv3.dlc`	`classification.json`
LiteRT	`inception_v3_quantized.tflite`	`classification.json`
Qualcomm AI Engine Direct	`inception_v3_quantized.bin`	`classification.json`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-classification --config-file=/etc/configs/config_classification.json

This command runs the application using the sample configuration in /etc/configs/config_classification.json.To view the available command-line options, run:

gst-ai-classification -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-classification --config-file=/path/to/config_classification.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config_classification.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config_classification.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config_classification.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config_classification.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config_classification.json "$XDG_CONFIG_HOME/qimsdk_samples/config_classification.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-classification

Configurations

The gst-ai-classification application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<path-to-input-video-file>",
  "ml-framework": "<snpe or tflite or qnn framework>",
  "model": "<path-to-model-file>",
  "labels": "<path-to-label-file>",
  "threshold": <post processsing threshold, integer value from 1 to 100>,
  "runtime": "<dsp, gpu, cpu runtime>",
  "output-type": "waylandsink or filesink or rtspsink"
}

Configuration field	Resolution of a relative path
`file-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`
`output-file`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`

For USB camera input, set the video-format, resolution, and framerate parameters in the config file to match the camera capabilities, see Configure USB camera.

Available configurations

Config #2
Config #1

Camera source, LiteRT model, DSP runtime

{
  "camera": 0,
  "ml-framework": "tflite",
  "model": "inception_v3_quantized.tflite",
  "labels": "classification.json",
  "threshold": 40,
  "runtime": "dsp",
  "output-type": "waylandsink"
}

Camera source, LiteRT model, CPU runtime

{
  "ml-framework": "tflite",
  "model": "inception_v3_quantized.tflite",
  "labels": "classification.json",
  "threshold": 40,
  "runtime": "cpu",
  "output-type": "waylandsink"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The classified object label and confidence score are overlaid on the video and displayed on the local display

Pipeline Flow

The following table lists the plugins used in the classification pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux`, which demultiplexes the stream. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`v4l2src`	• Captures the live stream from USB camera. • Uses `tee` to split the stream for inferencing.
`h264parse`	Parses the H.264 video bitstream.
`v4l2h264dec`	Hardware-decodes H.264 video to raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream on its source pad. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimlsnpe` `qtimltflite` `qtimlqnn`	1. After the inference runtime receives the tensor stream on its sink pad, it runs inference using the provided model. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	Handles inference results from any classification model. 1. Applies a threshold to the chosen number of results. 2. Loads the MobileNet-softmax postprocessing module. 3. Produces results as video frames with classification labels. 4. Sends these processed frames to the sink pad of `qtivcomposer`.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.
`filesink`	Receives the video stream on sink pad and saves it as an H.264-encoded MP4 file.
`qtirtspbin`	1. Serves as a network sink. 2. Transmits UDP packets to the network.

Config JSON Field Description

Field	Values / Description
`ml-framework`	Supported ML frameworks: - `snpe` — Qualcomm Neural Processing SDK - `tflite` — LiteRT - `qnn` — Qualcomm AI Engine Direct Default: `snpe`
`runtime`	Supported hardware runtimes: - `cpu` - `gpu` - `dsp` Default: `dsp`
`file-path`	Path to the input media file. Default media directory: `$HOME/Downloads/qimsdk_samples/media`
`rtsp-ip-port`	RTSP input URL.
`camera`	Camera selection: - `0` — Primary camera - `1` — Secondary camera
`enable-usb-camera`	Enables or disables the USB camera. Supported values: `TRUE` or `FALSE`
`model`	Path to the model file. Default model file: `inception_v3_quantized.tflite` Other supported models: - SNPE: `inceptionv3.dlc` - TFLite: `inception_v3_quantized.tflite` - QNN: `inception_v3_quantized.bin` Default model directory: `$HOME/Downloads/qimsdk_samples/models`
`labels`	Path to the label file. Default: `classification.json` Default labels directory: `$HOME/Downloads/qimsdk_samples/labels`
`output-type`	Supported output sinks: - `waylandsink` — Display output - `filesink` — MP4 file output - `rtspsink` — RTSP stream output
`output-file`	Output filename or path. Default: `$HOME/Downloads/qimsdk_samples/media/output_classification.mp4`
`output-ip-address`	RTSP output server IP address. Default: `127.0.0.1`
`port`	RTSP output server port. Default: `8900`
`video-format`	USB camera video format. Supported formats: - `nv12` - `yuy2` - `mjpeg`
`width`	USB camera video width.
`height`	USB camera video height.
`framerate`	USB camera frame rate.
`threshold`	Detection threshold from `0` to `100`. Default: `40`

Face Detection

The gst-ai-face-detection application collects the live video input from a camera, file, or an RTSP stream and uses the Qualcomm AI Engine direct and LiteRT face detection models to produce a preview with the overlaid AI model output on the HDMI display. The following figure shows the pipeline, which receives the input, preprocesses it, runs inferences on AI hardware, and displays the results on the screen. For information about the plugins used in the pipeline flow, see Pipeline flow.

Application: gst-ai-face-detection

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	No	No	No	Yes	No
Config #2	Yes	Yes	No	Yes	No	Yes	No

camera use-case is not supported on IQ-2390 and CQ-2390.

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	`face_det_lite_quantized.tflite`	`face_detection.json`
Qualcomm AI Engine Direct	`face_det_lite_quantized.bin`	`face_detection.json`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-face-detection --config-file=/etc/configs/config_face_detection.json

This command runs the application using the sample configuration in /etc/configs/config_face_detection.json.To view the available command-line options, run:

gst-ai-face-detection -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-face-detection --config-file=/path/to/config_face_detection.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config_face_detection.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config_face_detection.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config_face_detection.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config_face_detection.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config_face_detection.json "$XDG_CONFIG_HOME/qimsdk_samples/config_face_detection.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-face-detection

Configurations

The gst-ai-face-detection application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<input-video-path>",
  "ml-framework": "<tflite or qnn framework>",
  "model": "<path-to-model-file",
  "labels": "<path-to-label-file",
  "threshold": <post-processing threshold, integer value from 1-100>,
  "runtime": "<cpu, gpu or dsp runtime>"
}

Configuration field	Resolution of a relative path
`file-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
  "file-path": "video.mp4",
  "ml-framework": "tflite",
  "model":"face_det_lite_quantized.tflite",
  "labels": "face_detection.json",
  "threshold": 51,
  "runtime": "dsp"
}

File source, LiteRT model, CPU runtime

{
  "file-path": "video.mp4",
  "ml-framework": "tflite",
  "model":"face_det_lite_quantized.tflite",
  "labels": "face_detection.json",
  "threshold": 51,
  "runtime": "cpu"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

Pipeline Flow

The following table lists the plugins used in the face detection pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`h264parse`	• Parses the H.264 video bitstream.
`v4l2h264dec`	• Hardware-decodes H.264 video to raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite` `qtimlqnn`	1. After the inference runtime receives the tensor stream on its sink pad, it runs inference using the provided model. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	1. Handles inference results from any face detection model. 2. Applies a threshold to the chosen number of results.
`qtimetamux`	1. Receives string-based postprocessing output text with video frame and multiplexes it.
`qtivoverlay`	1. Receives the multiplexed stream. 2. Overlays the bounding boxes on the stream.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values/descriptions
`camera`	Select `0` for the primary camera or `1` for the secondary camera.
`file-path`	File source path. Default media directory: `$HOME/Downloads/qimsdk_samples/media`
`rtsp-ip-port`	RTSP input URL.
`ml-framework`	Supported ML frameworks: - `tflite` (LiteRT) - `qnn` (Qualcomm AI Engine Direct)
`model`	Optional model path that overrides the default model path. Default model: `face_det_lite_quantized.tflite` Other supported models: - TFLite default: `face_det_lite_quantized.tflite` - QNN default: `face_det_lite_quantized.bin` Default models directory: `$HOME/Downloads/qimsdk_samples/models`
`labels`	Optional labels path that overrides the default labels path. Default: `face_detection.json` Default labels directory: `$HOME/Downloads/qimsdk_samples/labels`
`threshold`	Detection threshold from `0` to `100`. Default: `51`
`runtime`	Supported hardware runtimes: - `cpu` - `gpu` - `dsp` Default: `dsp`

Semantic Segmentation

The gst-ai-segmentation application allows you to divide an image into different and meaningful parts or segments and assign a label to each homogeneous segment based on the similarity of the attributes. The application uses Qualcomm Neural Processing SDK runtime, Qualcomm AI Engine direct runtime, and LiteRT for image segmentation. The following figure shows the pipeline, which receives the input from a live camera feed, file, or an RTSP stream, preprocesses the video data, runs inferences using AI hardware, and displays the segmented data on the screen. For information about the plugins used in the pipeline flow, see Pipeline flow.

Application: gst-ai-segmentation

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	No	No	No	Yes	No
Config #2	Yes	Yes	No	Yes	No	Yes	No

Sample Model and Label Files

Runtime	Model file	Label file
Qualcomm Neural Processing SDK	`deeplabv3_plus_mobilenet.dlc`	`deeplabv3_resnet50.json`
LiteRT	`deeplabv3_plus_mobilenet_quantized.tflite`	`deeplabv3_resnet50.json`
Qualcomm AI Engine Direct	`deeplabv3_plus_mobilenet_quantized.bin`	`deeplabv3_resnet50.json`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-segmentation --config-file=/etc/configs/config_segmentation.json

This command runs the application using the sample configuration in /etc/configs/config_segmentation.json.To view the available command-line options, run:

gst-ai-segmentation -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-segmentation --config-file=/path/to/config_segmentation.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config_segmentation.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config_segmentation.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config_segmentation.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config_segmentation.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config_segmentation.json "$XDG_CONFIG_HOME/qimsdk_samples/config_segmentation.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-segmentation

Configurations

The gst-ai-segmentation application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<input-video-path>",
  "ml-framework": "<snpe, tflite, or qnn framework>",
  "model": "<path-to-model-file>",
  "labels": "<path-to-label-file>",
  "runtime": "<dsp, gpu, or cpu runtime>"
}

Configuration field	Resolution of a relative path
`file-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
 "file-path": "video.mp4",
 "ml-framework": "tflite",
 "model": "deeplabv3_plus_mobilenet_quantized.tflite",
 "labels": "deeplabv3_resnet50.json",
 "runtime": "dsp"
}

File source, LiteRT model, CPU runtime

{
  "file-path": "video.mp4",
  "ml-framework": "tflite",
  "model": "deeplabv3_plus_mobilenet_quantized.tflite",
  "labels": "deeplabv3_resnet50.json",
  "runtime": "cpu"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The segmented data is displayed on the local display.

Pipeline Flow

The following table lists the plugins used in the segmentation pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for concurrent display and ML inference.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux`, which demultiplexes the stream. • Uses `tee` to split the stream for processing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for processing.
`h264parse`	• Parses the H.264 video bitstream to ensure downstream elements can handle the payload.
`v4l2h264dec`	• Hardware-accelerated decoder that converts H.264 video into raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data. This preprocessing is done when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream on its source pad. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimlsnpe` `qtimltflite` `qtimlqnn`	1. After the inference runtime receives the tensor stream on its sink pad, it runs inference using the provided model. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	1. Converts the inference tensors received on its sink pad into video formats that multimedia plugins use for further processing.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Descriptions
`ml-framework`	Supported ML frameworks: - `tflite` — LiteRT - `qnn` — Qualcomm AI Engine Direct Default: `tflite`
`runtime`	Supported hardware runtimes: - `cpu` - `gpu` - `dsp` Default: `dsp`
`Input source`	Supported input sources: - `file-path` - `rtsp-ip-port` - `camera` Default input source: `camera`
`file-path`	Path to the input media file. Default: `$HOME/Downloads/qimsdk_samples/media`
`rtsp-ip-port`	RTSP input URL.
`camera`	Camera selection: - `0` — Primary camera - `1` — Secondary camera
`model`	Optional model path that overrides the default model path. Default model: `deeplabv3_plus_mobilenet_quantized.tflite` Default model directory: `$HOME/Downloads/qimsdk_samples/models`
`labels`	Optional labels path that overrides the default labels path. Default labels file: `deeplabv3_resnet50.json` Default labels directory: `$HOME/Downloads/qimsdk_samples/labels`

Pose Detection

The gst-ai-pose-detection application allows you to detect the body pose of the subject in an image or video. The use case processes input streams from a camera, file, or an RTSP source and uses LiteRT and Qualcomm AI Engine direct models for pose detection. The results are either displayed on the screen, saved as an encoded MP4 file, or streamed over the RTSP server. The following figure shows the pipeline, which receives the input from a live camera feed, file, USB source, or an RTSP stream, preprocesses it, conducts inference on AI hardware, and generates the output. This process allows for real-time pose detection and visualization of human poses. For information about the plugins used in the pipeline flow, see Pipeline flow.

Application: gst-ai-pose-detection

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	Yes	No	Yes	Yes	Yes
Config #2	Yes	Yes	Yes	Yes	Yes	Yes	Yes

camera use-case is not supported on IQ-2390 and CQ-2390.

Sample Model and Label Files

Runtime	Model file	Label files
LiteRT	`hrnet_pose_quantized.tflite`	`hrnet_pose.json`, `hrnet_settings.json`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-pose-detection --config-file=/etc/configs/config_pose.json

This command runs the application using the sample configuration in /etc/configs/config_pose.json.To view the available command-line options, run:

gst-ai-pose-detection -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-pose-detection --config-file=/path/to/config_pose.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config_pose.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config_pose.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config_pose.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config_pose.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config_pose.json "$XDG_CONFIG_HOME/qimsdk_samples/config_pose.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-pose-detection

Configurations

The gst-ai-pose-detection application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<input-video-path>",
  "ml-framework": "<tflite or qnn framework>",
  "model": "<path-to-model-file>",
  "labels": "<path-to-label-file>",
  "pose-settings-path": "<path-to-hrnet_settings-file>",
  "output-type": "waylandsink or filesink or rtspsink",
  "runtime": "<cpu, gpu or dsp runtime>"
}

Configuration field	Resolution of a relative path
`file-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`
`output-file`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`

For USB camera input, set the video-format, resolution, and framerate parameters in the config file to match the camera capabilities, see Configure USB camera.

To change the threshold, you must configure the confidence value in the hrnet_settings.json file.

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
  "file-path": "video.mp4",
  "ml-framework": "tflite",
  "model": "hrnet_pose_quantized.tflite",
  "labels": "hrnet_pose.json",
  "pose-settings-path":"hrnet_settings.json",
  "runtime": "dsp",
  "output-type": "waylandsink"
}

File source, LiteRT model, CPU runtime

{
  "file-path": "video.mp4",
  "ml-framework": "tflite",
  "model": "hrnet_pose_quantized.tflite",
  "labels": "hrnet_pose.json",
  "pose-settings-path":"hrnet_settings.json",
  "runtime": "cpu",
  "output-type": "waylandsink"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The displayed output shows the detected pose of the objects.

Pipeline Flow

The following table lists the plugins used in the pose detection pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for concurrent display and ML inference.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for processing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for processing.
`v4l2src`	• Captures the live stream from USB camera. • Uses `tee` to split the stream for processing.
`h264parse`	• Parses the H.264 video bitstream to ensure downstream elements can handle the payload.
`v4l2h264dec`	• Hardware-accelerated decoder that converts H.264 video into raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite` `qtimlqnn`	1. Uses the HRNet model for pose detection. 2. The application runs on the external delegate to execute the model using the Hexagon Tensor Processor. 3. After the inference runtime receives the tensor stream on its sink pad, it does the following: • Runs the inference. • Produces a tensor stream containing the inference results on its source pad. • Manages the inference results from the pose detection model.
`qtimlpostprocess`	1. Applies a threshold to the chosen number of results. 2. Loads corresponding modules for various pose detection models. 3. In this specific use case, `qtimlpostprocess` does the following: • Loads the HRNet module. • Produces results in the form of video frames with drawn poses. • Sends the results to the sink pad of `qtivcomposer` for further processing or display.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.
`filesink`	1. Receives the video stream on its sink pad and saves it as an H.264-encoded MP4 file.
`qtirtspbin`	1. Serves as a network sink. 2. Transmits UDP packets to the network.

Config JSON Field Description

Field	Values / Description
`ml-framework`	Supported ML frameworks: - `tflite` — LiteRT - `qnn` — Qualcomm AI Engine Direct Default: `tflite`
`runtime`	Supported hardware runtimes: - `cpu` - `gpu` - `dsp` Default: `dsp`
`file-path`	Path to the input media file. Default media directory: `$HOME/Downloads/qimsdk_samples/media`
`rtsp-ip-port`	RTSP input URL.
`camera`	Camera selection: - `0` — Primary camera - `1` — Secondary camera
`enable-usb-camera`	Enables or disables the USB camera. Supported values: `TRUE` or `FALSE`
`model`	Path to the model file. Default model file: `hrnet_pose_quantized.tflite` Other supported models: - TFLite: `hrnet_pose_quantized.tflite` - QNN: `hrnet_pose_quantized.bin` Default model directory: `$HOME/Downloads/qimsdk_samples/models`
`labels`	Path to the labels file. Default: `hrnet_pose.json` Default labels directory: `$HOME/Downloads/qimsdk_samples/labels`
`pose-settings-path`	Path to the pose settings file. Default: `hrnet_settings.json` Default settings directory: `$HOME/Downloads/qimsdk_samples/labels`
`output-type`	Supported output sinks: - `waylandsink` — Display output - `filesink` — MP4 file output - `rtspsink` — RTSP stream output
`output-file`	Output filename or path. Default: `$HOME/Downloads/qimsdk_samples/media/output_pose.mp4`
`output-ip-address`	RTSP output server IP address. Default: `127.0.0.1`
`port`	RTSP output server port. Default: `8900`
`video-format`	USB camera video format. Supported formats: - `nv12` - `yuy2` - `mjpeg` Applicable only to the USB camera source.
`width`	USB camera video width.
`height`	USB camera video height.
`framerate`	USB camera frame rate.

For better accuracy and detection results, use the gst-ai-daisychain-detection-pose application.

Mono Depth

The gst-ai-monodepth application allows you to infer depth of a source feed from a live camera stream, file, or an RTSP stream. The following figure shows the pipeline, which captures feed from the source, preprocesses it, and runs inferences using the AI hardware. For information about the plugins used in the pipeline, see Pipeline flow.

Application: gst-ai-monodepth

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	No	No	No	Yes	No
Config #2	Yes	Yes	No	Yes	No	Yes	No

Sample Model and Label Files

Runtime	Model file	Label file
Qualcomm Neural Processing SDK	`midasv2.dlc`	`monodepth.json`
LiteRT	`midas_quantized.tflite`	`monodepth.json`
Qualcomm AI Engine Direct	`midas_quantized.bin`	`monodepth.json`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-monodepth --config-file=/etc/configs/config_monodepth.json

This command runs the application using the sample configuration in /etc/configs/config_monodepth.json.To view the available command-line options, run:

gst-ai-monodepth -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-monodepth --config-file=/path/to/config_monodepth.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config_monodepth.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config_monodepth.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config_monodepth.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config_monodepth.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config_monodepth.json "$XDG_CONFIG_HOME/qimsdk_samples/config_monodepth.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-monodepth

Configurations

The gst-ai-monodepth application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<input video path>",
  "ml-framework": "<snpe, tflite, or qnn framework>",
  "model": "<path-to-model-file>",
  "labels": "<path-to-label-file>",
  "runtime": "<dsp, gpu, or cpu runtime>"
}

Configuration field	Resolution of a relative path
`file-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
  "file-path": "video.mp4",
  "ml-framework": "tflite",
  "model": "midas_quantized.tflite",
  "labels": "monodepth.json",
  "runtime": "dsp"
}

File source, LiteRT model, CPU runtime

{
  "file-path": "video.mp4",
  "ml-framework": "tflite",
  "model": "midas_quantized.tflite",
  "labels": "monodepth.json",
  "runtime": "cpu"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The overlaid model output stream is shown side by side with the live feed.

Pipeline Flow

The following table lists the plugins used in the monodepth pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for concurrent display and ML inference.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for processing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for processing.
`h264parse`	• Parses the H.264 video bitstream to ensure downstream elements can handle the payload.
`v4l2h264dec`	• Hardware-accelerated decoder that converts H.264 video into raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data. This preprocessing is done when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream on its source pad. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimlsnpe` `qtimltflite` `qtimlqnn`	Uses the Midasv2 model for monodepth calculation. 1. The inference runtime receives the tensor stream on its sink pad. 2. The runtime executes the inference. 3. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	1. Converts the inference tensors received on its sink pad into video formats that multimedia plugins use for further processing.
`qtivtransform`	1. Converts the buffers on its source pad to formats compatible with composition on `waylandsink`.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Descriptions
`ml-framework`	Supported ML frameworks: - `snpe` — Qualcomm Neural Processing SDK - `tflite` — LiteRT - `qnn` — Qualcomm AI Engine Direct Default: `snpe`
`runtime`	Supported hardware runtimes: - `cpu` - `gpu` - `dsp` Default: `dsp`
`file-path`	Path to the input media file. Default media directory: `$HOME/Downloads/qimsdk_samples/media`
`rtsp-ip-port`	RTSP input URL.
`camera`	Camera selection: - `0` — Primary camera - `1` — Secondary camera
`model`	Path to the model file. Default model file: `midas_quantized.tflite` Other supported models: - SNPE DLC: `midasv2.dlc` - TFLite: `midas_quantized.tflite` - QNN: `midas_quantized.bin` Default model directory: `$HOME/Downloads/qimsdk_samples/models`
`labels`	Path to the label file. Default: `monodepth.json` Default labels directory: `$HOME/Downloads/qimsdk_samples/labels`

Super Resolution

The gst-ai-superresolution application allows you to generate high resolution video frames from low-resolution input. The following figures shows the pipeline, which receives a video stream from a file source as input, processes it through the super resolution module using LiteRT, and displays the output.

This application isn’t supported in Config #1 of Qualcomm Linux because CPU runtime is not supported.

For information about the plugins used in the pipeline, see Pipeline flow.

Application: gst-ai-superresolution

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #2	Yes	No	No	No	Yes	Yes	No

Sample Model Files

Runtime	Model file
LiteRT	`quicksrnetsmall_quantized.tflite`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-superresolution --config-file=/etc/configs/config-superresolution.json

This command runs the application using the sample configuration in /etc/configs/config-superresolution.json.To view the available command-line options, run:

gst-ai-superresolution -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-superresolution --config-file=/path/to/config-superresolution.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config-superresolution.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config-superresolution.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config-superresolution.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config-superresolution.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config-superresolution.json "$XDG_CONFIG_HOME/qimsdk_samples/config-superresolution.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-superresolution

Configurations

The gst-ai-superresolution application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "input-file-path": "<input-video-path>",
  "model": "<path-to-model-file>",
  "output-file-path": "<path-to-output-video>"
}

Configuration field	Resolution of a relative path
`input-file-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`output-file-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`

The video super‑resolution application requires an input video resolution of 128 × 128.

Available configurations

Config #2

File source, LiteRT model, DSP runtime

{
  "input-file-path": "video.mp4",
  "model": "quicksrnetsmall_quantized.tflite"
}

Expected Output

The output is displayed on an HDMI monitor.

Pipeline Flow

The following table lists the plugins used in the superresolution pipeline:

Plugin	Description
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for processing.
`h264parse`	• Parses the H.264 video bitstream to ensure downstream elements can handle the payload.
`v4l2h264dec`	• Hardware-accelerated decoder that converts H.264 video into raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data. This preprocessing is done when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream on its source pad. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	• Runs on LiteRT and uses the `quicksrnetsmall_quantized` model for super resolution. 1. The inference runtime receives the tensor stream on its sink pad. 2. The runtime executes the inference. 3. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	• Handles inference results from any super resolution model. 1. Loads the SRNet module. 2. Produces results as high-resolution video frames. 3. Sends the processed frames to the sink pad of `qtivcomposer`.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Description
`input-file-path`	Path to the input media file. Default: `video.mp4` Default media directory: `$HOME/Downloads/qimsdk_samples/media` Alternatively, provide an absolute file path.
`model`	Path to the super-resolution model file. Default: `quicksrnetsmall_quantized.tflite` Default model directory: `$HOME/Downloads/qimsdk_samples/models` Alternatively, provide an absolute file path.
`output-file-path`	Output file path. For a relative path, the output is saved under `$HOME/Downloads/qimsdk_samples/media`. Alternatively, provide an absolute file path. If not specified, display output is selected.

AI Event Encoder

The gst-ai-event-encoder application receives the live video stream input from camera, file, or RTSP source. When a human enters the video frame the application preprocesses the video, runs inferences on the AI hardware, and encodes the video. The encoding stops 5 seconds after the human moves away from the frame and restarts when anyone enters the frame. The following figures show the event detection and recording pipelines for event encoder application. For information about the plugins used in the pipeline flow, see Pipeline flow.

Application: gst-ai-event-encoder

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	No	No	No	Yes	No
Config #2	Yes	Yes	No	Yes	No	Yes	No

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	`yolox_quantized.tflite`	`yolox.json`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-event-encoder --config-file=/etc/configs/config-event-encoder.json

This command runs the application using the sample configuration in /etc/configs/config-event-encoder.json.To view the available command-line options, run:

gst-ai-event-encoder -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-event-encoder --config-file=/path/to/config-event-encoder.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config-event-encoder.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config-event-encoder.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config-event-encoder.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config-event-encoder.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config-event-encoder.json "$XDG_CONFIG_HOME/qimsdk_samples/config-event-encoder.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-event-encoder

Configurations

The gst-ai-event-encoder application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<path to video file>",
  "model": "<path to model file>",
  "labels": "<path to label file>",
  "threshold": <integer between 1 and 100>,
  "runtime": "<cpu, gpu, or dsp runtime>"
}

Configuration field	Resolution of a relative path
`file-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
  "file-path": "video.mp4",
  "model": "yolox_quantized.tflite",
  "labels": "yolox.json",
  "threshold": 40,
  "runtime": "dsp"
}

File source, LiteRT model, CPU runtime

{
  "file-path": "video.mp4",
  "model": "yolox_quantized.tflite",
  "labels": "yolox.json",
  "threshold": 40,
  "runtime": "cpu"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The output is saved as an MP4 file within ${HOME}/Downloads/qimsdk_samples/media folder as output-1.mp4, output-2.mp4, and so on.

Pipeline Flow

The following table lists the plugins used in the event encoder pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for concurrent display and ML inference.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for processing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for processing.
`h264parse`	• Parses the H.264 video bitstream to ensure downstream elements can handle the payload.
`v4l2h264dec`	• Hardware-accelerated decoder that converts H.264 video into raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	1. After the inference runtime receives the tensor stream on its sink pad, it executes the inference. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	Instance 1 (Detection Overlay): • Applies a threshold to the chosen number of results. • Loads the YOLOv8 module. • Produces video frames with only bounding boxes for object overlay. • Sends processed frames to the sink pad of `qtivcomposer`. Instance 2 (Metadata Generation): • Produces output in text format (bounding box coordinates and labels). • Connects to an `appsink` plugin where metadata is read, parsed, and logged. • Uses bounding box information to count the number of humans in each frame.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Descriptions
`runtime`	Supported hardware runtimes: - `cpu` - `gpu` - `dsp` Default: `dsp`
`file-path`	Path to the input media file. Default media directory: `$HOME/Downloads/qimsdk_samples/media` Alternatively, provide an absolute file path.
`rtsp-ip-port`	RTSP input URL.
`camera`	Camera selection: - `0` — Primary camera - `1` — Secondary camera
`model`	Optional model path that overrides the default model path. Default: `yolox_quantized.tflite` Default model directory: `$HOME/Downloads/qimsdk_samples/models` Alternatively, provide an absolute file path.
`labels`	Optional labels path that overrides the default labels path. Default: `yolox.json` Default labels directory: `$HOME/Downloads/qimsdk_samples/labels` Alternatively, provide an absolute file path.
`threshold`	Detection threshold from `0` to `100`. Default: `40`

Metadata Parser

The gst-ai-metadata-parser-example application receives the live video stream input from camera, file, or RTSP source, and passes the stream to the YOLO models for object detection and preview. The overlaid AI model output, including labels and bounding boxes, is displayed on an HDMI display. The extracted metadata is logged to the console and used to count the number of humans in the frame. The following figure shows the pipeline for metadata parsing. For information about the plugins used in the pipeline flow, see Pipeline flow.

Application: gst-ai-metadata-parser-example

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	No	No	No	Yes	No
Config #2	Yes	Yes	No	Yes	No	Yes	No

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	`yolox_quantized.tflite`	`yolox.json`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-metadata-parser-example --config-file=/etc/configs/config-metadata-parser.json

This command runs the application using the sample configuration in /etc/configs/config-metadata-parser.json.To view the available command-line options, run:

gst-ai-metadata-parser-example -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-metadata-parser-example --config-file=/path/to/config-metadata-parser.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config-metadata-parser.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config-metadata-parser.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config-metadata-parser.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config-metadata-parser.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config-metadata-parser.json "$XDG_CONFIG_HOME/qimsdk_samples/config-metadata-parser.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-metadata-parser-example

Configurations

The gst-ai-metadata-parser-example application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<path to video file>",
  "model": "<path to model file>",
  "labels": "<path to label file>",
  "threshold": <integer between 1 and 100>,
  "runtime": "<cpu, gpu, or dsp runtime>"
}

Configuration field	Resolution of a relative path
`file-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
  "file-path": "video.mp4",
  "model": "yolox_quantized.tflite",
  "labels": "yolox.json",
  "threshold": 40,
  "runtime": "dsp"
}

File source, LiteRT model, CPU runtime

{
  "file-path": "video.mp4",
  "model": "yolox_quantized.tflite",
  "labels": "yolox.json",
  "threshold": 40,
  "runtime": "cpu"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

Pipeline Flow

The following table lists the plugins used in the metadata parser pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream into two for inferencing and composing.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream into two for inferencing and composing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream into two for inferencing and composing.
`h264parse`	• Parses the H.264 video.
`v4l2h264dec`	• Decodes the video.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream on its source pad. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	• After the inference runtime receives the tensor stream on its sink pad, it runs the inference. • Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	Instance 1 (Object Detection): • Applies a threshold to the chosen number of results. • Loads the YOLOv8 module. • Produces video frames with only bounding boxes that can be overlaid on objects. • Sends these processed frames to the sink pad of `qtivcomposer`. Instance 2 (Human Counting): • Produces the output in a text format (bounding box coordinates and labels). • This output is connected to `appsink` plugin where the metadata is read, parsed, and logged. • The bounding box information is used to count the number of humans in each frame.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. `waylandsink` submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Descriptions
`runtime`	Supported hardware runtimes: - `cpu` - `gpu` - `dsp` Default: `dsp`
`file-path`	Path to the input media file. Default media directory: `$HOME/Downloads/qimsdk_samples/media` Alternatively, provide an absolute file path.
`rtsp-ip-port`	RTSP input URL.
`camera`	Camera selection: - `0` — Primary camera - `1` — Secondary camera
`model`	Path to the model file. Default: `yolox_quantized.tflite` Default model directory: `$HOME/Downloads/qimsdk_samples/models` Alternatively, provide an absolute file path.
`labels`	Path to the label file. Default: `yolox.json` Default labels directory: `$HOME/Downloads/qimsdk_samples/labels` Alternatively, provide an absolute file path.
`threshold`	Detection threshold from `0` to `100`. Default: `40`

AI USB Camera

The gst-ai-usb-camera-app streams video from a USB webcam connected to the Qualcomm EVK. This webcam should be accessible as a /dev/videoX device. Additionally, you can perform object detection and preview the results. You can choose to preview the output on Wayland, or encode to a video file, or live stream through the RTSP. Alternatively, you can set enable-object-detection as True to perform object detection. The following figures show a pipeline, which processes the input from the USB camera to generate various outputs. For information about the plugins used in this pipeline, see Pipeline flow.

Application: gst-ai-usb-camera-app

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	No	No	Yes	No	Yes	Yes	Yes
Config #2	No	No	Yes	No	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model Files	Label Files
`Qualcomm Neural Processing SDK`	`yolonas.dlc` `yolov5.dlc` `yolov8.dlc`	`yolonas.json` `yolov5.json` `yolov8.json`
`LiteRT`	`yolov8_det_w8a8.tflite` `yolonas_quantized.tflite` `yolov5.tflite` `yolox_quantized.tflite`	`yolov8.json` `yolonas.json` `yolov5.json` `yolox.json`
`Qualcomm AI Engine direct`	`yolov8_det_quantized.bin`	`yolov8.json`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-usb-camera-app --config-file=/etc/configs/config-usb-camera-app.json

This command runs the application using the sample configuration in /etc/configs/config-usb-camera-app.json.To view the available command-line options, run:

gst-ai-usb-camera-app -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-usb-camera-app --config-file=/path/to/config-usb-camera-app.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config-usb-camera-app.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config-usb-camera-app.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config-usb-camera-app.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config-usb-camera-app.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config-usb-camera-app.json "$XDG_CONFIG_HOME/qimsdk_samples/config-usb-camera-app.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-usb-camera-app

Configurations

The gst-ai-usb-camera-app application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "width": "<Supported USB camera width>",
  "height": "<Supported USB camera Height>",
  "framerate": "<Supported USB camera fps>",
  "video-format": "<Supported USB camera video format like yuy2, mjpeg, nv12>",
  "output": "<output type like waylandsink, filesink or rtspsink>",
  "ip-address": "<Device IP address in case of rtsp streaming>",
  "port": "<Device port num in case of rtsp streaming>",
  "enable-object-detection": "<to enable the object-detection>",
  "ml-framework": "<snpe, or tflite or qnn framework>",
  "yolo-model-type": "<yolov8 or yolonas or yolov5 or yolox>",
  "model": "<Model Path>",
  "labels": "<Label Path>",
  "threshold": "<Post-processing threshold, integer value from 1-100>",
  "runtime": "<dsp, cpu or gpu runtime>",
  "snpe-tensors": "<Model output tensor name>"
}

Configuration field	Resolution of a relative path
`model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`
`output-file`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`

For USB camera input, set the video-format, resolution, and framerate parameters in the config file to match the camera capabilities.

You can run the Yolo-NAS-Quantized.tflite model using the YOLOv8 value in the yolo-model-type field.

The snpe-tensors field applies only to the SNPE runtime. To retrieve the output tensor names for a DLC model, open the model in Netron.

When using DLC models from the AI Hub, the snpe-tensors field is optional.

If the USB camera isn’t detected on the target device, Download the required firmware. See Download PCIe to USB controller firmware.

Available configurations

Config #2
Config #1

RTSP input, LiteRT, YOLOx model and DSP runtime

{
  "width": 1920,
  "height": 1080,
  "framerate": 30,
  "output":"waylandsink",
  "video-format":"yuy2",
  "model":"yolox_quantized.tflite",
  "labels":"yolox.json",
  "output-file":"output.mp4",
  "ip-address":"127.0.0.1",
  "port":"8900",
  "enable-object-detection": "TRUE",
  "ml-framework": "tflite",
  "yolo-model-type": "yolox",
  "threshold": 75,
  "runtime": "dsp"
}

RTSP input, LiteRT, YOLOx model and CPU runtime

{
  "width": 1920,
  "height": 1080,
  "framerate": 30,
  "output":"waylandsink",
  "video-format":"yuy2",
  "model":"yolox_quantized.tflite",
  "labels":"yolox.json",
  "output-file":"output.mp4",
  "ip-address":"127.0.0.1",
  "port":"8900",
  "enable-object-detection": "TRUE",
  "ml-framework": "tflite",
  "yolo-model-type": "yolox",
  "threshold": 75,
  "runtime": "cpu"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

Pipeline flow

The following table lists the plugins used in AI USB camera pipelines:

Pipeline	Description
Dump the USB camera to a filesink	• USB camera captures the live camera stream. • `qtivtransform` transforms the stream data. • `capsfilter` is applied to enforce constraints on the raw video data. • `filesink` is used to dump the data into a file.
Video encoding	• USB camera captures the live camera stream. • `qtivtransform` transforms the stream data. • `capsfilter` is applied to enforce constraints on the raw video data. • `v4l2h264enc` is used to encode the video using the H.264 format. • `h264parse` is used to parse the video. • `mp4mux` is used to multiplex the video into an MP4 container. • `filesink` is used to write the video to a file.
RTSP streaming	• USB camera captures the live camera stream. • `qtivtransform` transforms the stream data. • `capsfilter` is applied to enforce constraints on the raw video data. • `v4l2h264enc` is used to encode the video using the H.264 format. • `h264parse` is used to parse the video. • `qtirtspbin` is used to load the stream to RTSP.
USB camera and object detection on RTSP	• USB camera captures the live camera stream. • `capsfilter` is applied to enforce constraints on the raw video data. • `tee` is used to split the stream for inferencing. • `qtivtransform` transforms the stream data. • `qtimlvconverter` performs preprocessing and converts the video stream to a tensor stream, which is used for inferencing. • `qtimlsnpe`, `qtimltflite`, or `qtimlqnn` run the inference on the stream. • `qtimlpostprocess` handles the inference results from any object detection model and produces video frames. • `qtivcomposer` composes the video frames and shares them with `qtirtspbin`. • `qtirtspbin` submits the composed video stream to Weston, which renders it on the local display.
USB camera and object detection on wayland	• USB camera captures the live camera stream. • `capsfilter` is applied to enforce constraints on the raw video data. • `tee` is used to split the stream for inferencing. • `qtivtransform` transforms the stream data. • `qtimlvconverter` performs preprocessing and converts the video stream to a tensor stream, which is used for inferencing. • `qtimlsnpe`, `qtimltflite`, or `qtimlqnn` run the inference on the stream. • `qtimlpostprocess` handles the inference results from any object detection model and produces video frames. • `qtivcomposer` composes the video frames and shares them with `waylandsink`. • `waylandsink` submits the composed video stream to Weston, which renders it on the local display.
Object detection using USB camera and file encode	• USB camera captures the live camera stream. • `capsfilter` is applied to enforce constraints on the raw video data. • `tee` is used to split the stream for inferencing. • `qtivtransform` transforms the stream data. • `qtimlvconverter` performs preprocessing and converts the video stream to a tensor stream, which is used for inferencing. • `qtimlsnpe`, `qtimltflite`, or `qtimlqnn` run the inference on the stream. • `qtimlpostprocess` handles the inference results from any object detection model and produces video frames. • `qtivcomposer` composes the video frames and shares them with `filesink`. • `filesink` writes the composed video stream to a file.

Config JSON field description

Field	Values / Descriptions
`width`	USB camera resolution width.
`height`	USB camera resolution height.
`framerate`	USB camera frame rate.
`video-format`	USB camera video format. Supported formats: - `nv12` - `yuy2` - `mjpeg`
`output`	Supported output sinks: - `waylandsink` — Display output - `filesink` — MP4 file output - `rtspsink` — RTSP stream output
`output-file`	Output filename or path. Default: `$HOME/Downloads/qimsdk_samples/media/output.mp4` Alternatively, provide an absolute file path.
`ip-address`	RTSP output server IP address. Default: `127.0.0.1`
`port`	RTSP output server port. Default: `8900`
`enable-object-detection`	Enables or disables object detection. Supported values: `TRUE` or `FALSE`
`yolo-model-type`	YOLO model version: - `yolov5` - `yolov8` - `yolox` - `yolonas` YOLOv7 TFLite models use `yolov8` as the `yolo-model-type`.
`ml-framework`	Supported ML frameworks: - `snpe` — Qualcomm Neural Processing SDK - `tflite` — LiteRT - `qnn` — Qualcomm AI Engine Direct Default: `snpe`
`model`	Optional model path that overrides the default model path. Default model file: `yolox_quantized.tflite` Other supported models: - YOLOv5 SNPE DLC: `yolov5.dlc` - YOLOv8 SNPE DLC: `yolov8.dlc` - YOLO NAS SNPE DLC: `yolonas.dlc` - YOLOv5 TFLite: `yolov5.tflite` - YOLOv8 TFLite: `yolov8_det_quantized.tflite` - YOLO NAS TFLite: `yolonas_quantized.tflite` - YOLOv7 TFLite: `yolov7_quantized.tflite` - YOLOX TFLite: `yolox_quantized.tflite` - YOLOv8 QNN: `yolov8_det_quantized.bin` Default model directory: `$HOME/Downloads/qimsdk_samples/models` Alternatively, provide an absolute file path.
`labels`	Optional labels path that overrides the default labels path. Default label file: `yolox.json` Other label files: - YOLOv5: `yolov5.json` - YOLOv8: `yolov8.json` - YOLOX: `yolox.json` - YOLO NAS: `yolonas.json` - YOLOv7: `yolov7.json` Default labels directory: `$HOME/Downloads/qimsdk_samples/labels` Alternatively, provide an absolute file path.
`threshold`	Detection threshold from `0` to `100`. Default: `40`
`runtime`	Supported hardware runtimes: - `cpu` - `gpu` - `dsp` Default: `dsp`
`snpe-tensors`	JSON array specifying the output tensors for the SNPE model.

AI Audio Applications

Audio Classification

The gst-ai-audio-classification application shows audio classification using input from either a file source or a microphone. It displays both the classification results and a video preview. The following figure shows the pipeline, which gets the input from a file or a microphone, preprocesses it, and runs inferences on AI hardware. The results are displayed on the screen. For information about the plugins used in the pipeline flow, see Pipeline flow.

Application: gst-ai-audio-classification

Try me

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	`yamnet.tflite`	`yamnet.json`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-audio-classification --config-file=/etc/configs/config-audio-classification.json

This command runs the application using the sample configuration in /etc/configs/config-audio-classification.json.To view the available command-line options, run:

gst-ai-audio-classification -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-audio-classification --config-file=/path/to/config-audio-classification.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config-audio-classification.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config-audio-classification.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config-audio-classification.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config-audio-classification.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config-audio-classification.json "$XDG_CONFIG_HOME/qimsdk_samples/config-audio-classification.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-audio-classification

Configurations

The gst-ai-audio-classification application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "file-path": "<path to video file>",
  "model": "<path to model file>",
  "labels": "<path to label file>",
  "threshold": <integer between 1 and 100>,
  "runtime": "<cpu or gpu>",
  "codec": "<mp3 or flac>"
}

Configuration field	Resolution of a relative path
`file-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`

Available configurations

Config #2
Config #1

File source, LiteRT model, CPU runtime

{
  "file-path": "video-mp3.mp4",
  "model": "yamnet.tflite",
  "labels": "yamnet.json",
  "runtime": "cpu",
  "threshold": 20,
  "codec": "mp3"
}

File source, LiteRT model, CPU runtime

{
  "file-path": "video-mp3.mp4",
  "model": "yamnet.tflite",
  "labels": "yamnet.json",
  "runtime": "cpu",
  "threshold": 20,
  "codec": "mp3"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The output video and classified audio are played on the screen.

Pipeline Flow

The following table lists the plugins used in the audio classification pipeline:

Plugin	Description
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for processing.
`h264parse`	• Parses the H.264 video.
`v4l2h264dec`	• Decodes the video bitstream into raw frames.
`mpegaudioparse` / `flacparse`	• Parses the audio bitstream (MP3 or FLAC) to ensure downstream elements can handle the payload.
`mpg123audiodec` / `flacdec`	• Decodes the compressed audio (MP3 or FLAC) into raw audio buffers.
`audioconvert`	• Converts raw audio buffers between various possible formats to ensure compatibility.
`audioresample`	• Resamples the audio buffers to different sample rates as required by the model.
`pulsesrc`	• Reads the live audio stream from the microphone.
`audiobuffersplit`	• Splits the incoming audio buffers into equal-sized chunks for consistent processing.
qtimlaconverter`	1. Receives the audio stream on its sink pad. 2. Performs preprocessing on the audio stream data. 3. Converts the stream to a tensor stream for inferencing in the later stages of the pipeline.
`qtimltflite`	1. Receives the tensor stream on its sink pad. 2. Performs inferencing using the YAMNet model. 3. Produces a tensor stream with the results on its source pad.
`qtimlpostprocess`	• Uses the `yamnet` module to handle audio classification inference results: • Applies a threshold to the chosen number of results. • Creates a text overlay for the identified audio classes.
`qtivcomposer`	• Combines the text overlay for classification results and the video preview into a single composed frame.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Descriptions
`file-path`	Path to the input audio file. If not provided, `pulsesrc` is selected as the input source.
`model`	Path to the model file. Default: `$HOME/Downloads/qimsdk_samples/models/yamnet.tflite`
`labels`	Path to the labels file. Default: `$HOME/Downloads/qimsdk_samples/labels/yamnet.json`
`threshold`	Classification threshold from `0` to `100`. Default: `40`
`runtime`	Supported hardware runtimes: - `cpu` - `gpu` Default: `cpu`
`codec`	Audio codec for the input file: - `mp3` - `flac` Default: `mp3`

AI Multi-Model Applications

Face Recognition

The gst-ai-face-recognition application collects the live video input from a camera or an RTSP stream and shares this input for face detection, facial landmarking, and face recognition. It uses the face_det_quantized models for face detection, facemap_3dmm_quantized model for facial landmarking, and face_attrib_net_quantized model for face recognition labels. The result is a preview of the overlaid AI model on the HDMI display.

This application isn’t supported in Config #1 of Qualcomm Linux because CPU runtime is not supported.

The following figure shows the pipeline, which receives the input, preprocesses it, runs inferences on AI hardware, and displays the results on the screen.

Application: gst-ai-face-recognition For information about the plugins used in the pipeline flow, see Pipeline flow.

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #2	No	Yes	No	Yes	No	Yes	No

camera use-case is not supported on IQ-2390 and CQ-2390.

Sample Model and Label Files

Runtime	Model files	Label files
Qualcomm AI Engine Direct & LiteRT	`face_det_lite_quantized.tflite`, `facemap_3dmm_quantized.tflite`, `face_attrib_net_quantized.tflite`, `face_det_lite_quantized.bin`, `facemap_3dmm_quantized.bin`, `face_attrib_net_quantized.bin`	`face_detection.json`, `face_recognition_settings.json`, `face_recognition.json`, `facemap_3dmm_settings.json`

Register a face for facial recognition

Before running the gst-ai-face-recognition application, you can register a face for secure verification and authentication:

Ensure that you complete the Prerequisites.

To register a face, use the following gst-pipeline on the target device shell:

gst-pipeline-app -e \
qtimlvconverter name=stage_01_preproc mode=image-batch-non-cumulative \
qtimltflite name=stage_01_inference model=${HOME}/Downloads/qimsdk_samples/models/face_det_lite_quantized.tflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
external-delegate-options="QNNExternalDelegate,backend_type=htp;" \
qtimlpostprocess name=stage_01_postproc settings="{\"confidence\": 40.0}" results=4 module=qfd labels=${HOME}/Downloads/qimsdk_samples/labels/face_detection.json \
qtimlvconverter name=stage_03_preproc mode=roi-batch-cumulative \
qtimltflite name=stage_03_inference model=${HOME}/Downloads/qimsdk_samples/models/face_attrib_net_quantized.tflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
external-delegate-options="QNNExternalDelegate,backend_type=htp;" \
qticamsrc video_0::type=video name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080 ! queue ! waylandsink fullscreen=true sync=false \
camsrc.image_1 ! video/x-raw,width=1920,height=1080 ! qtivtransform ! video/x-raw,format=NV12 ! tee name=t_split_1 \
t_split_1. ! queue ! metamux_1. \
t_split_1. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! stage_01_inference. stage_01_inference. ! queue ! \
stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! metamux_1. \
qtimetamux name=metamux_1 ! queue ! tee name=t_split_3 \
t_split_3. ! queue ! stage_03_preproc. stage_03_preproc. ! queue ! stage_03_inference. stage_03_inference. ! queue ! \
multifilesink location=${HOME}/Downloads/qimsdk_samples/data/tensor_%d.bin sync=true async=false enable-last-sample=false

To prepare for capturing a facial image, do the following:

Select the following options from the list. Choose the number corresponding to the option:

PLAYING: Move the pipeline to the Playing state.
Plugin Mode ➔ camsrc ➔ capture-image: Capture the image using a camera source.

Using the live preview on the display, face the camera and ensure that the camera is pointed straight and there is only one person in the frame.

In the terminal, enter 1 for the following values:

GstImageCaptureMode for arg0.
guint for arg1.

To capture all the sides of your face, select capture-image do the following for each side:

Left and right: Turn your head left by 40° while keeping the landmarks visible, then repeat steps 3 and 4. Turn your head right (by 40°) and repeat.

Up and down: Raise your head by 30° while keeping the landmarks visible, then repeat steps 3 and 4. Lower your head (by 30°) and repeat.

To stop the pipeline, use (b)Back and (q)Quit.
After running the pipeline, five individual tensor bins are created (tensor_0.bin to tensor_4.bin) with facial properties recorded for each side of the face.

On the target device, go to ${HOME}/Downloads/qimsdk_samples/data/, find the tensor bins.

cd ${HOME}/Downloads/qimsdk_samples/data

To merge the tensor bins with all the facial properties into a cohesive image, download and run the facedb.py script in the same directory as the tensor bins on the target device.

Download the facedb.py script:

curl -L -O https://raw.githubusercontent.com/quic/sample-apps-for-qualcomm-linux/refs/heads/main/qualcomm-linux/scripts/facedb.py

Run the script. Note that <Name of the person> is case and style sensitive. Ensure that you use the same name consistently.

python3 ./facedb.py "<Name of the person>" 512 32 tensor_0.bin tensor_1.bin tensor_2.bin tensor_3.bin tensor_4.bin

A face.bin binary is created.

Rename the generated face.bin file to face{N}.bin, replacing {N} with the next serial number—for example, face0.bin, face1.bin and so on.

Use the following reference face_recognition.json file as an example for registering a new person in the database. The sample configuration demonstrates face registration for two individuals.

[
  {"id": 0, "color": "0x00FF00FF", "label": "<Name of Person>"},
  {"id": 1, "color": "0xFFFF00FF", "label": "<Name of Person>"}
]

Update the ID field according to the number in the list. If more faces are registered, add the structure in a new line within face_recognition.json.

Use the following face_recognition_settings.json file as a reference when specifying the registered face database files:

{
  "confidence": 51.0,
    "databases":[
      {"id":  0, "database": "/root/Downloads/qimsdk_samples/data/face0.bin"},
      {"id":  1, "database": "/root/Downloads/qimsdk_samples/data/face1.bin"}
    ]
}

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-face-recognition --config-file=/etc/configs/config-face-recognition.json

This command runs the application using the sample configuration in /etc/configs/config-face-recognition.json.To view the available command-line options, run:

gst-ai-face-recognition -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-face-recognition --config-file=/path/to/config-face-recognition.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config-face-recognition.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config-face-recognition.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config-face-recognition.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config-face-recognition.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config-face-recognition.json "$XDG_CONFIG_HOME/qimsdk_samples/config-face-recognition.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-face-recognition

Configurations

The gst-ai-face-recognition application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "ml-framework": "<qnn or tflite>",
  "face-detection-model": "<path for face detection model file>",
  "face-landmark-model":"<path for face landmark model file>",
  "face-recognition-model":"<path for face recognition model file>",
  "face-detection-labels":"<path for face detection labels file>",
  "face-recognition-labels":"<path for face recognition labels file>",
  "face-recognition-settings": "<Path of face recognition settings>",
  "facemap-3dmm-settings": "<Path of facemap-3dmm settings>"
}

Configuration field	Resolution of a relative path
`face-detection-model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`face-landmark-model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`face-recognition-model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`face-detection-labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`
`face-recognition-labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`
`face-recognition-settings`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`
`facemap-3dmm-settings`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`

Available configurations

Config #2

Camera source, LiteRT, and DSP runtime

{
  "ml-framework":"tflite",
  "face-detection-model":"face_det_lite_quantized.tflite",
  "face-landmark-model":"facemap_3dmm_quantized.tflite",
  "face-recognition-model":"face_attrib_net_quantized.tflite",
  "face-detection-labels": "face_detection.json",
  "face-recognition-labels": "face_recognition.json",
  "face-recognition-settings": "face_recognition_settings.json",
  "facemap-3dmm-settings": "facemap_3dmm_settings.json"
}

Expected output

Pipeline Flow

The following table lists the plugins used in the face recognition pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`h264parse`	• Parses the H.264 video bitstream.
`v4l2h264dec`	• Hardware-decodes H.264 video to raw frames.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	1. After the inference runtime receives the tensor stream on its sink pad, it runs the inference. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess` (Pose-estimation)	• Uses the `lite-3dmm` module to perform facial pose recognition.
`qtimlpostprocess` (Classification)	• Uses the `qfr` module to receive the stream from `qtimetamux` and classifies the face.
`qtimetamux`	1. Receives the output of the face detection models from `qtimlpostprocess` and multiplexes it. 2. Receives the output of facial pose from `qtimlpostprocess` and multiplexes it.
`tee`	• Splits the stream for inferencing.
`qtivoverlay`	1. Receives the multiplexed stream. 2. Overlays the bounding boxes on the stream.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Description
`camera`	Select the camera: - `0` – Primary camera - `1` – Secondary camera
`rtsp-ip-port`	Provide the RTSP input URI. Example: `rtsp://192.168.1.110:8554/live.mkv`
`ml-framework`	Select the ML framework: - `tflite` – Execute the model using TFLite. Default - `qnn` – Execute the model using QNN
`tflite-face-detection-model`	Optional path to the TFLite face detection model. Overrides the default path. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` An absolute file path can also be provided.
`tflite-face-landmark-model`	Optional path to the TFLite face landmark model. Overrides the default path. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` An absolute file path can also be provided.
`tflite-face-recognition-model`	Optional path to the TFLite face recognition model. Overrides the default path. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` An absolute file path can also be provided.
`qnn-face-detection-model`	Optional path to the QNN face detection model. Overrides the default path. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` An absolute file path can also be provided.
`qnn-face-landmark-model`	Optional path to the QNN face landmark model. Overrides the default path. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` An absolute file path can also be provided.
`qnn-face-recognition-model`	Optional path to the QNN face recognition model. Overrides the default path. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` An absolute file path can also be provided.
`face-detection-labels`	Optional path to the face detection labels file. Overrides the default path. Label files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` An absolute file path can also be provided.
`face-recognition-labels`	Optional path to the face recognition labels file. Overrides the default path. Label files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` An absolute file path can also be provided.
`facemap-3dmm-settings`	Optional path to the face map settings file. Overrides the default path. Settings files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` An absolute file path can also be provided.
`face-recognition-settings`	Optional path to the face recognition settings file. Overrides the default path. Settings files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` An absolute file path can also be provided.

Daisychain Detection + Classification

The gst-ai-daisychain-detection-classification application allows you to perform cascaded object detection and classification with a camera, file source, or RTSP stream. The use case involves detecting objects and classifying the detected objects. The following figures show the pipeline workflow, which captures the video stream from the source, preprocesses it, and runs inferences using AI hardware. The results are either displayed on the screen, saved as an encoded MP4 file, or streamed over the RTSP server. For information about the plugins used in this pipeline, see Pipeline flow.

Application: gst-ai-daisychain-detection-classification

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	Yes	No	Yes	Yes	Yes
Config #2	Yes	Yes	Yes	Yes	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model files	Label files
LiteRT	detection: `yolox_quantized.tflite`, classification: `inception_v3_quantized.tflite`	`yolox.json`, `classification.json`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-daisychain-detection-classification --config-file=/etc/configs/config_daisychain_detection_classification.json

This command runs the application using the sample configuration in /etc/configs/config_daisychain_detection_classification.json.To view the available command-line options, run:

gst-ai-daisychain-detection-classification -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-daisychain-detection-classification --config-file=/path/to/config_daisychain_detection_classification.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config_daisychain_detection_classification.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config_daisychain_detection_classification.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config_daisychain_detection_classification.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config_daisychain_detection_classification.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config_daisychain_detection_classification.json "$XDG_CONFIG_HOME/qimsdk_samples/config_daisychain_detection_classification.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-daisychain-detection-classification

Configurations

The gst-ai-daisychain-detection-classification application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "input-file": "<input-video-path>",
  "detection-model": "<path-to-detection model>",
  "detection-labels": "<path-to-detection-labels>",
  "classification-model": "<path-to-classification-model>",
  "classification-labels": "<path-to-classification-labels>",
  "detection-runtime": "<Can be dsp or cpu or gpu>",
  "classification-runtime": "<Can be dsp or cpu or gpu>"
}

Configuration field	Resolution of a relative path
`input-file`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`detection-model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`classification-model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`detection-labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`
`classification-labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`
`output-file`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`

For USB camera input, set the video-format, resolution, and framerate parameters in the config file to match the camera capabilities, see Configure USB camera.

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
  "input-file": "video.mp4",
  "detection-model": "yolox_quantized.tflite",
  "detection-labels": "yolox.json",
  "classification-model": "inception_v3_quantized.tflite",
  "classification-labels": "classification.json",
  "detection-runtime": "dsp",
  "classification-runtime": "dsp"
}

File source, LiteRT model, CPU runtime

{
  "input-file": "video.mp4",
  "detection-model": "yolox_quantized.tflite",
  "detection-labels": "yolox.json",
  "classification-model": "inception_v3_quantized.tflite",
  "classification-labels": "classification.json",
  "detection-runtime": "cpu",
  "classification-runtime": "cpu"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The cropped video frame is overlaid on the frame and displayed on a local device.

The classification models trained on the Imagenet dataset don’t contain the person class.

Pipeline Flow

The following table lists the plugins used in the daisychain detection and classification pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`v4l2src`	• Captures the live stream from USB camera. • Uses `tee` to split the stream for inferencing.
`h264parse`	• Parses the H.264 video bitstream.
`v4l2h264dec`	• Hardware-decodes H.264 video to raw frames.
`qtimetamux`	• Multiplexes the stream.
`qtivsplit`	• Crops the full frame into smaller frames based on the detected bounding boxes (maximum 4).
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	1. After the inference runtime receives the tensor stream on its sink pad, it runs the inference. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess` (Detection)	• Handles inference results from any object detection model: • Applies a threshold to the chosen number of results. • Loads the YOLOv8 module. • Produces video frames with only bounding boxes that can be overlaid on objects. • Produces video frames with only bounding boxes that can be cropped.
`qtimlpostprocess` (Classification)	• Processes results on the cropped frame: • Applies the threshold to the chosen number of results on the cropped frame. • Loads the MobileNet-softmax module. • Produces results as video frames with classification labels. • Sends them to the sink pad of `qtivcomposer`.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`filesink`	• Receives the video stream on its sink pad and saves it as an H.264-encoded MP4 file.
`qtirtspbin`	1. Serves as a network sink. 2. Transmits UDP packets to the network.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Description
`input-file`	Path to the input media file. The media file should be placed in: `$HOME/Downloads/qimsdk_samples/media` Alternatively, provide an absolute file path.
`rtsp-ip-port`	Provide the RTSP input URI. Example: `rtsp://192.168.1.110:8554/live.mkv` If neither `input-file` nor `rtsp-ip-port` is provided, camera input is selected.
`detection-model`	Path to the YOLOX detection model file. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` Alternatively, provide an absolute file path.
`detection-labels`	Path to the YOLOX labels file. Label files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` Alternatively, provide an absolute file path.
`classification-model`	Path to the classification model file. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` Alternatively, provide an absolute file path.
`classification-labels`	Path to the classification labels file. Label files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` Alternatively, provide an absolute file path.
`enable-usb-camera`	Enables or disables the USB camera input. Accepted values: - `TRUE` - `FALSE`
`width`	USB camera resolution width.
`height`	USB camera resolution height.
`framerate`	USB camera frame rate.
`video-format`	USB video format. Supported values: - `nv12` - `yuy2` - `mjpeg`
`output-type`	Select the output type. Supported values: - `waylandsink` - `filesink` - `rtspsink`
`output-file`	Path to the output file. Output files should be placed in: `$HOME/Downloads/qimsdk_samples/media` Alternatively, provide an absolute file path.
`output-ip-address`	IP address for the RTSP output. Example: `127.0.0.1` Default IP address: `127.0.0.1`
`port`	Port for the RTSP output. Example: `8900` Default port: `8900`
`classification-runtime`	Runtime used for classification. Supported values: - `cpu` - `gpu` - `dsp`
`detection-runtime`	Runtime used for detection. Supported values: - `cpu` - `gpu` - `dsp`

Daisychain Detection + Pose

The gst-ai-daisychain-detection-pose application allows you to perform cascaded object detection and pose detection with a camera, file source, or an RTSP stream. The use cases involve detecting objects and estimating the body poses of the subject in an image or a video. The following figure show the application workflow, which receives the source, postprocesses it, and runs inferences on AI hardware. The results are either displayed on the screen, saved as an encoded MP4 file, or streamed over the RTSP server. For information about the plugins used in the pipeline flow, see Pipeline flow.

Application: gst-ai-daisychain-detection-pose

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #1	Yes	Yes	Yes	No	Yes	Yes	Yes
Config #2	Yes	Yes	Yes	Yes	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model files	Label files
LiteRT	detection: `yolox_quantized.tflite`, pose: `hrnet_pose_quantized.tflite`	detection: `yolox.json`, pose: `hrnet_pose.json`, pose: `hrnet_settings.json`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-daisychain-detection-pose --config-file=/etc/configs/config-daisychain-detection-pose.json

This command runs the application using the sample configuration in /etc/configs/config-daisychain-detection-pose.json.To view the available command-line options, run:

gst-ai-daisychain-detection-pose -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-daisychain-detection-pose --config-file=/path/to/config-daisychain-detection-pose.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config-daisychain-detection-pose.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config-daisychain-detection-pose.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config-daisychain-detection-pose.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config-daisychain-detection-pose.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config-daisychain-detection-pose.json "$XDG_CONFIG_HOME/qimsdk_samples/config-daisychain-detection-pose.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-daisychain-detection-pose

Configurations

The gst-ai-daisychain-detection-pose application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "input-file": "<input-video-path>",
  "rtsp-ip-port": "<RTSP-IP-port-address>",
  "detection-model": "<path-to-detection model>",
  "detection-labels": "<path-to-detection-labels>",
  "detection-runtime": "<It can take dsp, gpu or cpu as runtime. Runs detection model on given runtime>",
  "pose-runtime": "<It can take dsp, gpu or cpu as runtime. Runs pose model on given runtime>",
  "output-file": "<output-file-path>"
}

Configuration field	Resolution of a relative path
`input-file`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`detection-model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`pose-model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`detection-labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`
`pose-labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`
`pose-settings-path`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`
`output-file`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`

For QCS6490, if file-path and rtsp-ip-port are not present in the configuration file, then the camera input is selected.

For USB camera input, set the video-format, resolution, and framerate parameters in the config file to match the camera capabilities, see Configure USB camera.

Available configurations

Config #2
Config #1

File source, LiteRT model, DSP runtime

{
 "input-file": "video.mp4",
 "pose-runtime":"dsp",
 "detection-runtime":"dsp",
 "detection-model": "yolox_quantized.tflite",
 "detection-labels": "yolox.json",
 "pose-model": "hrnet_pose_quantized.tflite",
 "pose-labels": "hrnet_pose.json",
 "pose-settings-path":"hrnet_settings.json"
}

File source, LiteRT model, CPU runtime

{
 "input-file": "video.mp4",
 "pose-runtime":"cpu",
 "detection-runtime":"cpu",
 "detection-model": "yolox_quantized.tflite",
 "detection-labels": "yolox.json",
 "pose-model": "hrnet_pose_quantized.tflite",
 "pose-labels": "hrnet_pose.json",
 "pose-settings-path":"hrnet_settings.json"
}

Config #1 supports only LiteRT models and the CPU runtime.

Expected Output

The cropped video frame is overlaid on the frame and displayed on a local device.

Pipeline Flow

The following table lists the plugins used in the daisychain detection and pose pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`v4l2src`	• Captures the live stream from USB camera. • Uses `tee` to split the stream for inferencing.
`h264parse`	• Parses the H.264 video bitstream.
`v4l2h264dec`	• Hardware-decodes H.264 video to raw frames.
`qtimetamux`	• Multiplexes the stream.
`qtivsplit`	• Crops the full frame into smaller frames based on the detected bounding boxes (maximum 4).
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	1. After the inference runtime receives the tensor stream on its sink pad, it runs the inference. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess` (Detection)	• Handles inference results from any object detection model: • Applies a threshold to the chosen number of results. • Loads the YOLOv8 module. • Produces video frames with only bounding boxes that can be overlaid on objects. • Produces video frames with only bounding boxes that can be cropped.
`qtimlpostprocess` (Pose)	• Applies a threshold to the chosen number of results. • Loads corresponding modules for various pose detection models. In this specific use case, `qtimlpostprocess` does the following: 1. Loads the HRNet module. 2. Produces results in the form of video frames with drawn poses. 3. Sends the results to the sink pad of `qtivcomposer` for further processing or display.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`filesink`	• Receives the video stream on sink pad and saves it as an H.264-encoded MP4 file.
`qtirtspbin`	1. Serves as a network sink. 2. Transmits UDP packets to the network.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Description
`input-file`	Path to the input media file. The media file should be placed in: `$HOME/Downloads/qimsdk_samples/media` Alternatively, provide an absolute file path.
`rtsp-ip-port`	Provide the RTSP input URI. Example: `rtsp://192.168.1.110:8554/live.mkv` If neither `input-file` nor `rtsp-ip-port` is provided, camera input is selected.
`detection-model`	Path to the YOLOX detection model file. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` Alternatively, provide an absolute file path.
`detection-labels`	Path to the YOLOX labels file. Label files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` Alternatively, provide an absolute file path.
`pose-model`	Path to the pose detection model file. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` Alternatively, provide an absolute file path.
`pose-labels`	Path to the pose detection labels file. Label files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` Alternatively, provide an absolute file path.
`pose-settings-path`	Path to the pose settings file. Settings files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` Alternatively, provide an absolute file path.
`output-file`	Path to the output file. For a relative path, the output is saved under: `$HOME/Downloads/qimsdk_samples/media` Alternatively, provide an absolute file path. If this field is not specified, display output is selected.
`enable-usb-camera`	Enables or disables the USB camera input. Accepted values: - `TRUE` - `FALSE`
`width`	USB camera resolution width.
`height`	USB camera resolution height.
`framerate`	USB camera frame rate.
`video-format`	USB video format. Supported values: - `nv12` - `yuy2` - `mjpeg`
`output-ip-address`	IP address for the RTSP output. Example: `127.0.0.1`
`port`	Port for the RTSP output. Example: `8900`
`pose-runtime`	Runtime used for pose detection. Supported values: - `cpu` - `gpu` - `dsp`
`detection-runtime`	Runtime used for detection. Supported values: - `cpu` - `gpu` - `dsp`

Multistream Inference

The gst-ai-multistream-inference application shows AI inference (object detection and classification) on up to 32 input streams coming from camera, file, or RTSP stream. The following figure shows the pipeline, which receives several input streams, preprocesses them, runs AI inferences, combines the streams, and merges them all into a single video output. The maximum number of input streams supported on each SoC as verified on 1080P and 720P are follows:

QCS6490–8
Dragonwing IQ-8275–16
Dragonwing IQ-9075–32
This application isn’t supported in Config #1 of Qualcomm Linux because CPU runtime is not supported.
The output is displayed on an HDMI display, saved as an H.264 encoded MP4 file, or converted into an RTSP stream.

For information about the plugins used in this pipeline, see Pipeline flow.

Application: gst-ai-multistream-inference

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #2	Yes	Yes	No	Yes	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	detection: `yolox_quantized.tflite` classification: `inception_v3_quantized.tflite`	detection: `yolox.json` classification: `classification.json`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-multistream-inference --config-file=/etc/configs/config-multistream-inference.json

This command runs the application using the sample configuration in /etc/configs/config-multistream-inference.json.To view the available command-line options, run:

gst-ai-multistream-inference -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-multistream-inference --config-file=/path/to/config-multistream-inference.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config-multistream-inference.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config-multistream-inference.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config-multistream-inference.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config-multistream-inference.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config-multistream-inference.json "$XDG_CONFIG_HOME/qimsdk_samples/config-multistream-inference.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-multistream-inference

Configurations

The gst-ai-multistream-inference application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "input-file-path": ["<input file 1>", "<input file 2>"],
  "input-rtsp-path": ["<rtsp stream 1>", "<rtsp stream 2 >"],
  "input-type": "<h264 or h265>",
  "model": "<path to model file>",
  "labels": "<path to label file>",
  "output-display": "<0 or 1>",
  "output-file-path": "<output file path>",
  "output-ip-address": "<ip address of test kit>",
  "output-port-number": "<port number over which rtsp stream can be listened>",
  "use-case": "<0 or 1>"
}

Configuration field	Resolution of a relative path
`input-file-path`	`$HOME/Downloads/qimsdk_samples/medias/<file-name>`
`model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`
`output-file-path`	`$HOME/Downloads/qimsdk_samples/medias/<file-name>`

Available configurations

Config #2

Object Detection on 8 H.264 file inputs, LiteRT model, DSP runtime

{
  "input-file-path":
    [
      "video.mp4",
      "video.mp4",
      "video.mp4",
      "video.mp4",
      "video.mp4",
      "video.mp4",
      "video.mp4",
      "video.mp4"
    ],
  "model": "yolox_quantized.tflite",
  "labels": "yolox.json",
  "input-type": "h264",
  "output-display": 1,
  "use-case": 0
}

Expected Output

Pipeline Flow

The following table lists the plugins used in the multistream inference pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`, followed by `qtdemux`, which demultiplexes the stream. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`h264parse`	• Parses the H.264 video.
`v4l2h264dec`	• Decodes the video.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data. This preprocessing is done when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization 3. Converts the preprocessed video stream to a tensor stream on its source pad. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	1. After the inference runtime receives the tensor stream on its sink pad, it runs the inference. 2. Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	Handles the inference results from any object detection, classification, pose detection, and segmentation model. Detection use case: • Applies a threshold to the chosen number of results. • Loads the YOLOv8 module. • Produces video frames with only bounding boxes that can be overlaid on objects, sending them to the sink pad of `qtivcomposer`. Classification use case: • Applies the threshold to the chosen number of results. • Loads the MobileNet-softmax module. • Produces results as video frames with classification labels, sending them to the sink pad of `qtivcomposer`.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. `waylandsink` submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.
`filesink`	1. Receives the video stream on its sink pad. 2. Saves the stream as a H.264-encoded MP4 file.
`qtirtspbin`	1. Serves as a network sink. 2. Transmits UDP packets to the network.

Config JSON Field Description

Field	Values / Description
`num-camera`	Number of camera streams. Supported values: - `1` - `2` Maximum number of camera streams: `2`
`camera-id`	Camera ID used as the input source. Supported values: - `0` - `1` If no other input is selected, camera `0` is used by default. This parameter is ignored when `num-camera` is set to `2`.
`input-file-path`	JSON array containing the input media file paths. Example: `["video1.mp4", "video2.mp4"]` Relative paths are resolved under: `$HOME/Downloads/qimsdk_samples/media` Absolute file paths are also supported. Maximum number of input files: `32`
`input-rtsp-path`	JSON array containing the input RTSP stream paths. Example: `["rtsp://127.0.0.1:8554/live1.mkv", "rtsp://127.0.0.1:8554/live2.mkv"]` Maximum number of RTSP input streams: `32`
`model`	Path to the model file. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` Alternatively, provide an absolute file path.
`labels`	Path to the labels file. Label files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` Alternatively, provide an absolute file path.
`output-file-path`	Path to save the H.264-encoded output file. Relative paths are resolved under: `$HOME/Downloads/qimsdk_samples/media` Absolute file paths are also supported.
`output-ip-address`	IP address on which the RTSP server listens. Example: `127.0.0.1` Adding this parameter enables RTSP stream output.
`output-port-number`	Port number on which the RTSP server listens. Example: `8554` Adding this parameter enables RTSP stream output.
`output-display`	Enables or disables output on the Wayland display. Set the value to `true` to enable Wayland display output.
`use-case`	Select the application use case. Supported values: - `0` – Detection - `1` – Classification
`input-type`	Encoding type for file or RTSP input. Supported values: - `h264` - `h265` The default encoding type is H.264.

Multi-Stream Batch Inference

The gst-ai-multistream-batch-inference application shows batched AI inference (object detection and segmentation) on up to 24 input streams from video files. The following figure shows the pipeline, which receives several input streams, preprocesses them, runs AI inferences, combines the streams with inference, and merges them into a single video output. The maximum number of input streams supported on each SoC are follows: QCS6490–8 Dragonwing IQ-8275–4 Dragonwing IQ-9075–4 The output is displayed either on an HDMI display or saved as an H.264 encoded MP4 file. For information about the plugins used in this pipeline, see Pipeline flow.

This application isn’t supported in Config #1 of Qualcomm Linux because CPU runtime is not supported.

Application: gst-ai-multistream-batch-inference

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #2	Yes	No	No	No	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	segmentation: `deeplabv3_plus_mobilenet_quantized.tflite` detection: `yolov8_det_w8a8.tflite`	segmentation:`deeplabv3_resnet50.json` detection: `yolov8.json`
Qualcomm AI Engine direct	segmentation: `deeplabv3_plus_mobilenet_quantized.bin` detection: `yolov8_det_quantized.bin`	segmentation:`deeplabv3_resnet50.json` detection: `yolov8.json`
Qualcomm Neural Processing SDK	segmentation: `deeplabv3_plus_mobilenet_quantized.dlc` detection: `yolov8_det_quantized.dlc`	segmentation:`deeplabv3_resnet50.json` detection: `yolov8.json`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-multistream-batch-inference --config-file=/etc/configs/config-multistream-batch-inference.json

This command runs the application using the sample configuration in /etc/configs/config-multistream-batch-inference.json.To view the available command-line options, run:

gst-ai-multistream-batch-inference -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-multistream-batch-inference --config-file=/path/to/config-multistream-batch-inference.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config-multistream-batch-inference.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config-multistream-batch-inference.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config-multistream-batch-inference.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config-multistream-batch-inference.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config-multistream-batch-inference.json "$XDG_CONFIG_HOME/qimsdk_samples/config-multistream-batch-inference.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-multistream-batch-inference

Configurations

The gst-ai-multistream-batch-inference application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "output-type": "wayland or filesink",
  "out-file":"<Path to output file if output-type is filesink>",
  "pipeline-info":[
    {
      "id": "<Batch id takes value from 0 to 5>",
      "Input type": "<Input source type like file>",
      "input-file-path": [
        {
        "<stream[i]: Comma separated input file/RTSP path>"
        }
      ],
      "mlframework": "<tflite or snpe or qnn>",
      "model-path": "<path-to-model-file>",
      "labels": "<path-to-label-file>",
      "post processing plugin": "qtimlvdetection"
    }
  ]
}

Configuration field	Resolution of a relative path
`input-file-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`model-path`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`
`out-file`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`

For 16 and 24 streams, add the required elements in the pipeline-info parameter. The id parameter takes the values from 0 to 5 for each added batch.

Available configurations

Config #2

File source, LiteRT model, DSP runtime

{
  "output-type":"wayland",
  "pipeline-info":[
    {
      "id":0,
      "input-type":"file",
      "input-file-path":[
        {
            "stream-0":"video.mp4",
            "stream-1":"video.mp4",
            "stream-2":"video.mp4",
            "stream-3":"video.mp4"
        }
      ],
      "mlframework":"tflite",
      "model-path":"yolov8_det_w8a8.tflite",
      "labels-path":"yolov8.json",
      "post-process-plugin": "qtimlvdetection"
    }
  ]
}

Expected Output

Pipeline Flow

The following table lists the plugins used in the multistream batch inference pipeline:

Plugin	Description
`filesrc`	1. Captures the video stream using `filesrc`. 2. `qtdemux` demultiplexes the stream. 3. Uses `tee` to split the stream for inferencing.
`h264parse`	Parses the H.264 video.
`v4l2h264dec`	Decodes the video.
`qtibatch`	1. Reads input from the streams on its sink pad. 2. Batches the streams for preprocessing.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization 3. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	1. After the inference runtime receives the tensor stream on its sink pad, it runs the inference. 2. Produces a tensor stream with the inference results on its source pad.
`qtimldemux`	1. Demultiplexes the batched output. 2. Splits the output corresponding to the input streams.
`qtimlpostprocess`	Converts the inference tensors received on the sink pad into video formats that the multimedia plugins can use for further processing.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.
`filesink`	Takes the video stream received on its sink pad and saves it as an H.264-encoded MP4 file.

Config JSON Field Description

Field	Values / Description
`input-type`	Specifies the input type. The application accepts file input.
`input-file-path`	Path to the input media file. Relative paths are resolved under: `$HOME/Downloads/qimsdk_samples/media` Absolute file paths are also supported.
`model-path`	Path to the model file. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` Absolute file paths are also supported.
`labels-path`	Path to the labels file. Label files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` Absolute file paths are also supported.
`post-process-plugin`	Select the post-processing plugin. Supported values: - `qtimlvsegmentation` - `qtimlvdetection`
`mlframework`	Select the ML framework. Supported values: - `tflite` - `snpe` - `qnn`
`snpe-tensors`	JSON array containing the output tensors for the SNPE model. Example: `["boxes", "scores", "class_idx"]`
`output-type`	Select the output type. Supported values: - `wayland` - `filesink`
`out-file`	Path to the output file. Relative paths are resolved under: `$HOME/Downloads/qimsdk_samples/media` Absolute file paths are also supported.

Multi input/output object detection

The gst-ai-multi-input-output-object-detection application allows you to perform object detection, object classification, pose detection, and image segmentation on an input stream from different sources such as a camera, a file, or an RTSP network. The use cases implement the LiteRT models for object detection, image segmentation, classification, and pose detection. The following figure shows the pipeline workflow, which captures video streams for inferencing from different sources such as camera, file, or RTSP. For information about the plugins used in the pipeline, see Pipeline flow.

This application isn’t supported in Config #1 of Qualcomm Linux because CPU runtime is not supported.

Application: gst-ai-multi-input-output-object-detection

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #2	Yes	Yes	No	Yes	Yes	Yes	Yes

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	`yolov5.tflite`	`yolov5.json`

Prerequisites

Update the following commands according to the Python version in your Linux host computer.

Create the Python 3.8 virtual environment:

sudo apt-get install python3.8

sudo apt-get install python3.8-venv

python3.8 -m venv py3.8

source py3.8/bin/activate

Generate the yolov5.tflite model:

git clone https://github.com/ultralytics/yolov5.git

cd yolov5

python -m pip install -r requirements.txt tensorflow-cpu

python export.py --weights yolov5m.pt --img 320 --include tflite --int8 --data data/coco128.yaml

In the terminal of the host computer, run the following command to push the model to the target device:

scp yolov5m-int8.tflite root@<ip address of the device>:~/Downloads/qimsdk_samples/models/yolov5.tflite

If any model isn’t available after downloading the script file, you can download the model from IoT– Qualcomm AI Hub.

In the terminal of the host computer, run the following command to push the model files to the target device:

scp <model_filename> root@< address of target device>:~/Downloads/qimsdk_samples/models

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-multi-input-output-object-detection --config-file=/etc/configs/config-multi-input-output-object-detection.json

This command runs the application using the sample configuration in /etc/configs/config-multi-input-output-object-detection.json.To view the available command-line options, run:

gst-ai-multi-input-output-object-detection -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-multi-input-output-object-detection --config-file=/path/to/config-multi-input-output-object-detection.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config-multi-input-output-object-detection.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config-multi-input-output-object-detection.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config-multi-input-output-object-detection.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config-multi-input-output-object-detection.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config-multi-input-output-object-detection.json "$XDG_CONFIG_HOME/qimsdk_samples/config-multi-input-output-object-detection.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-multi-input-output-object-detection

Configurations

The gst-ai-multi-input-output-object-detection application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "num-camera": "<number-of-cameras>",
  "camera-id": "<specific-camera-id>",
  "input-file-path": "<JSON-array-of-input-file>",
  "input-rtsp-path": "<JSON-array-of-RTSP-URLs>",
  "model": "<path-to-model-file>",
  "labels": "<path-to-label-file>",
  "output-file-path": "<path-to-output-file>",
  "output-ip-address": "<ip-address-of-output>",
  "output-port-number": "<port number over which rtsp stream can be listened>",
  "output-display": "<true or false>"
}

Configuration field	Resolution of a relative path
`input-file-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`labels`	`$HOME/Downloads/qimsdk_samples/labels/<file-name>`
`output-file-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`

Ensure that the total number of input streams from the camera, RTSP, and file source doesn’t exceed 6.

For QCS6490, if file-path and rtsp-ip-port are not present in the configuration file, then the camera input is selected.

Available configurations

Config #2

File source, LiteRT model, DSP runtime

{
  "input-file-path":
      [
          "video1.mp4",
          "video2.mp4"
      ],
  "model": "yolov5.tflite",
  "labels": "yolov5.json",
  "output-display": true
}

Expected Output

Based on the use case, the results are either displayed on an HDMI screen, saved as an H.264 encoded MP4 file, or streamed over the RTSP server.

Pipeline Flow

The following table lists the plugins used in the multi input output object detection pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from the camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`. • Followed by `qtdemux`, which demultiplexes the stream. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`. • Followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`h264parse`	Parses the H.264 video.
`v4l2h264dec`	Decodes the video.
`qtimlvconverter`	• Receives the video stream on its sink pad. • Performs the following preprocessing on the stream data when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization • The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	• Runs on LiteRT and uses the `yolov5.tflite` model for object detection. • After the inference runtime receives the tensor stream on its sink pad, it runs the inference. • Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess`	• Converts the inference tensors that it receives on its sink pad into video formats that the multimedia plugins can process later.
`qtivcomposer`	• Composes frames with contents from its sink pads. • Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	• `waylandsink` submits the video stream received on its sink pad to the Wayland compositor. • Renders the video stream on a local display.
`filesink`	Takes the video stream that it receives on its sink pad and saves it as an H.264-encoded MP4 file.
`qtirtspbin`	• Serves as a network sink. • Transmits UDP packets to the network.

Config JSON Field Description

Field	Values / Description
`num-camera`	Number of camera streams. Supported values: - `1` - `2` Maximum number of camera streams: `2`
`camera-id`	Camera ID used as the input source. Supported values: - `0` - `1` Camera `0` is selected by default if no other input is specified. This parameter is ignored when `num-camera` is set to `2`.
`input-file-path`	JSON array containing the input file paths. Example: `["video1.mp4", "video2.mp4"]` Relative paths are resolved under: `$HOME/Downloads/qimsdk_samples/media` Absolute file paths are also supported. Maximum number of input files: `6`
`input-rtsp-path`	JSON array containing the input RTSP stream paths. Example: `["rtsp://127.0.0.1:8554/live1.mkv", "rtsp://127.0.0.1:8554/live2.mkv"]` Maximum number of RTSP input streams: `6`
`model`	Path to the model file. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` Alternatively, provide an absolute file path.
`labels`	Path to the labels file. Label files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` Alternatively, provide an absolute file path.
`output-file-path`	Path to save the H.264-encoded output file. Relative paths are resolved under: `$HOME/Downloads/qimsdk_samples/media` Absolute file paths are also supported.
`output-ip-address`	IP address on which the RTSP server listens. Example: `127.0.0.1` Adding this parameter enables RTSP stream output.
`output-port-number`	Port number on which the RTSP server listens. Example: `8554` Adding this parameter enables RTSP stream output.
`output-display`	Enables or disables output on the Wayland display. Set the value to `true` to enable Wayland display output. If no output is selected, Wayland output is selected by default.

Parallel Inferencing

The gst-ai-parallel-inference application allows you to perform object detection, object classification, pose detection, and image segmentation on an input stream from different sources such as a camera, a file, or an RTSP network. The use cases implement the LiteRT models for object detection, image segmentation, classification, and pose detection. The following figure shows the pipeline, which receives input streams from a camera, file, or an RTSP stream, performs the parallel inferencing for the four use cases, and displays the results side by side on the screen.

This application isn’t supported in Config #1 of Qualcomm Linux because CPU runtime is not supported.

For information about the plugins used in this pipeline, see Pipeline flow.

Application: gst-ai-parallel-inference

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output
Config #2	Yes	Yes	No	Yes	No	Yes	No

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	detection: `yolox_quantized.tflite` classification: `inception_v3_quantized.tflite` segmentation: `deeplabv3_plus_mobilenet_quantized.tflite` pose: `hrnet_pose_quantized.tflite`	detection: `yolox.json` classification: `classification.json` segmentation: `deeplabv3_resnet50.json` pose: `hrnet_pose.json`, `hrnet_settings.json`

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device.

Sample configuration files are available in /etc/configs. You can use the sample configuration or create a custom configuration file.

Run the application with the sample configuration file

To run the application with the sample configuration file, use:

gst-ai-parallel-inference --config-file=/etc/configs/config-parallel-inference.json

This command runs the application using the sample configuration in /etc/configs/config-parallel-inference.json.To view the available command-line options, run:

gst-ai-parallel-inference -h

Press CTRL + C to stop the execution.

Run the application with a custom configuration file

You can use a custom configuration file by specifying its path with --config-file:

gst-ai-parallel-inference --config-file=/path/to/config-parallel-inference.json

If --config-file is specified, the application uses that file. Otherwise, it searches for config-parallel-inference.json in the following order:

Priority	Location	Description
1	The path provided with `--config-file`.	Uses the configuration file specified on the command line.
2	`$XDG_CONFIG_HOME/qimsdk_samples/config-parallel-inference.json`	Uses the user-defined configuration directory when the environment variable is set.
3	`$HOME/.config/qimsdk_samples/config-parallel-inference.json`	Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`.
4	`/etc/configs/config-parallel-inference.json`	Uses the system-wide configuration file as the final fallback.

To create a user-specific configuration, copy the sample file to the user configuration directory:

export XDG_CONFIG_HOME="$HOME/.config"
mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
cp /etc/configs/config-parallel-inference.json "$XDG_CONFIG_HOME/qimsdk_samples/config-parallel-inference.json"

Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in configurations.

Run the application:

gst-ai-parallel-inference

Configurations

The gst-ai-parallel-inference application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See Config JSON Field Description for all fields.

{
  "camera": "<camera-id>",
  "file-path": "<input-video-path>",
  "rtsp-ip-port": "<RTSP-IP-Port-address>",
  "detection-model": "<path-to-detection model>",
  "detection-labels": "<path-to-detection-labels>",
  "pose-model": "<path-to-pose-model>",
  "pose-labels": "<path-to-pose-label>",
  "pose-settings-path": "<path-to-pose-settings-file>",
  "segmentation-model": "<path-to-segmentation-model>",
  "segmentation-labels": "<path-to-segmentation-labels>",
  "classification-model": "<path-to-classification-model>",
  "classification-labels": "<path-to-classification-labels>"
}

Configuration field	Resolution of a relative path
`file-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`detection-model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`pose-model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`segmentation-model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`classification-model`	`$HOME/Downloads/qimsdk_samples/models/<file-name>`
`detection-labels`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`pose-labels`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`pose-settings-path`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`segmentation-labels`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`
`classification-labels`	`$HOME/Downloads/qimsdk_samples/media/<file-name>`

For QCS6490, if file-path and rtsp-ip-port are not present in the configuration file, then the camera input is selected.

Available configurations

Config #2

File source, LiteRT model, DSP runtime

{
    "file-path": "video.mp4",
    "detection-model": "yolox_quantized.tflite",
    "detection-labels": "yolox.json",
    "pose-model": "hrnet_pose_quantized.tflite",
    "pose-labels": "hrnet_pose.json",
    "pose-settings-path": "hrnet_settings.json",
    "segmentation-model": "deeplabv3_plus_mobilenet_quantized.tflite",
    "segmentation-labels": "deeplabv3_resnet50.json",
    "classification-model": "inception_v3_quantized.tflite",
    "classification-labels": "classification.json"
}

Expected Output

After performing the four parallel inferences, the results are displayed side by side on the screen.

Pipeline Flow

The following table lists the plugins used in the parallel inference pipeline:

Plugin	Description
`qticamsrc`	• Captures the live stream from the camera. • Uses `tee` to split the stream for inferencing.
`filesrc`	• Captures the video stream using `filesrc`. • Followed by `qtdemux`, which demultiplexes the stream. • Uses `tee` to split the stream for inferencing.
`rtspsrc`	• Captures the RTSP stream using `rtspsrc`. • Followed by `rtph264depay` for video extraction. • Uses `tee` to split the stream for inferencing.
`h264parse`	Parses the H.264 video.
`v4l2h264dec`	Decodes the video.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization 3. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite`	• After the inference runtime receives the tensor stream on its sink pad, it runs the inference. • Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess` — detection	a. Receives the inference tensors from the object detection model. b. Converts the inference tensors on its sink pad into formats such as video or text that the multimedia plugins can process later. c. Applies the threshold to the chosen number of results. d. Loads the corresponding modules for detection models. In this use case, `qtimlpostprocess` does the following: • Loads the YOLOv8 submodule. • Produces results as structures of text. • Sends them to the sink pad of `qtimetamux`.
`qtimlpostprocess` — classification	a. Receives the inference tensors from the classification model. b. Converts the inference tensors on its sink pad into formats such as video or text that the multimedia plugins can process later. c. Applies the threshold to the chosen number of results. d. Loads the corresponding modules for classification models. In this use case, `qtimlpostprocess` does the following: • Loads the submodule of the model. • Produces results as video frames with classification labels. • Sends them to the sink pad of `qtivcomposer`.
`qtimlpostprocess` — segmentation	a. Receives the inference tensors on its sink pad. b. Converts the inference tensors into video formats that the multimedia plugins can process later. c. Produces the semantic segmentations for the frame. d. Loads the corresponding modules for the segmentation models. In this use case, `qtimlpostprocess` does the following: • Loads the deeplab-argmax submodule. • Produces video frames with segmentation masks. • Sends them to the sink pad of `qtivcomposer`.
`qtimlpostprocess` — pose	a. Receives the inference tensors on its sink pad. b. Converts the inference tensors into video formats that the multimedia plugins can process later. c. Applies the threshold to the chosen number of results. d. Loads the corresponding modules for various pose estimation models. In this use case, `qtimlpostprocess` does the following: • Loads the HRNet module. • Produces results as video frames with poses drawn. • Sends them to the sink pad of `qtivcomposer`.
`qtivcomposer`	1. Composes frames with contents from its sink pads. 2. Pushes the GStreamer buffers containing these composed frames to its source pad.
`waylandsink`	1. Submits the video stream received on its sink pad to Weston. 2. Weston renders the video stream on a local display.

Config JSON Field Description

Field	Values / Description
`camera`	Select the camera: - `0` – Primary camera - `1` – Secondary camera
`file-path`	Path to the input media file. The media file should be placed in: `$HOME/Downloads/qimsdk_samples/media` Alternatively, provide an absolute file path.
`rtsp-ip-port`	Provide the RTSP input URI. Example: `rtsp://192.168.1.110:8554/live.mkv`
`detection-model`	Path to the object detection model. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` Alternatively, provide an absolute file path.
`detection-labels`	Path to the object detection labels. Label files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` Alternatively, provide an absolute file path.
`pose-model`	Path to the pose detection model. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` Alternatively, provide an absolute file path.
`pose-labels`	Path to the pose detection labels. Label files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` Alternatively, provide an absolute file path.
`segmentation-model`	Path to the segmentation model. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` Alternatively, provide an absolute file path.
`segmentation-labels`	Path to the segmentation labels. Label files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` Alternatively, provide an absolute file path.
`classification-model`	Path to the classification model. Model files should be placed in: `$HOME/Downloads/qimsdk_samples/models` Alternatively, provide an absolute file path.
`classification-labels`	Path to the classification labels. Label files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` Alternatively, provide an absolute file path.
`pose-settings-path`	Path to the pose settings file. Settings files should be placed in: `$HOME/Downloads/qimsdk_samples/labels` Alternatively, provide an absolute file path.

Gesture Recognition

The gst-gesture-recognition provides four-stage cascading pipeline that performs palm detection, hand landmark estimation, gesture embedding, and gesture classification on a live camera stream using ROI-based metadata propagation.

This application is not supported on the QLI GA 2.1 release for the IQ-2390 and CQ-2390 platforms.

For information about the plugins used in the pipeline flow, see Pipeline flow

Application: gst-gesture-recognition

Try me

Input and Output Capabilities

Config	File src	RTSP	USB camera	MIPI camera	File output	Display	RTSP output	Web RTC
Config #2	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes

camera use-case is not supported on IQ-2390 and CQ-2390.

Sample Model and Label Files

The gesture recognition pipeline uses the following floating-point LiteRT models and associated metadata files:

Pipeline stage	Model file	Label and settings files
Palm detection	`hand_detector.tflite`	`palmd_labels.json` `palmd_settings.json`
Hand landmark estimation	`hand_landmark.tflite`	`hlandmark_labels.json` `hlandmark_settings.json`
Gesture embedding	`gesture_embedder.tflite`
Gesture classification	`canned_gesture_classifier.tflite`	`gesture_labels.json`

All AI models use floating-point precision.

Run the application on the target device

Download artifacts

Complete the Prerequisites to copy the sample models, labels, media files, and configuration files to the target device. The application reads models from $HOME/models and labels from $HOME/labels by default.
Enable the GPU delegate before running the application, as it supports execution on GPU only.

Run the application

To run the application with the ISP camera and display the annotated stream:

gst-gesture-recognition --input-type=isp --input-config=0 --width=1280 --height=720

Display output is enabled by default but can be disabled with --no-display

To use a USB camera:

gst-gesture-recognition \
  --input-type=usb \
  --input-config=/dev/video0 \
  --width=1280 \
  --height=720 \
  --framerate=30

To process a video file and save the annotated output:

gst-gesture-recognition \
  --input-type=file \
  --input-config=${HOME}/Downloads/qimsdk_samples/media/video.mp4 \
  --output-type=file \
  --output-config=output.mp4

To view all supported options, run:

gst-gesture-recognition --help

Press CTRL + C to stop the execution.

Expected Output

The application detects hands, estimates keypoints, and recognizes gestures. Results are overlaid on each frame and rendered on the display.

Pipeline Flow

The following table lists the plugins used in the gesture recognition pipeline:

Plugin	Description
`qticamsrc`	Captures live video from the ISP camera as the pipeline source.
tee	Splits the stream for palm detection and downstream ROI-based stages.
`qtimlvconverter`	Preprocesses full frames (Stage 1) and ROI-cropped patches (Stage 2) for inference.
`qtimltflite`	Runs palm detection, hand landmark, gesture embedder, and gesture classifier inference sequentially.
`qtimlpostprocess`	Post-processes each stage’s tensors (palm ROIs, landmarks, gesture labels).
qtimetatransform	Transforms ROI palm-detection metadata into cropped regions for the landmark stage.
`qtimetamux`	Merges video and metadata/text streams, attaching inference results as GST buffer metadata.
`qtivoverlay`	Overlays inference results (labels, bounding boxes, keypoints) onto the video frame using CL.
`waylandsink`	Renders the final annotated video stream to a local display via Weston.

Config JSON Field Description

gst-gesture-recognition does not use a JSON configuration file. Configure the application with the following command-line options:

Option	Values and default
`--input-type`	Input source: `usb`, `isp`, `rtsp`, or `file`. Defaults to `isp`.
`--input-config`	Input-specific configuration. Use `/dev/videoX` for `usb`, a camera ID for `isp`, an RTSP URL for `rtsp`, or a file path for `file`. Defaults to `/dev/video0` for USB and `0` for ISP. RTSP and file inputs require a value.
`--output-type`	Output destination: `none`, `file`, `rtsp`, or `webrtc`. Defaults to `none`.
`--output-config`	Output-specific configuration. Use an output file path for `file`, a port for `rtsp`, or a WebSocket signaling URL for `webrtc`. Defaults to `output.mp4`, `8554`, and `ws://127.0.0.1:8443`, respectively.
`--model-base-path`	Base directory containing the required `models` and `labels` directories. Defaults to `$HOME`.
`--no-display`	Disables on-screen display output.
`--width`, `--height`, `--framerate`	Raw video properties. Defaults are `1920`, `1080`, and `30`, respectively.
`--webrtc-id`	Local WebRTC signaling ID. Defaults to `1010`.

Hardware benchmarking application

The hardware benchmarking application monitors the device hardware usage for a defined set of sample applications to capture metrics such as CPU/GPU/NPU usage and device thermals. These metrics explain the resource usage and throttling, which help to tune your AI use cases according to the requirements. The following figure shows the pipeline, which processes the input from a set of USB cameras to generate various outputs.

This application isn’t supported in the QLI GA 2.1 release.

For information about the plugins used in this pipeline, see Pipeline flow.

Try me

Sample Model and Label Files

Runtime	Model file	Label file
LiteRT	• inception_v3_quantized.tflite • deeplabv3_plus_mobilenet_quantized.tflite • hrnet_pose_quantized.tflite • midas_quantized.tflite • yolox_quantized.tflite	• classification.json • deeplabv3_resnet50.json • hrnet_pose.json • monodepth.json • yolox.json

Setup the target device

To access the target device from your Linux host computer, set up SSH. For instructions, see Sign in using SSH.

If SSH is already set up, you can skip this step.

Use the HDMI port to connect the display to the device. For instructions, see Set up HDMI display.If you face issues with display, see Troubleshoot display issues.

Connect two USB cameras and a mouse to the target device.

Install the Qualcomm® Profiler on the Linux host computer. For installation instructions see Qualcomm Profiler.After connecting the device to the PC, run InstallerLE from the following locations:

For Linux:

cd /opt/qcom/Shared/QualcommProfiler/API/target-le
./InstallerLE

For Windows:

cd “C:\Program Files (x86)\Qualcomm\Shared\QualcommProfiler\API\target-le”
.\InstallerLE.exe

Run the application on the target device

Clone the repository for the demo application and push it to the target device:

git clone https://github.com/Avnet/QCS6490-Vision-AI-Demo.git
cd QCS6490-Vision-AI-Demo
git checkout QLI_2.0
scp -r ../QCS6490-Vision-AI-Demo root@${IP_ADDRESS_OF_TARGET_DEVICE}:/opt

cd /opt/QCS6490-Vision-AI-Demo && bash install.sh

Start the application:

bash launch_visionai_with_env.sh

Select the preferred sample applications from the Camera 1 and Camera 2 drop-downs. The system thermal and hardware usage details appear at the bottom of the screen.You may run different sample applications to check the output and understand the hardware utilization.

Example 1: Choose the Camera option from Camera 1 and Camera 2 drop-down lists to observe the preview streams on the screen.
Example 2: Choose any sample application from Camera 1 and Camera 2 drop-down lists to observe the AI inferencing camera streams on the screen.

For more information and features of the application, select the Info icon.

Select the Exit icon to close the application.

Pipeline Flow

The following table lists the plugins used in the hardware benchmarking pipeline:

Plugin	Description
`v4l2src`	• Captures the live stream from the USB camera. • Uses `tee` to split the stream for inferencing.
`qtimlvconverter`	1. Receives the video stream on its sink pad. 2. Performs the following preprocessing on the stream data when the model expects floating-point values as input: • Color conversion • Scaling (up or down) • Normalization 3. The tensor stream is used for inferencing in the later stages of the pipeline.
`qtimltflite` `qtimltsnpe` `qtimlqnn`	• After the inference runtime receives the tensor stream on its sink pad, it runs the inference. • Produces a tensor stream with the inference results on its source pad.
`qtimlpostprocess` — detection	a. Receives the inference tensors from the object detection model. b. Converts the inference tensors on its sink pad into formats such as video or text that the multimedia plugins can process later. c. Applies the threshold to the chosen number of results. d. Loads the corresponding modules for detection models. In this use case, `qtimlpostprocess` does the following: • Loads the YOLO (`YOLOv5`, `YOLOv8`, `YOLOX`, or `YOLO-NAS`) submodule. • Produces results as structures of text. • Sends them to the sink pad of `qtimetamux`.
`qtimlpostprocess` — classification	a. Receives the inference tensors from the classification model. b. Converts the inference tensors on its sink pad into formats such as video or text that the multimedia plugins can process later. c. Applies the threshold to the chosen number of results. d. Loads the corresponding modules for classification models. In this use case, `qtimlpostprocess` does the following: • Loads the submodule of the model. • Produces results as video frames with classification labels. • Sends them to the sink pad of `qtivcomposer`.
`qtimlpostprocess` — segmentation	a. Receives the inference tensors on its sink pad. b. Converts the inference tensors into video formats that the multimedia plugins can process later. c. Produces the semantic segmentations for the frame. d. Loads the corresponding modules for the segmentation models. In this use case, `qtimlpostprocess` does the following: • Loads the `deeplab-argmax` submodule. • Produces video frames with segmentation masks. • Sends them to the sink pad of `qtivcomposer`.
`qtimlpostprocess` — pose	a. Receives the inference tensors on its sink pad. b. Converts the inference tensors into video formats that the multimedia plugins can process later. c. Applies the threshold to the chosen number of results. d. Loads the corresponding modules for various pose estimation models. In this use case, `qtimlpostprocess` does the following: • Loads the `HRNet` module. • Produces results as video frames with poses drawn. • Sends them to the sink pad of `qtivcomposer`.

Troubleshooting

Model not available after running the download script?

If any model isn’t available after downloading the script file, you can download the model manually from IoT — Qualcomm AI Hub and push it to the target device:

scp <model filename> root@<ip addr of the target device>:~/Downloads/qimsdk_samples/models

For example:

scp mobilenet_v2_quantized.tflite root@<ip addr of the target device>:~/Downloads/qimsdk_samples/models

Running sample applications from the UART shell

Remount the file system with read/write permissions:For Qualcomm Linux:

mount -o remount,rw /usr

Cannot locate the qticamsrc plugin?

If you cannot locate the qticamsrc plugin, ensure that the camera server is running and clear the GStreamer cache using the following commands:

ps -ef | grep cam-server

rm ~/.cache/gstreamer-1.0/registry.aarch64.bin

GStreamer logging and debugging

To enable basic GStreamer logging, run the following before launching the application:

export GST_DEBUG=2

To increase verbosity for specific plugins, use a comma-separated list with log levels (1–7):

export GST_DEBUG=3,qticamsrc:5,qtimlvconverter:5,qtimltflite:5

To redirect logs to a file for offline analysis:

export GST_DEBUG=3 
export GST_DEBUG_FILE=/tmp/gst_classification_debug.log 

​Prerequisites

​AI Vision Applications

​Object Detection

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​Image Classification

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​Face Detection

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​Semantic Segmentation

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​Pose Detection

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​Mono Depth

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​Super Resolution

​Input and Output Capabilities

​Sample Model Files

​Run the application on the target device

Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​AI Event Encoder

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​Metadata Parser

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

Configurations

​Expected Output

Pipeline Flow

Config JSON Field Description

​AI USB Camera

​Input and Output Capabilities

​Sample Model and Label Files

​Run the application on the target device

Configurations

​Expected Output

Prerequisites

AI Vision Applications

Object Detection

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Expected Output

Image Classification

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Expected Output

Face Detection

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Expected Output

Semantic Segmentation

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Expected Output

Pose Detection

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Expected Output

Mono Depth

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Expected Output

Super Resolution

Input and Output Capabilities

Sample Model Files

Run the application on the target device

Expected Output

AI Event Encoder

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Expected Output

Metadata Parser

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Expected Output

AI USB Camera

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Expected Output

AI Audio Applications

Audio Classification

Sample Model and Label Files

Run the application on the target device

Expected Output

AI Multi-Model Applications

Face Recognition

Input and Output Capabilities

Sample Model and Label Files

Register a face for facial recognition

Run the application on the target device

Expected output

Daisychain Detection + Classification

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Expected Output

Daisychain Detection + Pose

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Expected Output

Multistream Inference

Input and Output Capabilities

Sample Model and Label Files

Run the application on the target device

Expected Output

Multi-Stream Batch Inference