# Debugging Guide
Source: https://imsdkdocs.qualcomm.com/advanced/debugging

Developer guide for debugging, profiling, and tracing QIM SDK GStreamer pipelines.

This guide covers practical debugging techniques for QIM SDK AI pipelines — from enabling verbose logs to measuring per-stage latency, tracing buffers, and diagnosing camera or performance issues.

***

## 1. Enable Debug Logs

QIM SDK uses the standard GStreamer debug system. Pass `--gst-debug=<level>` to `gst-launch-1.0`, or set the `GST_DEBUG` environment variable for Python/C++ apps.

### Debug Levels

| Level | Name | Description |
| - | - | - |
| 0 | none | No debug information is output. |
| 1 | ERROR | Logs all fatal errors. These are errors that do not allow the core or elements to perform the requested action. The application can still recover if programmed to handle the conditions that triggered the error. |
| 2 | WARNING | Logs all warnings. Typically these are non-fatal, but user-visible problems are expected to happen. |
| 3 | FIXME | Logs all "fixme" messages. Those typically that a codepath that is known to be incomplete has been triggered. It may work in most cases, but may cause problems in specific instances. |
| 4 | INFO | Logs all informational messages. These are typically used for events in the system that only happen once, or are important and rare enough to be logged at this level. |
| 5 | DEBUG | Logs all debug messages. These are general debug messages for events that happen only a limited number of times during an object's lifetime; these include setup, teardown, change of parameters, etc. |
| 6 | LOG | Logs all log messages. These are messages for events that happen repeatedly during an object's lifetime; these include streaming and steady-state conditions. This is used for log messages that happen on every buffer in an element for example. |
| 7 | TRACE | Logs all trace messages. Those are message that happen very very often. This is for example is each time the reference count of a GstMiniObject, such as a GstBuffer or GstEvent, is modified. |
| 9 | MEMDUMP | Logs all memory dump messages. This is the heaviest logging and may include dumping the content of blocks of memory. |

### Global Level

```bash theme={null}
gst-launch-1.0 --gst-debug=3 \
filesrc location=... ! ...
```

### Per-Element Level

Override the global level for a specific plugin — useful for isolating one stage without drowning in output from everything else:

```bash theme={null}
# Global = 2 (warnings), but inference plugin = 6 (verbose)
gst-launch-1.0 --gst-debug=2,qtimltflite:6 \
filesrc location=... ! ...
```

### Common Starting Points

| Flag | When to use |
| - | - |
| `--gst-debug=2` | Production runs — quiet, errors/warnings only |
| `--gst-debug=2,qtimltflite:5` | Prints all ERROR and WARNING messages globally, plus ERROR, WARNING, INFO, and DEBUG messages specifically from the qtimltflite plugin |
| `--gst-debug=2,qtimlvtransform:4,qtivcomposer:7` | Prints all ERROR and WARNING messages globally, ERROR, WARNING, and INFO messages from qtimlvtransform, and ERROR, WARNING, INFO, DEBUG, LOG, and TRACE messages from qtivcomposer |

### Python / C++ Apps

```bash theme={null}
GST_DEBUG=3 python3 my_pipeline.py
```

```bash theme={null}
GST_DEBUG=5 ./my_pipeline_app
```

### Redirect to File

```bash theme={null}
GST_DEBUG=5 GST_DEBUG_FILE=/data/gst.log GST_DEBUG_COLOR_MODE=off gst-launch-1.0 ...
```

***

## 2. Measure Pipeline KPIs (Latency per Stage)

Use the built-in GStreamer **latency tracer** to get per-element mean/min/max processing time for every stage — no custom instrumentation required.

### Enable Tracing

```bash theme={null}
GST_TRACERS="latency(flags=pipeline+element)" \
GST_DEBUG="GST_TRACER:7" \
gst-launch-1.0 -e \
  qtimlvconverter name=stage_01_preproc \
  qtimltflite name=stage_01_inference \
    model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME \
    delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
    external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" \
  qtimlpostprocess name=stage_01_postproc module=yolov8 \
    labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME \
    settings="{\"confidence\": 51.0}" \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
  v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
  tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! \
    fpsdisplaysink name=display sync=false text-overlay=false \
      video-sink="waylandsink" \
  t. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! \
  stage_01_inference. stage_01_inference. ! queue ! \
  stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! obj_mux. \
2> /tmp/trace.log
```

### Parse Results

```bash theme={null}
gst-stats-1.0 /tmp/trace.log
```

### Reading the Output

```
Element Latency Statistics:
  stage_01_preproc.src:   mean=2.19ms  min=1.24ms  max=12.2ms   ← preprocessing
  stage_01_inference.src: mean=5.71ms  min=4.95ms  max=6.78ms   ← inference (NPU)
  stage_01_postproc.src:  mean=0.24ms  min=0.12ms  max=0.44ms   ← postprocessing
  obj_mux.src:            mean=0.76ms  min=0.14ms  max=2.03ms   ← metadata sync
```

<Note>
  At 30 fps (33 ms budget per frame), the typical AI pipeline uses \~37% of the frame budget, leaving headroom for additional stages.
</Note>

### Measure FPS Live (On-Screen)

Add `fpsdisplaysink` in place of `waylandsink` to see FPS on the display in real time:

```bash theme={null}
... ! qtivoverlay ! fpsdisplaysink sync=false video-sink="waylandsink sync=false"
```

***

## 3. Measure Inference Performance

### GST\_TRACER (inferences/sec)

From the [tracer output](/advanced/debugging#enable-tracing), calculate inferences/sec:

```
inferences/sec = 1 / mean_inference_latency
```

Example: `mean = 5.71 ms` → `1 / 0.00571 ≈ 175 inferences/sec`

### pidstat (CPU/Memory alongside inference)

```bash theme={null}
# Run in a second terminal while the pipeline is running
pidstat -u -r -p $(pgrep gst-launch-1.0) 1
```

This reports per-second CPU (`%usr`, `%sys`) and memory (`RSS`, `VSZ`) for the pipeline process.

***

## 4. Debug Performance Issues

### Frame Drops

Frame drops manifest as:

* Stuttering on the display
* `queue` elements reporting overflows in debug logs (`--gst-debug=4`)
* Tracer showing high `max` latency on inference (`stage_01_inference`)

**Check for queue overflows:**

```bash theme={null}
gst-launch-1.0 --gst-debug=4,queue:5 ...
```

Look for lines like:

```
WARN   queue  gstqueue.c:... overrun ...
```

**Fix:** Add `max-size-time=0 max-size-bytes=0 max-size-buffers=3` to queues around inference:

```bash theme={null}
... ! queue max-size-buffers=3 ! stage_01_preproc. ...
```

### Slow Throughput

If end-to-end latency is higher than expected:

1. Run the [latency tracer](/advanced/debugging#2-measure-pipeline-kpis-latency-per-stage) to find which stage is the bottleneck.
2. Check if the inference delegate is correct — GPU delegate is much slower than HTP (NPU):

```bash theme={null}
# Slow — GPU
qtimltflite delegate=gpu ...

# Fast — NPU via HTP
qtimltflite delegate=external \
  external-delegate-path=libQnnTFLiteDelegate.so \
  external-delegate-options="QNNExternalDelegate,backend_type=htp;"
```

3. Check model quantization — W8A8 (quantized) is significantly faster on the NPU than float32.

### Memory Copies

Unnecessary buffer copies add latency. To detect them, enable memory tracing:

```bash theme={null}
GST_DEBUG=4,GST_BUFFER:5 gst-launch-1.0 ... 2>&1 | grep -i "copy\|memcpy\|alloc"
```

Use `capture-io-mode=4 output-io-mode=4` on decoder/encoder elements to enable zero-copy DMA buffer passing:

```bash theme={null}
v4l2h264dec capture-io-mode=4 output-io-mode=4
```

***

## 5. Trace Buffers

Buffer tracing lets you follow a specific buffer through the pipeline to diagnose where it is dropped, delayed, or corrupted.

### Enable Buffer Tracing

```bash theme={null}
GST_DEBUG=GST_SCHEDULING:5 gst-launch-1.0 ... 2>&1 | head -200
```

Each line shows a buffer's `PTS` (presentation timestamp), `DTS`, `size`, and which pad it passed through.

### Check for PTS Discontinuities

If frames are out of order or the pipeline stalls, look for timestamp jumps:

```bash theme={null}
GST_DEBUG=GST_SCHEDULING:5 gst-launch-1.0 ... 2>&1 | grep -i "pts\|discont\|gap"
```

### Use the Dot Graph (Pipeline Visualization)

Dump the pipeline as a `.dot` file to visually inspect element connections:

```bash theme={null}
GST_DEBUG_DUMP_DOT_DIR=/tmp gst-launch-1.0 -e ... &
# After pipeline is running:
dot -Tpng /tmp/*.dot -o pipeline.png
```

***

## 6. Debug Camera Issues

### Common Errors

**QMMF Recorder StartCamera Failed:**

```
ERROR qticamsrc qmmf_source_context.cc: QMMF Recorder StartCamera Failed!
WARN  qticamsrc qmmf_source.c: error: Failed to Open Camera!
```

**Fix:** Kill and restart the camera server:

```bash theme={null}
sudo killall cam-server
```

Then re-run the pipeline. The camera server restarts automatically on the next `gst-launch-1.0` invocation.

### Enable Camera Server Debug Logs

```bash theme={null}
GST_DEBUG=2,qticamsrc:6 gst-launch-1.0 \
  qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
  waylandsink fullscreen=true sync=false
```

### Camera Not Detected

```bash theme={null}
# List available V4L2 devices
v4l2-ctl --list-devices

# Check camera capabilities
v4l2-ctl -d /dev/video0 --all
```

### ISP Camera (Config #2 / qticamsrc)

To switch from libcamera (Config #1) to `qticamsrc` (Config #2), write `"camx"` to the EFI variable:

```bash theme={null}
echo -n "camx" > /var/data
efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -w -f /var/data
sync && reboot
```

### Verify Camera Stream is Running

```bash theme={null}
# Quick sanity check — should show frames on display
gst-launch-1.0 \
  qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
  waylandsink fullscreen=true sync=false
```

***

## 7. Quick Diagnostic Checklist

Run this sequence when a pipeline doesn't work:

```bash theme={null}
# Step 1 — Check GStreamer can see the plugins
gst-inspect-1.0 qtimltflite
gst-inspect-1.0 qtimlvconverter
gst-inspect-1.0 qticamsrc

# Step 2 — Run with level 3 logs
gst-launch-1.0 --gst-debug=3 [your pipeline] 2>&1 | grep -E "ERROR|WARN|FIXME"

# Step 3 — Check camera server
sudo killall cam-server   # reset if camera errors appear

# Step 4 — Add latency tracing
GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG="GST_TRACER:7" \
  gst-launch-1.0 [your pipeline] 2> /tmp/trace.log
gst-stats-1.0 /tmp/trace.log
```

***

## 8. Report an Issue

When filing a bug or asking for help, include:

1. **Platform** — Device model (e.g. RB3 Gen 2), OS (Ubuntu / Qualcomm Linux), QIM SDK version
2. **Full pipeline command** — the exact `gst-launch-1.0` or Python script
3. **Debug log** — captured with `--gst-debug=5 2> debug.log`
4. **Tracer output** — from `gst-stats-1.0` if a performance issue
5. **Error message** — the exact error line from the log
6. **Expected vs actual behaviour**

File the issue through the [Qualcomm Support](https://www.qualcomm.com/support) portal.

```bash theme={null}
# Collect everything in one command
GST_DEBUG=5 GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG_FILE=/tmp/debug.log \
  gst-launch-1.0 [your pipeline]
gst-stats-1.0 /tmp/debug.log >> /tmp/debug.log
# Attach /tmp/debug.log to your issue
```


# QIM SDK Docker
Source: https://imsdkdocs.qualcomm.com/advanced/docker

Build, configure, and run the QIM SDK Docker container on Qualcomm platforms.

## Overview

Containers are lightweight, isolated environments that package everything needed to run an application — code, dependencies, and system libraries — so it behaves consistently across platforms.

Advantages of containerization include:

* **Consistency:** Containers provide a consistent environment across multiple platforms and operating systems. Developers can create images that transform into containers when deployed, ensuring uniformity regardless of the underlying infrastructure.
* **Automation:** Containerization supports automated tasks. Developers can schedule actions without manual intervention, reducing repetitive work.
* **Stability:** Containers are based on Linux, with the Linux kernel present in every container. This stability extends to systems and devices, including when containers run on macOS or Windows.
* **Space efficiency:** Unlike traditional virtual machines, which consume significant server space and memory, containers include only application code and dependencies. They can run entirely in the cloud and minimize resource requirements.

<Frame>
  <img alt="Container architecture overview" />
</Frame>

## QIM SDK in Docker

QIM SDK in a Docker container means the full Qualcomm multimedia and AI software stack is delivered as containerized environments instead of being manually installed directly on a host or target system.

In practice, a build container compiles and assembles all required components, including cross-compiled artifacts for Qualcomm ARM64 targets. The build then produces a deploy container that contains only the runtime pieces needed on the device.

**Key benefits of this approach:**

* **Reproducible builds:** Use the same toolchain, package versions, and scripts across developers and CI. Reduces "works on my machine" failures.
* **Build and runtime separation:** Use a heavy build image to compile artifacts and a lean deploy image that contains only runtime essentials.
* **Efficient cross-compilation:** Build on the host and target ARM64 Qualcomm devices reliably, faster than compiling directly on the target.
* **Secure runtime posture:** Run as a non-root user by default, avoid broad privileged-container dependencies, and use CDI-based explicit hardware passthrough for GPU, camera, and media devices.
* **Operational consistency:** Use versioned images that are testable, promotable, and rollback-friendly, with a predictable deployment flow using environment and CDI configuration on the target.

### Features

* **Debian (Trixie) base:** Built on Debian Trixie for a streamlined and lightweight foundation.
* **No platform dependencies:** Entirely self-sufficient — no eSDK or host platform content required to build or deploy.
* **Self-contained dependencies:** All necessary dependencies are installed directly inside the container, minimizing host-container compatibility issues and simplifying deployment on edge devices.
* **GStreamer 1.26:** Includes GStreamer 1.26 with the latest `gst-qti-oss-patches` for optimized performance on Qualcomm platforms.
* **Broad target support:** Validated across QLI 2.x, Ubuntu Server, and Ubuntu Desktop. Supports IQ615, QCS6490, IQ8275, IQ9075, IQ-X5121 and IQ-X7181 platforms.

### Versions

| Version | Status |
| - | - |
| 2.1.0 | Latest |

This version of image targets only the following platforms. Support for more platforms & OS will be added in upcoming releases.

<Tabs>
  <Tab title="QLI 2.x">
    * QCS6490
    * IQ-9075
    * IQ-8275
    * IQ-615
    * IQ-X7181
    * IQ-X5121
  </Tab>

  <Tab title="Ubuntu 24.04">
    * QCS6490
    * IQ-9075
    * IQ-8275
  </Tab>

  <Tab title="Ubuntu 26.04">
    * IQ-X7181
    * IQ-X5121
  </Tab>
</Tabs>

## Architecture

The following diagram explains the build and deployment flow of the QIM SDK Docker image.

<Frame>
  <img alt="QIM SDK Docker build and deployment flow" />
</Frame>

* The host machine runs a development container that compiles all QIM SDK sources and dependencies into Debian artifacts.
* The generated artifacts are transferred to a Debian-based deploy container with CDI-managed hardware access that runs on the Qualcomm device.

## Quick Start Guide

<Note>
  Before getting started, ensure the device is set up by following the [Installation](/installation) guide.
</Note>

<Warning>
  Camera is not supported in the container.
</Warning>

### Download configuration files

Before setting up the device, download the CDI and environment files for the target chipset and OS. The CDI JSON file provides platform-specific device mountings; the environment file sets the required variables inside the container.

Select your chipset to download the corresponding files:

<Tabs>
  <Tab title="QCS6490">
    | Target OS | CDI file | Environment file |
    | - | - | - |
    | QLI 2.x | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/qcs6490_qli_2x_qimsdk.json?inline=false">qcs6490\_qli\_2x\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/qcs6490_qli_2x_qimsdk.env?inline=false">qcs6490\_qli\_2x\_qimsdk.env</a> |
    | Ubuntu 24.04 Server | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/qcs6490_ubuntu_server_qimsdk.json?inline=false">qcs6490\_ubuntu\_server\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/qcs6490_ubuntu_server_qimsdk.env?inline=false">qcs6490\_ubuntu\_server\_qimsdk.env</a> |
    | Ubuntu 24.04 Desktop | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/qcs6490_ubuntu_desktop_qimsdk.json?inline=false">qcs6490\_ubuntu\_desktop\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/qcs6490_ubuntu_desktop_qimsdk.env?inline=false">qcs6490\_ubuntu\_desktop\_qimsdk.env</a> |
  </Tab>

  <Tab title="IQ-8275">
    | Target OS | CDI file | Environment file |
    | - | - | - |
    | QLI 2.x | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/qcs8275_qli_2x_qimsdk.json?inline=false">qcs8275\_qli\_2x\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/qcs8275_qli_2x_qimsdk.env?inline=false">qcs8275\_qli\_2x\_qimsdk.env</a> |
    | Ubuntu 24.04 Server | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/qcs8275_ubuntu_server_qimsdk.json?inline=false">qcs8275\_ubuntu\_server\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/qcs8275_ubuntu_server_qimsdk.env?inline=false">qcs8275\_ubuntu\_server\_qimsdk.env</a> |
    | Ubuntu 24.04 Desktop | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/qcs8275_ubuntu_desktop_qimsdk.json?inline=false">qcs8275\_ubuntu\_desktop\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/qcs8275_ubuntu_desktop_qimsdk.env?inline=false">qcs8275\_ubuntu\_desktop\_qimsdk.env</a> |
  </Tab>

  <Tab title="IQ-9075">
    | Target OS | CDI file | Environment file |
    | - | - | - |
    | QLI 2.x | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/qcs9075_qli_2x_qimsdk.json?inline=false">qcs9075\_qli\_2x\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/qcs9075_qli_2x_qimsdk.env?inline=false">qcs9075\_qli\_2x\_qimsdk.env</a> |
    | Ubuntu 24.04 Server | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/qcs9075_ubuntu_server_qimsdk.json?inline=false">qcs9075\_ubuntu\_server\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/qcs9075_ubuntu_server_qimsdk.env?inline=false">qcs9075\_ubuntu\_server\_qimsdk.env</a> |
    | Ubuntu 24.04 Desktop | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/qcs9075_ubuntu_desktop_qimsdk.json?inline=false">qcs9075\_ubuntu\_desktop\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/qcs9075_ubuntu_desktop_qimsdk.env?inline=false">qcs9075\_ubuntu\_desktop\_qimsdk.env</a> |
  </Tab>

  <Tab title="IQ-615">
    | Target OS | CDI file | Environment file |
    | - | - | - |
    | QLI 2.x | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/qcs615_qli_2x_qimsdk.json?inline=false">qcs615\_qli\_2x\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/qcs615_qli_2x_qimsdk.env?inline=false">qcs615\_qli\_2x\_qimsdk.env</a> |
  </Tab>

  <Tab title="IQ-X7181">
    | Target OS | CDI file | Environment file |
    | - | - | - |
    | QLI 2.x | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/iqx7181_qli_2x_qimsdk.json?inline=false">iqx7181\_qli\_2x\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/iqx7181_qli_2x_qimsdk.env?inline=false">iqx7181\_qli\_2x\_qimsdk.env</a> |
    | Ubuntu 26.04 Server | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/iqx7181_ubuntu_server_qimsdk.json?inline=false">iqx7181\_ubuntu\_server\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/iqx7181_ubuntu_server_qimsdk.env?inline=false">iqx7181\_ubuntu\_server\_qimsdk.env</a> |
    | Ubuntu 26.04 Desktop | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/iqx7181_ubuntu_desktop_qimsdk.json?inline=false">iqx7181\_ubuntu\_desktop\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/iqx7181_ubuntu_desktop_qimsdk.env?inline=false">iqx7181\_ubuntu\_desktop\_qimsdk.env</a> |
  </Tab>

  <Tab title="IQ-X5121">
    | Target OS | CDI file | Environment file |
    | - | - | - |
    | QLI 2.x | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/iqx5121_qli_2x_qimsdk.json?inline=false">iqx5121\_qli\_2x\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/iqx5121_qli_2x_qimsdk.env?inline=false">iqx5121\_qli\_2x\_qimsdk.env</a> |
    | Ubuntu 26.04 Server | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/iqx5121_ubuntu_server_qimsdk.json?inline=false">iqx5121\_ubuntu\_server\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/iqx5121_ubuntu_server_qimsdk.env?inline=false">iqx5121\_ubuntu\_server\_qimsdk.env</a> |
    | Ubuntu 26.04 Desktop | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/cdi/iqx5121_ubuntu_desktop_qimsdk.json?inline=false">iqx5121\_ubuntu\_desktop\_qimsdk.json</a> | <a href="https://git.codelinaro.org/clo/le/sdk-tools/-/raw/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/env/iqx5121_ubuntu_desktop_qimsdk.env?inline=false">iqx5121\_ubuntu\_desktop\_qimsdk.env</a> |
  </Tab>
</Tabs>

### Set up and run on the device

<Tabs>
  <Tab title="QLI 2.x">
    <Steps>
      <Step title="SSH into the device">
        ```bash theme={null}
        ssh <user>@<device-ip>
        ```
      </Step>

      <Step title="Create required directories">
        ```bash theme={null}
        mkdir -p /etc/cdi
        mkdir -p /etc/docker/env
        mkdir -p $HOME/Downloads/qimsdk_samples

        mkdir -p $HOME/Downloads/qimsdk_samples/media
        mkdir -p $HOME/Downloads/qimsdk_samples/models
        mkdir -p $HOME/Downloads/qimsdk_samples/labels
        ```
      </Step>

      <Step title="Copy the downloaded configuration files to the device">
        ```bash theme={null}
        scp <downloaded-cdi-file>.json <user>@<device-ip>:/etc/cdi/qimsdk.json
        scp <downloaded-env-file>.env <user>@<device-ip>:/etc/docker/env/qimsdk.env
        ```

        <AccordionGroup>
          <Accordion title="Additional step to configure nodes for video driver and USB camera">
            On platforms such as iq615, iqx7181, and iqx8121, the Qualcomm downstream video drivers (`video32` and `video33`) are unavailable. Therefore, the upstream video driver is used. Since the upstream driver may be assigned to any device node between `/dev/video0` and `/dev/video28`, an additional configuration step is required to determine and configure the appropriate video nodes.

            Run the following command to list the video devices:

            ```bash theme={null}
            v4l2-ctl --list-devices
            ```

            Example:

            ```bash theme={null}
                Qualcomm Venus video decoder (plat:aa00000.video-codec:dec):
                    /dev/video2

                Qualcomm Venus video encoder (plat:aa00000.video-codec:enc):
                    /dev/video3

                USB Camera (usb-0000:00:14.0-3):
                    /dev/video4
                    /dev/video5
            ```

            Identify the relevant video device nodes and add them to the cdi.json file that has been deployed to the target system before creating the container.

            ```
            "deviceNodes": [
              ...
              {
                "path": "/dev/video2",
                "uid": 0,
                "gid": 44
              },
              {
                "path": "/dev/video3",
                "uid": 0,
                "gid": 44
              },
              {
                "path": "/dev/video4",
                "uid": 0,
                "gid": 44
              },
              {
                "path": "/dev/video5",
                "uid": 0,
                "gid": 44
              },
              ...
            ]
            ```
          </Accordion>
        </AccordionGroup>
      </Step>

      <Step title="Pull the container image">
        <Tabs>
          <Tab title="Native Image">
            The QIM SDK native image is intended for native pipeline execution and includes all required runtime dependencies for QIM SDK applications. Python dependencies are intentionally excluded to keep the image lightweight and user can use the QIM SDK Python image for python support.

            ```bash theme={null}
            docker pull artifacts.codelinaro.org/iot-solutions-microservices/qimsdk:latest
            ```

            Tag the image locally:

            ```bash theme={null}
            docker tag artifacts.codelinaro.org/iot-solutions-microservices/qimsdk:latest qimsdk:latest
            ```
          </Tab>

          <Tab title="Python Image">
            The QIM SDK Python image includes all functionality available in the base image, along with Python support.

            ```bash theme={null}
            docker pull artifacts.codelinaro.org/iot-solutions-microservices/qimsdk-python:latest
            ```

            Tag the image locally:

            ```bash theme={null}
              docker tag artifacts.codelinaro.org/iot-solutions-microservices/qimsdk-python:latest qimsdk-python:latest
            ```
          </Tab>
        </Tabs>
      </Step>

      <Step title="Run the container">
        <AccordionGroup>
          <Accordion title="Additional step for SE-Linux enabled builds">
            An additional step is required for SE-Linux enabled builds to provide access to cdi, env & media artifacts.

            ```bash theme={null}
            # After pushing the CDI file to the device run the following command
            restorecon -Rv /etc/cdi

            # To get access to wayland and pipewire sockets.
            setsebool -P dockerd_connect_user_services on

            # To get access to the user Downloads directory
            setsebool -P dockerd_mounton_xdg_downloads on
            ```

            ```bash theme={null}
            # Run the following commands whenever new artifacts are uploaded to the paths below.
            chcon -t xdg_downloads_t /var/roothome/Downloads
            chcon -R -t xdg_downloads_t /var/roothome/Downloads/qimsdk_samples
            ```
          </Accordion>
        </AccordionGroup>

        ```bash theme={null}
        # The following docker run command mounts $HOME/Downloads/qimsdk_samples
        # into the container for media assets and application artifacts.
        # Using this path is recommended to avoid potential SELinux permission issues.

        docker run -it -d \
            --net host \
            --env-file /etc/docker/env/qimsdk.env \
            -v $HOME/Downloads/qimsdk_samples:/home/qimsdk/Downloads/qimsdk_samples \
            --device qualcomm.com/device=qimsdk \
            -h <host_name> \
            --name <desired-container-name> \
            <generated-image-name>
        ```

        After the container is running, execute a new terminal process in it:

        ```bash theme={null}
        docker exec -ti <desired-container-name> bash
        ```
      </Step>
    </Steps>
  </Tab>

  <Tab title="Ubuntu">
    <Steps>
      <Step title="SSH into the device">
        ```bash theme={null}
        ssh <user>@<device-ip>
        ```
      </Step>

      <Step title="Install docker">
        ```bash theme={null}
        sudo apt update
        sudo apt install docker.io
        sudo usermod -aG docker $USER
        # Log out and log back in for the group membership changes to take effect.
        ```
      </Step>

      <Step title="Create required directories">
        ```bash theme={null}
        sudo mkdir -p /etc/cdi
        sudo mkdir -p /etc/docker/env
        mkdir -p $HOME/Downloads/qimsdk_samples

        mkdir -p $HOME/Downloads/qimsdk_samples/media
        mkdir -p $HOME/Downloads/qimsdk_samples/models
        mkdir -p $HOME/Downloads/qimsdk_samples/labels

        ```
      </Step>

      <Step title="Copy the downloaded configuration files to the device">
        ```bash theme={null}
        scp <downloaded-cdi-file>.json <user>@<device-ip>:/etc/cdi/qimsdk.json
        scp <downloaded-env-file>.env <user>@<device-ip>:/etc/docker/env/qimsdk.env
        ```

        <AccordionGroup>
          <Accordion title="Additional step to configure nodes for video driver and USB camera">
            On platforms such as iq615, iqx7181, and iqx8121, the Qualcomm downstream video drivers (`video32` and `video33`) are unavailable. Therefore, the upstream video driver is used. Since the upstream driver may be assigned to any device node between `/dev/video0` and `/dev/video28`, an additional configuration step is required to determine and configure the appropriate video nodes.

            Run the following command to list the video devices:

            ```bash theme={null}
              sudo apt install v4l-utils
              v4l2-ctl --list-devices
            ```

            Example:

            ```bash theme={null}
                Qualcomm Venus video decoder (plat:aa00000.video-codec:dec):
                    /dev/video2

                Qualcomm Venus video encoder (plat:aa00000.video-codec:enc):
                    /dev/video3

                USB Camera (usb-0000:00:14.0-3):
                    /dev/video4
                    /dev/video5
            ```

            Identify the relevant video device nodes and add them to the cdi.json file that has been deployed to the target system before creating the container.

            ```
             "deviceNodes": [
               ...
               {
                 "path": "/dev/video2",
                 "uid": 0,
                 "gid": 44
               },
               {
                 "path": "/dev/video3",
                 "uid": 0,
                 "gid": 44
               },
               {
                 "path": "/dev/video4",
                 "uid": 0,
                 "gid": 44
               },
               {
                 "path": "/dev/video5",
                 "uid": 0,
                 "gid": 44
               },
               ...
             ]
            ```
          </Accordion>
        </AccordionGroup>
      </Step>

      <Step title="Pull the container image">
        <Tabs>
          <Tab title="Native Image">
            The QIM SDK native image is intended for native pipeline execution and includes all required runtime dependencies for QIM SDK applications. Python dependencies are intentionally excluded to keep the image lightweight and user can use the QIM SDK Python image for python support.

            ```bash theme={null}
            docker pull artifacts.codelinaro.org/iot-solutions-microservices/qimsdk:latest
            ```

            Tag the image locally:

            ```bash theme={null}
            docker tag artifacts.codelinaro.org/iot-solutions-microservices/qimsdk:latest qimsdk:latest
            ```
          </Tab>

          <Tab title="Python Image">
            The QIM SDK Python image includes all functionality available in the base image, along with Python support.

            ```bash theme={null}
            docker pull artifacts.codelinaro.org/iot-solutions-microservices/qimsdk-python:latest
            ```

            Tag the image locally:

            ```bash theme={null}
            docker tag artifacts.codelinaro.org/iot-solutions-microservices/qimsdk-python:latest qimsdk-python:latest
            ```
          </Tab>
        </Tabs>
      </Step>

      <Step title="Run the container">
        ```bash theme={null}
        # The following docker run command mounts $HOME/Downloads/qimsdk_samples
        # into the container for media assets and application artifacts.
        # Using this path is recommended to avoid potential SELinux permission issues.

        # Automatically detect and pass the active Wayland display to the container,
        # as the WAYLAND_DISPLAY name may vary across systems and display configurations

        sudo docker run -it -d \
          --net host \
          -e WAYLAND_DISPLAY=$(basename \
              $(ls ${XDG_RUNTIME_DIR}/wayland-* 2>/dev/null | head -n1) \
              2>/dev/null) \
          --env-file /etc/docker/env/qimsdk.env \
          -v $HOME/Downloads/qimsdk_samples:/home/qimsdk/Downloads/qimsdk_samples \
          --device qualcomm.com/device=qimsdk \
          -h <host_name> \
          --name <desired-container-name> \
          <generated-image-name>
        ```

        After the container is running, execute a new terminal process in it:

        ```bash theme={null}
        sudo docker exec -ti <desired-container-name> bash
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

Once inside the container, refer to the **Examples** section to run

* [AI pipelines](/sample-pipelines/aipipelines)
* [multimedia pipelines](/sample-pipelines/multimediapipelines)
* [GStreamer C/C++ AI applications](/sample-application/ai-sample-applications)
* [GStreamer C/C++ multimedia applications](/sample-application/multimedia-sample-applications)
* [C++ App Builder](/app-builder/cpp-quickstart)
* [Python App Builder](/app-builder/python-quickstart)

### Running GStreamer Sample applications inside container

By default, sample application configuration files are installed under `/etc/configs`. However, this location cannot be mounted into the container due to SELinux restrictions.

On the platform, copy the configuration files to `$HOME/Downloads/qimsdk_samples/configs`. Because `$HOME/Downloads/qimsdk_samples` is mounted into the container by the previous docker run command, the `configs` directory will be accessible from within the container.

```bash theme={null}
mkdir -p $HOME/Downloads/qimsdk_samples/configs
cp /etc/configs/* $HOME/Downloads/qimsdk_samples/configs
chmod -R 0777 $HOME/Downloads/qimsdk_samples/configs
```

Inside the container, use `--config-file` argument to pass config file to sample app:

```bash theme={null}
gst-ai-classification \
    --config-file=$HOME/Downloads/qimsdk_samples/configs/config_classification.json
```

## Build an QIM SDK container

<Note>
  All steps in this section are performed on a Ubuntu 22.04 or 24.04 machine which can be x86 or ARM64.
</Note>

### Prerequisites

<AccordionGroup>
  <Accordion title="Ubuntu packages">
    Install prerequisite packages on the host (one time only).

    <Tabs>
      <Tab title="ARM64">
        ```bash theme={null}
        sudo apt install -y jq tofrodos
        sudo wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/bin/yq
        sudo chmod +x /usr/bin/yq
        ```
      </Tab>

      <Tab title="x86">
        ```bash theme={null}
        sudo apt install -y jq tofrodos qemu-user-static qemu-system-arm
        sudo wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/bin/yq
        sudo chmod +x /usr/bin/yq
        wget http://archive.ubuntu.com/ubuntu/pool/universe/q/qemu/qemu-user-static_6.2+dfsg-2ubuntu6_amd64.deb
        sudo dpkg -i qemu-user-static_6.2+dfsg-2ubuntu6_amd64.deb
        rm qemu-user-static_6.2+dfsg-2ubuntu6_amd64.deb
        ```
      </Tab>
    </Tabs>

    <Warning>
      Do **not** install `yq` via snap. If it was installed via snap, remove it and reinstall using the commands above:

      ```bash theme={null}
      sudo snap remove yq
      sudo systemctl stop snapd
      ```
    </Warning>
  </Accordion>

  <Accordion title="File watcher limits">
    Increase the maximum user watches and instances on the host to prevent this error during development:

    ```text theme={null}
    System limit for number of file watchers reached
    ```

    Add these lines to `/etc/sysctl.conf`:

    ```text theme={null}
    fs.inotify.max_user_instances=8192
    fs.inotify.max_user_watches=542288
    ```

    Save the file and reboot the host system.
  </Accordion>

  <Accordion title="Swap image">
    To build Docker image files, the host system requires at least 64 GB of RAM and a swap image of at least half the available RAM plus a small reserve.

    For example, with 64 GB of RAM, the recommended swap size is at least 32 GB. With an 8 GB reserve, total RAM + swap + reserved memory is 104 GB.

    Check whether swap is enabled:

    ```bash theme={null}
    sudo swapon --show
    ```

    If swap is missing or too small, create a new swap file:

    ```bash theme={null}
    MEM_AVAIL=$(grep MemAvailable /proc/meminfo | awk '{printf "%.0f\n", $2/1024/1024}')
    let MEM_SWAP="$MEM_AVAIL / 2 + 8"
    sudo swapoff /swap.img
    sudo fallocate -l "${MEM_SWAP}G" /swap.img
    sudo chmod 600 /swap.img
    sudo mkswap /swap.img
    sudo swapon /swap.img
    ```

    Edit `/etc/default/grub` and set:

    ```text theme={null}
    GRUB_CMDLINE_LINUX_DEFAULT="text cgroup_enable=memory swapaccount=1"
    ```

    Update GRUB:

    ```bash theme={null}
    sudo update-grub
    ```

    Edit `/etc/fstab` and add this line at the end of the file:

    ```text theme={null}
    /swap.img none swap sw 0 0
    ```

    Save and close all opened system files, then reboot the host system.

    After rebooting, verify the swap image matches the expected size:

    ```bash theme={null}
    sudo swapon --show
    ```
  </Accordion>

  <Accordion title="Configure Docker">
    Docker must be configured on the host system once before building images.

    <AccordionGroup>
      <Accordion title="Remove old versions">
        Remove old Docker Desktop versions if present:

        ```bash theme={null}
        sudo apt remove docker-desktop
        rm -r ~/.docker/desktop
        sudo rm /usr/local/bin/com.docker.cli
        sudo apt purge docker-desktop
        ```
      </Accordion>

      <Accordion title="Set up the Docker repository">
        Install repository dependencies:

        ```bash theme={null}
        sudo apt-get update
        sudo apt-get install \
          ca-certificates \
          curl \
          gnupg \
          lsb-release
        ```

        Create the Docker keyring directory:

        ```bash theme={null}
        sudo mkdir -p /etc/apt/keyrings
        ```

        Add the Docker GPG key:

        ```bash theme={null}
        curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
        ```

        Add the Docker repository:

        ```bash theme={null}
        echo \
          "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
          $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
        ```
      </Accordion>

      <Accordion title="Install Docker Engine">
        ```bash theme={null}
        sudo apt-get update
        sudo apt-get install docker-ce docker-ce-cli
        ```
      </Accordion>

      <Accordion title="Add user to the Docker group">
        Add the user to the Docker group to run Docker commands without `sudo`:

        ```bash theme={null}
        sudo groupadd docker
        sudo usermod -aG docker $USER
        newgrp docker
        ```

        <Note>
          Until the host is rebooted, run `newgrp docker` in every newly opened terminal.
        </Note>
      </Accordion>

      <Accordion title="Add a registry mirror (optional)">
        <Warning>
          Tabs, unexpected spaces, and other invisible whitespace characters can break JSON configuration files and may cause a `docker.service failed to start` error.
        </Warning>

        Add the registry mirror configuration to `/etc/docker/daemon.json`:

        ```json theme={null}
        {
          "registry-mirrors": ["<docker-registry-mirror-url>"]
        }
        ```

        Restart Docker:

        ```bash theme={null}
        sudo fromdos -f /etc/docker/daemon.json
        sudo systemctl daemon-reload
        sudo systemctl restart docker
        ```
      </Accordion>

      <Accordion title="Configure proxy (optional)">
        If network downloads require a proxy, add the proxy configuration to the Docker daemon.

        <Warning>
          Tabs, unexpected spaces, and other invisible whitespace characters can break JSON configuration files and may cause a `docker.service failed to start` error.
        </Warning>

        Add the proxy configuration to `/etc/docker/daemon.json`:

        ```json theme={null}
        {
          "proxies": {
            "http-proxy": "<http-proxy-url>",
            "https-proxy": "<https-proxy-url>",
            "no-proxy": "<no-proxy-url>"
          }
        }
        ```

        Restart Docker:

        ```bash theme={null}
        sudo fromdos -f /etc/docker/daemon.json
        sudo systemctl daemon-reload
        sudo systemctl restart docker
        ```

        Set proxy environment variables before invoking Docker build commands:

        ```bash theme={null}
        export http_proxy=<http-proxy-url>
        export https_proxy=<https-proxy-url>
        export no_proxy=<no-proxy-url>
        ```

        This ensures that both the Docker daemon and build processes use the same proxy settings as the host system.

        <Note>
          Until the host is rebooted, run `newgrp docker` in every newly opened terminal.
        </Note>
      </Accordion>

      <Accordion title="Test the Docker setup (optional)">
        Run the Docker hello-world container:

        ```bash theme={null}
        docker run hello-world
        ```
      </Accordion>

      <Accordion title="Move the Docker directory (optional)">
        If the build image script fails with a `No space left on device` error, move the Docker directory to `/local/mnt`.

        <Warning>
          All containers must be restarted after moving the Docker directory because the Docker service is stopped during this procedure.
        </Warning>

        Stop Docker:

        ```bash theme={null}
        service docker stop
        ```

        Verify that no Docker process is running:

        ```bash theme={null}
        ps faux
        ```

        Check the Docker directory structure:

        ```bash theme={null}
        sudo ls /var/lib/docker/
        ```

        Back up the current Docker directory:

        ```bash theme={null}
        tar -zcC /var/lib docker > /mnt/pd0/var_lib_docker-backup-$(date +%s).tar.gz
        ```

        Move the Docker directory to the new partition:

        ```bash theme={null}
        mv /var/lib/docker /local/mnt/docker
        ```

        Create a symlink to the Docker directory in the new partition:

        ```bash theme={null}
        ln -s /local/mnt/docker /var/lib/docker
        ```

        Verify that the Docker directory structure is unchanged:

        ```bash theme={null}
        sudo ls /var/lib/docker/
        ```

        Start Docker:

        ```bash theme={null}
        service docker start
        ```

        List container names:

        ```bash theme={null}
        docker ps -a
        ```

        Start a listed container:

        ```bash theme={null}
        docker start <container-name>
        ```
      </Accordion>
    </AccordionGroup>
  </Accordion>

  <Accordion title="Configure system memory">
    Check that there are no active cgroup CPU and memory utilization restrictions on the host. The Docker service cgroup memory configuration entries should have `max` set as the value.

    ```bash theme={null}
    sudo systemctl list-unit-files | grep -Ei docker
    sudo systemctl status docker.service
    cat /sys/fs/cgroup/system.slice/docker.service/
    cat /sys/fs/cgroup/system.slice/docker.service/memory.max
    cat /sys/fs/cgroup/system.slice/docker.service/memory.swap.max
    ```

    <AccordionGroup>
      <Accordion title="Tune VM settings (optional)">
        If the host system has 64 GB RAM or less, tune system VM page swap and dentry and inode cache reclaim behavior.

        **`vm.swappiness`**

        Higher values cause more aggressive application memory page swapping; lower values favor keeping pages in memory longer. The default is usually `60` on recent Ubuntu and Debian versions, which may not fit the Docker build environment on systems with low available RAM. A value of `10` is recommended.

        **`vm.vfs_cache_pressure`**

        Intermittent high VFS pressure can result from creating many small temporary files during package installation, causing the OS to prematurely swap memory pages associated with cached dentries and inodes. The default is usually `100`. Setting it to `400` relaxes cache management by freeing cached pages earlier, reducing the chance of running out of memory during high multi-threaded load.

        **`vm.min_free_kbytes`**

        In heavy multi-stage Docker builds with many buildx threads, the host OS can exhaust available RAM quickly. The mandatory VM free memory watermark is controlled by `vm.min_free_kbytes`. Too low risks deadlocks under load; too high causes premature OOM kills. A value of `262144` KB better balances free memory under excessive load.

        Edit `/etc/sysctl.conf` and add:

        ```text theme={null}
        vm.swappiness=10
        vm.vfs_cache_pressure=400
        vm.min_free_kbytes=262144
        ```
      </Accordion>
    </AccordionGroup>
  </Accordion>
</AccordionGroup>

### Build steps

<Steps>
  <Step title="Fetch the source code">
    Run the following commands on the host machine:

    ```bash theme={null}
    git clone https://git.codelinaro.org/clo/le/sdk-tools.git -b imsdk-tools.lnx.1.0
    cd sdk-tools/qimsdk-debian/
    git checkout fcc71f5f12791f77d8174cb5772eab6c785578d7
    ```
  </Step>

  <Step title="Build the image">
    <Tabs>
      <Tab title="Native image">
        Build `qimsdk_deploy_arm64`, which contains the minimal set of runtime binaries needed to execute GStreamer use cases:

        ```bash theme={null}
        docker build \
          --build-arg QIMSDK_ARG_QNP_VERSION=<QAIRT_SDK_VERSION> \
          --build-arg QIMSDK_ARG_CAMERA_SERVICE_TAG=<CAMERA_SERVICE_TAG_OR_COMMIT> \
          --build-arg QIMSDK_ARG_GST_PLUGINS_TAG=<QIMSDK_PLUGINS_TAG_OR_COMMIT>\
          --build-arg QIMSDK_ARG_SOLUTIONS_MICROSERVICES_TAG=<SOLUTIONS_MICROSERVICES_TAG_OR_COMIT> \
          --build-arg QIMSDK_ARG_GST_QTI_OSS_PATCHES_TAG=<GST_QTI_OSS_PATCHES_TAG_OR_COMMIT> \
          --build-arg QIMSDK_ARG_MAX_JOBS=<MAX_JOB_THREADS> \
          --target qimsdk_deploy_arm64 \
          -t qimsdk \
          .
        ```

        For example:
        Docker build command for `2.1.0` release

        ```bash theme={null}
          docker build \
            --build-arg QIMSDK_ARG_QNP_VERSION=2.50.0.260828 \
            --build-arg QIMSDK_ARG_CAMERA_SERVICE_TAG=1.0.6 \
            --build-arg QIMSDK_ARG_GST_PLUGINS_TAG=2.0.2 \
            --build-arg QIMSDK_ARG_SOLUTIONS_MICROSERVICES_TAG=805fd443c8ffe75eed16ca54a68602535c80e232 \
            --build-arg QIMSDK_ARG_GST_QTI_OSS_PATCHES_TAG=a4db6968a1f467875407a2008d7c13974a826fe4 \
            --build-arg QIMSDK_ARG_MAX_JOBS=8 \
            --target qimsdk_deploy_arm64 \
            -t qimsdk \
            .
        ```
      </Tab>

      <Tab title="Python image">
        Build `qimsdk_deploy_py_arm64`, which contains the minimal set of runtime binaries needed to execute GStreamer use cases along with python support:

        ```bash theme={null}
        docker build \
          --build-arg QIMSDK_ARG_QNP_VERSION=<QAIRT_SDK_VERSION> \
          --build-arg QIMSDK_ARG_CAMERA_SERVICE_TAG=<CAMERA_SERVICE_TAG_OR_COMMIT> \
          --build-arg QIMSDK_ARG_GST_PLUGINS_TAG=<QIMSDK_PLUGINS_TAG_OR_COMMIT>\
          --build-arg QIMSDK_ARG_SOLUTIONS_MICROSERVICES_TAG=<SOLUTIONS_MICROSERVICES_TAG_OR_COMIT> \
          --build-arg QIMSDK_ARG_GST_QTI_OSS_PATCHES_TAG=<GST_QTI_OSS_PATCHES_TAG_OR_COMMIT> \
          --build-arg QIMSDK_ARG_MAX_JOBS=<MAX_JOB_THREADS> \
          --target qimsdk_deploy_py_arm64 \
          -t qimsdk-python \
          .
        ```

        For example:
        Docker build command for `2.1.0` release

        ```bash theme={null}
          docker build \
            --build-arg QIMSDK_ARG_QNP_VERSION=2.50.0.260828 \
            --build-arg QIMSDK_ARG_CAMERA_SERVICE_TAG=1.0.6 \
            --build-arg QIMSDK_ARG_GST_PLUGINS_TAG=2.0.2 \
            --build-arg QIMSDK_ARG_SOLUTIONS_MICROSERVICES_TAG=805fd443c8ffe75eed16ca54a68602535c80e232 \
            --build-arg QIMSDK_ARG_GST_QTI_OSS_PATCHES_TAG=a4db6968a1f467875407a2008d7c13974a826fe4 \
            --build-arg QIMSDK_ARG_MAX_JOBS=8 \
            --target qimsdk_deploy_py_arm64 \
            -t qimsdk-python \
            .
        ```
      </Tab>
    </Tabs>

    #### Build Arguments

    | Argument | Description | Default |
    | - | - | - |
    | *QIMSDK\_ARG\_QNP\_VERSION* | Specifies the QAIRT SDK version to use. If not provided, the QNN and SNPE plugins are not built. | Not set |
    | *QIMSDK\_ARG\_CAMERA\_SERVICE\_TAG* | Commit ID or tag from the [Qualcomm Camera Service repository](https://github.com/qualcomm/camera-service) | `main` |
    | *QIMSDK\_ARG\_GST\_PLUGINS\_TAG* | Commit ID or tag from the [qimsdk](https://github.com/qualcomm/qimsdk) repository. | `main` |
    | *QIMSDK\_ARG\_SOLUTIONS\_MICROSERVICES\_TAG* | Commit ID or tag from the [solutions-microservices](https://git.codelinaro.org/clo/le/solutions-microservices/) repository. | `iot-solutions.lnx.1.0` |
    | *QIMSDK\_ARG\_GST\_QTI\_OSS\_PATCHES\_TAG* | Commit ID or tag from the [gst-qti-oss-patches](https://github.com/qualcomm/gst-qti-oss-patches) repository. | `1.26.2` |
    | *QIMSDK\_ARG\_MAX\_JOBS* | Limits the maximum number of parallel build jobs used during image creation. Valid values range from `1` to the number of available CPU cores (`nproc`). | All available CPU cores |

    <Info>
      See [qimsdk-debian/README.md](https://git.codelinaro.org/clo/le/sdk-tools/-/blob/fcc71f5f12791f77d8174cb5772eab6c785578d7/qimsdk-debian/README.md) for more information about the Docker image composition and workflow.
    </Info>
  </Step>

  <Step title="Save the image">
    Save the built image as a tar file:

    ```bash theme={null}
    docker save -o <image_name>.tar <generated-image-name>
    ```
  </Step>

  <Step title="Copy and load the image on the device">
    Copy the tar file to the device:

    ```bash theme={null}
    scp <image_name>.tar <user>@<device-ip>:~/Downloads/
    ```

    Load the image on the device:

    ```bash theme={null}
    docker load -i ~/Downloads/<image_name>.tar
    ```

    <Note>
      The home directory (\~) varies by platform:

      * QLI 2.x: /root
      * Ubuntu Noble: /home/ubuntu
      * Ubuntu Resolute: /home/qcom
    </Note>
  </Step>

  <Step title="Run the container on the device">
    With the image copied to the device, follow the [Set up and run on the device](#set-up-and-run-on-the-device) steps to configure CDI and start the container.
  </Step>
</Steps>

## Install additional packages inside the container

Use this procedure to install packages into an already running `qimsdk-debian` deploy container. The container must have been started with the `--net host` option. The following steps use vi as an example.

<Steps>
  <Step title="Execute a shell in the container as root">
    ```bash theme={null}
    docker exec -ti qimsdk sh
    ```
  </Step>

  <Step title="Install apt packages">
    In the root shell, install the required packages:

    ```bash theme={null}
    apt-get update
    DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC \
        apt-get install -y --no-install-recommends vi
    apt -y upgrade
    apt-get autoremove -y
    apt-get clean
    ```
  </Step>
</Steps>

After installing the vi package in the previous step, it can be launched from any new shell session started as the `qimsdk` user:

```bash theme={null}
docker exec -ti qimsdk bash
```

## Debugging

The following techniques apply to the QIM SDK container running on the Qualcomm device.

### Check container logs

<Steps>
  <Step title="Stream live logs">
    ```bash theme={null}
    docker logs -f <container-name>
    ```
  </Step>

  <Step title="Print the last 100 lines and follow">
    ```bash theme={null}
    docker logs --tail 100 -f <container-name>
    ```
  </Step>
</Steps>

### Inspect container state

<Steps>
  <Step title="List all containers and their status">
    ```bash theme={null}
    docker ps -a
    ```
  </Step>

  <Step title="Inspect container metadata">
    Inspect detailed container metadata including environment variables, mounts, and network settings:

    ```bash theme={null}
    docker inspect <container-name>
    ```
  </Step>
</Steps>

### Restart a container

<Steps>
  <Step title="Stop the container">
    ```bash theme={null}
    docker stop <container-name>
    ```
  </Step>

  <Step title="Start the container">
    ```bash theme={null}
    docker start <container-name>
    ```
  </Step>
</Steps>


# Technology Overview
Source: https://imsdkdocs.qualcomm.com/advanced/technologyoverview

Qualcomm Technology components used in by QIM SDK

# GStreamer plugin architecture

The QIM SDK encapsulates hardware complexity within a modular plugin architecture, freeing developers from the burden of managing low-level platform libraries or hardware-specific details that vary across Qualcomm chipsets and generations.

Each [QIM SDK plugin](../plugin-reference/introduction) in the SDK maps directly to a dedicated hardware accelerator — including video encode/decode, camera ISP, GPU, display, and AI/ML accelerators — giving developers a clean, unified API surface to build sophisticated multimedia and AI pipelines with minimal integration overhead

<img alt="Figure:Qualcomm QIM SDK GStreamer plugin architecture" />

## Camera Architecture

The QIM SDK abstracts the underlying camera driver and hardware through a client-server architecture, shielding developers from the complexities of low-level camera management and enabling rapid construction of single-stream, multi-stream, multi-client, and multi-camera applications.

At its core, the [`qticamsrc`](../plugin-reference/qticamsrc) GStreamer element operates as a client to the le-camera service — a system-level service responsible for managing the camera HAL, stream lifecycle, buffer allocation, and capture control. This separation of responsibilities allows [`qticamsrc`](../plugin-reference/qticamsrc) to surface the full camera capability set to the GStreamer pipeline through a developer-friendly interface of source pads, camera properties, and control signals.

At runtime, [`qticamsrc`](../plugin-reference/qticamsrc) performs the following on behalf of the pipeline:

* Opens and configures the target camera device
* Creates and manages video and image output streams
* Receives captured frames from the le-camera service
* Wraps camera buffers as GstBuffer objects and pushes them downstream through the corresponding source pads
* Returns buffers to the camera service upon downstream consumption

This architecture enables zero-copy camera output by retaining buffer ownership and stream resources within the service layer — maximizing pipeline throughput while minimizing memory overhead.

<img alt="Camera pipeline" />

### Related Information

[Camera use cases](../sample-pipelines/multimediapipelines#1-camera)

## AI Inference Architecture

The QIM SDK provides end-to-end support for AI/ML video and audio analytics pipelines, covering the full processing chain from raw media ingestion through model inference to real-time result visualization.

AI pipelines in QIM SDK are built around a tensor-based, three-stage processing model: preprocessing, inference, and postprocessing. Each stage is handled by a dedicated plugin and executes in parallel across consecutive frames — source capture at frame N-2, preprocessing at N-1, inference at N, and postprocessing at N+1 — delivering maximum pipeline throughput and optimal hardware utilization. The pipeline source can be live camera frames via [`qticamsrc`](../plugin-reference/qticamsrc) (YUV) or offline media via `filesrc` (with format conversion); the sink can be an on-screen display via [`waylandsink`](../plugin-reference/waylandsink) or local storage via `filesink`.

The [`qtimlvconverter`](../plugin-reference/qtimlvconverter) preprocessing plugin converts raw video frames into normalized tensors by performing color space conversion, resizing, and mean subtraction. It queries tensor dimensions and format requirements directly from the downstream inference plugin at runtime, making it a fully generic, model-agnostic preprocessing stage compatible with all inference backends. All inference plugins operate in tensor-in / tensor-out mode and share the same [`qtimlvconverter`](../plugin-reference/qtimlvconverter) for preprocessing. QIM SDK supports the following inference engines, each with hardware delegate support for acceleration across Qualcomm's neural processing subsystems:

| Plugin | Engine |
| - | - |
| [`qtimltflite`](../plugin-reference/qtimltflite) | LiteRT / TensorFlow Lite |
| [`qtimlsnpe`](../plugin-reference/qtimltflite) | Snapdragon Neural Processing Engine (SNPE) |
| [`qtimlqnn`](../plugin-reference/qtimlqnn) | Qualcomm Neural Network (QNN) |
| [`qtimlonnx`](../plugin-reference/qtimlonnx) | ONNX Runtime |

Inference output tensors are passed directly to task-specific postprocessing plugins that decode multi-dimensional tensor data — bounding boxes, class labels, segmentation masks, and keypoints — into structured ML metadata. The postprocessing layer follows a modular sub-module architecture, allowing developers to author custom postprocessing modules to support proprietary or non-standard model architectures. Finally, the [`qtivoverlay`](../plugin-reference/qtivoverlay) plugin consumes the structured ML metadata and renders inference results directly onto the video buffer in real time. The annotated stream can then be rendered to a display, encoded and stored locally, or streamed over a network.

<img alt="Figure:AI Inference Architecture" />

### Related information

[Run AI/ML use cases](../sample-pipelines/aipipelines)

## Video Architecture

The QIM SDK provides hardware-accelerated video encode and decode through V4L2-based plugins that interface directly with the Qualcomm multimedia engine. Video encoding and decoding are performed entirely on the dedicated VPU (Video Processing Unit), offloading this work from the CPU and allowing concurrent camera capture, AI inference, and video processing without resource contention.

The architecture supports AVC (H.264) and HEVC (H.265) formats for both encode and decode paths. Encoded output can be written to file via a multiplexer or streamed over the network. Decoded output is delivered as DMA-buf-backed NV12 frames, compatible with all downstream QIM SDK plugins including AI preprocessing and display.

### Encode

Video frames captured from the camera via [`qticamsrc`](../plugin-reference/qticamsrc) are passed directly to the hardware encoder. The encoded bitstream is parsed and multiplexed into an MP4 or MPEGTS container before being written to file or transmitted over the network.

<img alt="Video encode pipeline" />

| Component | Description |
| - | - |
| [`qticamsrc`](../plugin-reference/qticamsrc) | Captures video streams from the ISP camera. See [Camera Architecture](/advanced/technologyoverview#camera-architecture). |
| [`v4l2h264enc`](../plugin-reference/v4l2h264enc) | Encodes the video stream to AVC (H.264) format using the V4L2 driver. |
| [`v4l2h265enc`](../plugin-reference/v4l2h265enc) | Encodes the video stream to HEVC (H.265) format using the V4L2 driver. |
| `h264parse` / `h265parse` | Parses the encoded bitstream and inserts headers required for downstream muxing or streaming. |
| `mp4mux` / `mpegtsmux` | Multiplexes the encoded stream into an MP4 or MPEGTS container. |
| `filesink` | Writes the muxed container to the file system. |

### Decode

A video file is demultiplexed, the elementary stream is parsed, and the hardware decoder produces raw NV12 frames for display or downstream AI processing. Decode parameters are exposed as GStreamer element properties for direct application control.

<img alt="Video decode pipeline" />

| Component | Description |
| - | - |
| `filesrc` | Reads the video container from the file system. |
| `qtdemux` | Demultiplexes the container into separate elementary streams. |
| `h264parse` / `h265parse` | Parses the encoded elementary stream before decoding. |
| [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | Decodes the AVC (H.264) video stream to raw NV12 frames using the V4L2 driver. |
| [`v4l2h265dec`](../plugin-reference/v4l2h265dec) | Decodes the HEVC (H.265) video stream to raw NV12 frames using the V4L2 driver. |
| [`waylandsink`](../plugin-reference/waylandsink) | Receives decoded NV12 frames as GBM buffers and submits them to the Weston compositor for display. |

### Related Information

[Video encode and decode use cases](../sample-pipelines/multimediapipelines#2-camera-and-video-encode)

[Video playback use cases](../sample-pipelines/multimediapipelines#5-video-playback-use-cases)

## Audio Architecture

Audio capture and playback in QIM SDK are handled through the `pulsesrc` and `pulsesink` GStreamer plugins, which interface with the system-level PulseAudio server. The PulseAudio server in turn communicates with the ALSA driver to interact with the underlying audio hardware.

Audio encode and decode use open-source GStreamer plugins (such as `flacenc`, `flacparse`, `flacdec`, `lamemp3enc`, `mpegaudioparse`, `mpg123audiodec`) combined with the PulseAudio integration, enabling a complete audio pipeline without requiring custom hardware-specific code.

### Capture

The `pulsesrc` plugin captures raw PCM audio from the microphone and passes it to the PulseAudio server, which handles hardware interaction via the ALSA driver. The PCM stream can then be encoded or written directly to file.

<img alt="Audio capture pipeline" />

| Component | Description |
| - | - |
| `pulsesrc` | Captures PCM audio samples from the microphone via the PulseAudio server. |
| PulseAudio server | Interfaces with the ALSA driver to acquire audio data from the hardware. |
| Encode / `filesink` | Optionally encodes and writes the captured audio to a file. |

### Playback

The `pulsesink` plugin receives PCM audio data and routes it through the PulseAudio server for hardware playback. It supports both live audio sources and pre-encoded audio files after decoding.

<img alt="Audio playback pipeline" />

| Component | Description |
| - | - |
| `filesrc` | Reads audio data from a file. |
| Decode | Decodes encoded audio (e.g., MP3, FLAC, WAV) to raw PCM using open-source decoder plugins. |
| `pulsesink` | Sends PCM audio data to the PulseAudio server for playback on the audio hardware. |
| PulseAudio server | Interfaces with the ALSA driver to route audio to the hardware output. |

### Encode

Captured PCM audio is encoded using an open-source encoder, parsed, and multiplexed into a container before being written to file.

<img alt="Audio encoding pipeline" />

| Component | Description |
| - | - |
| `pulsesrc` | Captures PCM audio from the microphone. |
| PulseAudio server | Interacts with the ALSA driver to supply raw audio data. |
| Encode | Encodes PCM to a compressed format (MP3, FLAC, AAC) using open-source plugins. |
| Parse | Parses the encoded bitstream before muxing. |
| `mp4mux` / `mpegtsmux` | Multiplexes the encoded audio into an MP4 or MPEGTS container. |
| `filesink` | Writes the container to the file system. |

### Decode

An encoded audio file is read, demultiplexed, decoded back to PCM, and played back through `pulsesink`.

<img alt="Audio decode pipeline" />

| Component | Description |
| - | - |
| `filesrc` | Reads the audio container from the file system. |
| `qtdemux` | Demultiplexes the container to extract the audio elementary stream. |
| Decode | Decodes the compressed audio to raw PCM using open-source decoder plugins. |
| `pulsesink` | Sends the decoded PCM to the PulseAudio server for playback. |
| PulseAudio server | Interfaces with the ALSA driver to route audio to the hardware output. |

### Related Information

[Audio use cases](../sample-pipelines/multimediapipelines#6-audio-use-cases)

## Graphics and Display Architecture

The QIM SDK uses the Wayland display protocol with the Weston compositor to manage display composition and output. Weston runs as an independent process and handles all display composition using OpenGL ES, communicating with the display hardware through the DRM/KMS subsystem.

QIM SDK plugins deliver video buffers as GBM-backed DMA-buf handles. Weston receives these buffers via the Wayland protocol and composites them directly onto the display without intermediate CPU copies, maintaining the zero-copy path from camera capture or video decode through to the screen.

<img alt="Weston/Wayland architecture" />

| Component | Description |
| - | - |
| Wayland/GLES client | GStreamer plugins such as [`waylandsink`](../plugin-reference/waylandsink) implement the Wayland client protocol to submit video buffers to Weston. |
| Weston server | Implements the Wayland compositor. Uses KMS to configure the display and OpenGL ES with DRM for hardware-accelerated compositing. |
| SDM back-end | Display hardware abstraction layer (HAL) that provides the DRM/KMS platform implementation for Weston. |
| libGBM | Buffer allocation and sharing library. Provides a DMA-backed allocator enabling zero-copy buffer sharing between the GPU, display, and other hardware blocks. |
| EGL sub-driver | Interface between GBM/EGL and the Wayland protocol, allowing GStreamer elements to share hardware-allocated buffers with the Weston compositor. |

### Related Information

[Display and composition use cases](../sample-pipelines/multimediapipelines)


# Test Automation
Source: https://imsdkdocs.qualcomm.com/advanced/test-automation

Automated testing strategies and tools for QIM SDK pipelines.

<Note>
  This section is coming soon. Content will cover automated pipeline testing, regression frameworks, and CI/CD integration for QIM SDK.
</Note>


# Ubuntu platform
Source: https://imsdkdocs.qualcomm.com/advanced/ubuntu-build



This section explains how to compile QIM SDK plugins for Ubuntu platform.

<Tabs>
  <Tab title="On-device Compilation">
    This section explains how to compile QIM SDK plugins directly on the **target device** running Ubuntu.

    ## Prerequisites

    Before you begin, ensure the following:

    * You have access to a Qualcomm development kit running Ubuntu Linux.
    * You have administrative privileges (sudo access) for installing packages.
    * Internet connectivity for downloading packages and source code.

    ## Steps to Compile and Install

    To download the source code and compile the sample applications, run the following commands on the Qualcomm development kit.

    <Steps>
      <Step title="Update user permissions for your session">
        ```bash theme={null} theme={null}
        sudo su
        ```
      </Step>

      <Step title="Configure Ubuntu package manager to download source code">
        ```bash theme={null} theme={null}
        apt-add-repository -s ppa:ubuntu-qcom-iot/qcom-ppa
        ```
      </Step>

      <Step title="Build the dependencies">
        ```bash theme={null} theme={null}
        apt build-dep gst-plugins-qti-oss
        ```
      </Step>

      <Step title="Change directory to home folder">
        ```bash theme={null} theme={null}
        cd /home/ubuntu
        ```
      </Step>

      <Step title="Download the sources of the QIM Package">
        ```bash theme={null} theme={null}
        apt source gst-plugins-qti-oss
        ```
      </Step>

      <Step title="Go to QIM root folder">
        For example:

        ```bash theme={null} theme={null}
        cd gst-plugins-qti-oss-1.8.1+ubuntu3
        ```
      </Step>

      <Step title="Run CMake to build and install">
        ```bash theme={null} theme={null}
        cmake -B build -S . \
           -DCMAKE_INSTALL_PREFIX=/usr \
           -DCMAKE_INSTALL_INCLUDEDIR=include \
           -DENABLE_GST_PLUGIN_BASE=1 \
           -DENABLE_GST_IMSDK_PLUGINS=1 \
           -DENABLE_GST_PLUGIN_QMMFSRC=1 \
           -DENABLE_GST_PYTHON_EXAMPLES=1 \
           -DENABLE_GST_SAMPLE_APPS=1 \
           -DENABLE_GST_TEST_FRAMEWORK=1 \
           -DENABLE_GST_PLUGIN_TOOLS=1 \
           -DENABLE_GST_PLUGIN_MLACLASSIFICATION=1 \
           -DENABLE_GST_PLUGIN_MLVCLASSIFICATION=1 \
           -DENABLE_GST_PLUGIN_MLVDETECTION=1 \
           -DENABLE_GST_PLUGIN_MLVPOSE=1 \
           -DENABLE_GST_PLUGIN_MLVSEGMENTATION=1 \
           -DENABLE_GST_PLUGIN_MLVSUPERRESOLUTION=1 \
        && cmake --build build \
        && cmake --install build
        ```
      </Step>
    </Steps>

    <Note>
      After modifying any part of the source code, re-run step 7 to rebuild and reinstall your changes.
    </Note>

    ## Steps to build custom application

    Go through the [prerequisites](#prerequisites) and [Steps to Compile and Install](#steps-to-compile-and-install) section before starting this.

    <Steps>
      <Step title="Create a subdirectory in gst-sample-apps">
        ```bash theme={null} theme={null}
        cd gst-plugins-qti-oss-1.8.1+ubuntu3/gst-sample-apps && mkdir gst-<your_sample_app>
        ```
      </Step>

      <Step title="Build application">
        Follow [Step 7: Run CMake to build and install](#steps-to-compile-and-install)
      </Step>
    </Steps>
  </Tab>

  <Tab title="On-host Compilation">
    <Note>
      On-host compilation for Ubuntu is not yet available in this documentation. Please refer to the on-device compilation instructions.
    </Note>
  </Tab>
</Tabs>


# Yocto platform
Source: https://imsdkdocs.qualcomm.com/advanced/yocto-build



This section explains how to compile QIM SDK plugins for Yocto platform.

<Tabs>
  <Tab title="On-host Compilation">
    This section explains how to compile QIM SDK plugins directly on a x86 and Arm host machine running Ubuntu.

    ## Prerequisites

    Before you begin, ensure the following:

    * Internet connectivity for downloading packages and source code.
    * Sufficient space on the host to download the SDK. Upto \~5GB.

    ## Download and install the sdk.

    <Steps>
      <Step title="Download SDK">
        <Tabs>
          <Tab title="x86 Host">
            ```bash theme={null} theme={null}
            wget -O sdk.zip https://artifacts.codelinaro.org/artifactory/qli-ci/flashable-binaries/meta-qcom/qcom-distro/qcom-armv8a/x64-qli-2.0-qimsdk-2.0.0-standardsdk.zip
            ```
          </Tab>

          <Tab title="Arm Host">
            ```bash theme={null} theme={null}
            wget -O sdk.zip https://artifacts.codelinaro.org/artifactory/qli-ci/flashable-binaries/meta-qcom/qcom-distro/qcom-armv8a/arm-qli-2.0-qimsdk-2.0.0-standardsdk.zip
            ```
          </Tab>
        </Tabs>
      </Step>

      <Step title="Extract and Install the SDK">
        ```bash theme={null} theme={null}
        unzip sdk.zip && cd images/qcom-armv8a/sdk/
        ```

        <Tabs>
          <Tab title="x86 Host">
            ```bash theme={null} theme={null}
            bash qcom-distro-x86_64-qcom-multimedia-proprietary-image-armv8a-qcom-armv8a-toolchain-2.0.sh -d <path_to_install_sdk> -y
            ```
          </Tab>

          <Tab title="Arm Host">
            ```bash theme={null} theme={null}
            bash qcom-distro-aarch64-qcom-multimedia-proprietary-image-armv8a-qcom-armv8a-toolchain-2.0.sh -d <path_to_install_sdk> -y
            ```
          </Tab>
        </Tabs>
      </Step>

      <Step title="Source the cross-compilation toolchain environment">
        ```bash theme={null} theme={null}
        source <path_to_install_sdk>/environment-setup-armv8a-qcom-linux
        ```
      </Step>
    </Steps>

    <Note>
      Each time you wish to use the SDK in a new shell session, you need to source the environment setup script in Step 3
    </Note>

    ## Steps to Compile

    To download the source code and compile the sample applications, run the following commands.

    <Steps>
      <Step title="Clone the sources from github">
        ```bash theme={null} theme={null}
        git clone https://github.com/qualcomm/qimsdk.git
        cd qimsdk
        ```
      </Step>

      <Step title="Compile base libraries">
        ```bash theme={null} theme={null}
        cmake -B build-base -S . \
          -DCMAKE_INSTALL_PREFIX=/usr \
          -DENABLE_GST_PLUGIN_BASE=1 \
        && cmake --build build-base
        ```
      </Step>

      <Step title="Install base libraries to SDK sysroot path">
        ```bash theme={null} theme={null}
        DESTDIR="${SDKTARGETSYSROOT}" cmake --install build-base --prefix /usr
        ```
      </Step>

      <Step title="Compile plugins and app builders sample apps with updated base libraries">
        ```bash theme={null} theme={null}
        cmake -B build -S . \
          -DCMAKE_INSTALL_PREFIX=/usr \
          -DENABLE_GST_IMSDK_PLUGINS=1 \
          -DENABLE_GST_PLUGIN_MLTFLITE=1 \
          -DENABLE_GST_PYTHON_EXAMPLES=1 \
          -DENABLE_GST_SAMPLE_APPS=1 \
          -DENABLE_GST_SAMPLE_APPS_CAMERA=1 \
          -DENABLE_GST_PLUGIN_TOOLS=1 \
          -DENABLE_GST_CAMERA_PLUGINS=1 \
          -DENABLE_APP_BUILDER_CPP=1 \
          -DENABLE_APP_BUILDER_PYTHON=1 \
        && cmake --build build
        ```
      </Step>
    </Steps>

    <Note>
      After modifying any part of the source code, re-run step 4 to rebuild your changes.
      The corresponding libs from build directory can then be copied on to device.
    </Note>

    ## Steps to build custom application

    Go through the [prerequisites](#prerequisites) and [Steps to Compile](#steps-to-compile) section before starting this.

    <Tabs>
      <Tab title="GStreamer Sample App">
        <Steps>
          <Step title="Create a subdirectory in gst-sample-apps">
            ```bash theme={null} theme={null}
            cd qimsdk/gstreamer/gst-sample-apps && mkdir gst-<your_sample_apps>
            ```
          </Step>

          <Step title="Build the application">
            Follow [Steps to Compile](#steps-to-compile)
          </Step>

          <Step title="Copy the binary to target device">
            ```bash theme={null} theme={null}
            #The application will be found in the below location
            qimsdk/build/gstreamer/gst-sample-apps/gst-<your_sample_app>/
            ```
          </Step>
        </Steps>
      </Tab>

      <Tab title="C++ App Builder">
        <Steps>
          <Step title="Create a subdirectory in cpp reference-apps">
            ```bash theme={null} theme={null}
            cd qimsdk/cpp/examples/reference-apps/ && mkdir <your_sample_app>
            ```
          </Step>

          <Step title="Build the application">
            Follow [Steps to Compile](#steps-to-compile)
          </Step>

          <Step title="Copy the binary to target device">
            ```bash theme={null} theme={null}
            #The application will be found in the below location
            qimsdk/build/cpp/examples/reference-apps/<your_sample_app>/
            ```
          </Step>
        </Steps>
      </Tab>
    </Tabs>
  </Tab>

  <Tab title="On-device Compilation">
    <Note>
      On-device compilation for Yocto is not yet available in this documentation. Please refer to the on-host compilation instructions.
    </Note>
  </Tab>
</Tabs>


# Coding Agent
Source: https://imsdkdocs.qualcomm.com/app-builder-coding-agent

Generate QIM SDK applications with AI coding agent skills

QIM SDK provides AI coding skills that generate complete applications from a natural-language description of your pipeline. Pick the coding agent that matches your development path, then follow its guide to generate your first application with an AI coding agent (Claude Code, Cursor, Codex, etc)

<CardGroup>
  <Card title="GStreamer App Builder" icon="diagram-project" href="/app-builder/gstreamer-coding-agent">
    <div>
      Generate GStreamer C apps and gst-launch scripts using QIM SDK plugins.
    </div>

    [Go to Coding Agent →](/app-builder/gstreamer-coding-agent)
  </Card>

  <Card title="C++ App Builder" icon="code" href="/app-builder/cpp-coding-agent">
    <div>
      Generate high-performance C++ applications on the QIM SDK C++ API.
    </div>

    [Go to Coding Agent →](/app-builder/cpp-coding-agent)
  </Card>

  <Card title="Python App Builder" icon="python" href="/app-builder/python-coding-agent">
    <div>
      Generate Python applications on the QIM SDK Python API for prototyping.
    </div>

    [Go to Coding Agent →](/app-builder/python-coding-agent)
  </Card>
</CardGroup>

The skills are available at [qimsdk-agentic-skills](https://github.com/qualcomm/qimsdk-agentic-skills/)


# Quickstart
Source: https://imsdkdocs.qualcomm.com/app-builder-quickstart

Choose a development path and jump into its quickstart guide

QIM SDK supports three ways to build AI and multimedia applications. Pick the path that best fits your workflow, then follow its quickstart guide to build your first pipeline.

<CardGroup>
  <Card title="GStreamer App Builder" icon="diagram-project" href="/quickstart">
    <div>
      Build pipelines with native GStreamer elements and QIM SDK plugins.
    </div>

    [Go to Quickstart →](/quickstart)
  </Card>

  <Card title="C++ App Builder" icon="code" href="/app-builder/cpp-quickstart">
    <div>
      Build high-performance applications with the fluent C++ App Builder API.
    </div>

    [Go to Quickstart →](/app-builder/cpp-quickstart)
  </Card>

  <Card title="Python App Builder" icon="python" href="/app-builder/python-quickstart">
    <div>
      Build applications quickly with the high-level Python App Builder API.
    </div>

    [Go to Quickstart →](/app-builder/python-quickstart)
  </Card>
</CardGroup>


# Sample Applications
Source: https://imsdkdocs.qualcomm.com/app-builder-sample-applications

Explore ready-to-run sample applications for each development path

QIM SDK provides ready-to-run sample applications for each development path. Pick the path that matches your workflow to browse its applications, or go to GitHub to view the source code.

<CardGroup>
  <Card title="GStreamer App Builder" icon="diagram-project" href="/sample-application/ai-sample-applications">
    <div>
      Sample applications built with the GStreamer App Builder.
    </div>

    [View applications →](/sample-application/ai-sample-applications)

    [View source on GitHub →](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps)
  </Card>

  <Card title="C++ App Builder" icon="code" href="/app-builder/cpp-reference-applications">
    <div>
      Reference applications built with the C++ App Builder API.
    </div>

    [View applications →](/app-builder/cpp-reference-applications)

    [View source on GitHub →](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps)
  </Card>

  <Card title="Python App Builder" icon="python" href="/app-builder/python-reference-applications">
    <div>
      Reference applications built with the Python App Builder API.
    </div>

    [View applications →](/app-builder/python-reference-applications)

    [View source on GitHub →](https://github.com/qualcomm/qimsdk/tree/main/python/examples/reference-apps)
  </Card>
</CardGroup>


# SDK Architecture and API Reference
Source: https://imsdkdocs.qualcomm.com/app-builder/cpp-architecture-api-reference

Design and API details of the QIM SDK Pipeline SDK (C++ and Python)

## Pipeline SDK Design

**Overview** This SDK delivers a streamlined, modern interfaces, engineered to facilitate the rapid development, execution, and management of sophisticated multimedia pipelines. By implementing a high-level abstraction layer, the SDK encapsulates complex underlying GStreamer based mechanisms — including element negotiation, signal management, event loops, and state transitions — allowing engineers to prioritize core application logic over boilerplate framework intricacies.

**Key Capabilities** The wrapper empowers developers to architect advanced multimedia and AI-enhanced workflows with precision and efficiency. It supports the seamless integration of diverse components, such as video capture sources, signal processing nodes, machine learning inference engines, graphical overlays, and rendering outputs. Uniquely designed to decouple implementation from the underlying framework, the SDK eliminates the need for deep GStreamer expertise or compile-time dependencies on GStreamer headers, ensuring a clean, robust, and accessible development experience.

<img alt="Introduction" />

## APIs

**Public APIs:**

* `Pipeline` — build & control pipelines fluently.
* `Element` — generic element wrapper with variadic property setting, and state control. \[it can be instantiated at create time as well as at runtime to update the pipeline.] Some element has special functions which can be controlled by dedicated elements types. Those elements derived from `Element` can be use as `Element` if special functions are not used.
  * `AppSrc` — app-facing source for zero-copy buffer pull. \[Providing the buffer from application to pipeline using standard lambdas, push APIs, EOS etc]
  * `AppSink` — app-facing sink for zero-copy buffer push \[Providing the buffer from pipeline to application with zero copy]
  * `CamSrc` - Build-in camera element which expose API for capture images \[provides functionality to take snapshots at runtime]
  * `MLVConverter` - ML video converter (`qtimlvconverter`). Expose API for custom pre-processing implementation at application utilizing standard lambdas.
  * `MLPostprocess` - ML postprocess element (`qtimlpostprocess`). Expose API for custom post-processing implementation at application utilizing standard lambdas. The callback is typed per ML task (classification, object detection, pose estimation, depth estimation, segmentation, raw tensors).
  * `MLVideoONNXBin` - ONNX inference bin (`qtimlvideoonnxbin`). Expose API for custom pre-processing and post-processing implementation at application utilizing standard lambdas.
  * `MLVideoQNNBin` - QNN inference bin (`qtimlvideoqnnbin`). Expose API for custom pre-processing and post-processing implementation at application utilizing standard lambdas.
  * `MLVideoSNPEBin` - SNPE inference bin (`qtimlvideosnpebin`). Expose API for custom pre-processing and post-processing implementation at application utilizing standard lambdas.
  * `MLVideoTFLiteBin` - TFLite inference bin (`qtimlvideotflitebin`). Expose API for custom pre-processing and post-processing implementation at application utilizing standard lambdas.
* `Buffer` — opaque buffer that bridges Native Buffer with timestamp helpers and zero-copy paths.
* `StreamFilter` and derived concrete stream helpers: `VideoFilter` (all RAW formats - YUV, RGB, bayer), `H264Filter`, `TensorFilter`, `TextFilter`, `AudioFilter`, `ImageFilter`. these filters are used to set characteristics of the streams.
* Logging controls — `SetImsdkLogLevel()` and `SetImsdkGstLogMode()` select the SDK's log verbosity and whether GStreamer's output is simplified or passed through raw. See [Logging and Diagnostics](/app-builder/cpp-logging).

**Reference flows**

### Pipeline

**Purpose**

A convenient, exception-safe wrapper around a `GstPipeline` that:

* Builds pipelines using element factory names or pre-constructed wrapper elements.
* Manages pipeline lifecycle and execution state.
* Performs delayed linking for dynamic/request-pad elements automatically via `pad-added`.

**Key Responsibilities**

* Create and own the underlying `GstPipeline` instance.
* Add elements, assign properties, and insert caps filters.
* Link elements (immediately when possible, deferred otherwise).
* Manage the full pipeline lifecycle (`prepare`, `start`, `wait`, `stop`).
* Install bus watches to handle `ERROR`, `EOS`, and state transitions.
* Integrate with runtime shutdown notifications.

**A minimal pipeline**

Add the elements in the order data flows through them, then run:

```cpp theme={null}
Pipeline pipeline("my-pipeline");

pipeline
    .add("filesrc", "src", "location", "/home/user/media/video.mp4")
    .add("qtdemux", "demux")
    .add("h264parse", "parse")
    .add("v4l2h264dec", "decoder", "output-io-mode", 4, "capture-io-mode", 4)
    .add_stream_filter("vf", VideoFilter().format("NV12"))
    .add("waylandsink", "display", "fullscreen", true)
    .execute();
```

`execute()` starts the pipeline, blocks until EOS/error/termination, and tears it down.

<Tip>
  **Adjacent elements are linked in add order.** In the example above `src` → `demux` → `parse` → `decoder` → `vf` → `display` are linked simply because that is the order of the `add()` calls, so a linear graph needs no explicit linking. Use [`link()`](#branching-with-explicit-links) when the graph is not linear — a `tee` splitting into a display branch and an inference branch, or a `qtimetamux` rejoining them.
</Tip>

<Note>
  `qtdemux` is an example of the delayed linking described above: its src pads appear only once the container has been parsed, so the `demux` → `parse` link is completed automatically when the pad shows up. No `pad-added` handling is needed in application code.
</Note>

**Building the pipeline**

| Method | Description |
| - | - |
| `Pipeline(const std::string &name)` | Creates an empty pipeline. `name` is the `GstPipeline` name, used in log messages. |
| `Pipeline(const std::string &name, const std::string &config)` | Builds a complete pipeline from a YAML description. `config` is the YAML **content**, not a path. See [Describing a pipeline in YAML](#describing-a-pipeline-in-yaml). |
| `Pipeline &add(String factory, String name, ...)` | Creates an element from the factory name and adds it under a unique instance name. Trailing arguments are property key/value pairs, applied immediately. |
| `Pipeline &add(const Element &element)` | Adds an externally constructed element. Required for the specialized wrappers (`AppSrc`, `AppSink`, `CamSrc`, `MLVConverter`, `MLPostprocess`, the ML bins) so callbacks can be attached before the element joins the pipeline. |
| `Pipeline &add_stream_filter(String name, const StreamFilter &filter)` | Inserts a caps filter between the previous and next element. When neighbouring elements have several formats in common, this fixates the one you want instead of leaving the choice to caps negotiation. See [StreamFilter](#streamfilter). |
| `Pipeline &link(names...)` | Links the named elements in the given order. Needed only for non-linear graphs. |

The optional key/value pairs passed to `add()` are the element's properties, and they are validated as soon as the element is added. Setting a property the element does not have throws `std::invalid_argument` naming the property, the element instance, and the element type — so a typo surfaces immediately at that `add()` call instead of being silently ignored:

```cpp theme={null}
// "fulscreen" is not a waylandsink property -> throws std::invalid_argument
pipeline.add("waylandsink", "display", "fulscreen", true);
```

**Branching with explicit links**

Add order cannot express a branch, so name the chains explicitly. Element instance names are the identifiers.

```cpp theme={null}
pipeline
    .add("filesrc", "src", "location", path)
    .add("qtdemux", "demux")
    .add("h264parse", "parse")
    .add("v4l2h264dec", "decoder", "output-io-mode", 4, "capture-io-mode", 4)
    .add_stream_filter("vf", VideoFilter().format("NV12"))
    .add("tee", "split")
    .add("queue", "q1")
    .add("qtimlvconverter", "preprocessing")
    .add("qtimltflite", "inferencing", "model", model_path)
    .add("qtimlpostprocess", "postprocessing", "module", "yolov8")
    .add_stream_filter("mlf", TextFilter())
    .add("qtimetamux", "mlmuxer")
    .add("qtivoverlay", "overlay")
    .add("waylandsink", "display", "fullscreen", true)
    // trunk, up to the tee
    .link("src", "demux", "parse", "decoder", "vf", "split")
    // branch 1: raw video to the metadata muxer
    .link("split", "mlmuxer")
    // branch 2: inference path, rejoining at the muxer
    .link("split", "q1", "preprocessing", "inferencing", "postprocessing",
          "mlf", "mlmuxer", "overlay", "display");
```

<Tip>
  Elements whose src pads are `SOMETIMES` or `REQUEST` — `qtdemux`, `tsdemux`, `tee`, `qtivsplit` — cannot be linked at construction time, because the pads do not exist yet. The pipeline detects this from the pad templates, defers the link, and completes it from a `pad-added` handler it installs itself. Request pads on the sink side (for example `qtimetamux`, `qtivcomposer`) are requested and released automatically. No `pad-added` callback is needed in application code.
</Tip>

<Note>
  A link between two elements that both have only static pads and no common caps fails immediately with `std::runtime_error` naming both elements. A deferred link that never resolves surfaces at run time instead; on error the pipeline logs which pads stayed unlinked.
</Note>

**Running the pipeline**

A pipeline is always in one of three states, and the lifecycle methods are the transitions between them:

<img alt="Introduction" />

| State | Meaning |
| - | - |
| `NULL` | No resources allocated. Initial state, and the state after `stop()`. Devices are closed, buffer pools released. |
| `PAUSED` | Elements are allocated and caps are negotiated between linked pads, but the clock is not running and no data flows. Sinks have pre-rolled. This is the state in which elements can be added, removed, or relinked. |
| `PLAYING` | The clock is running and buffers flow through the graph. |

`execute()` performs the whole `NULL → PLAYING → NULL` cycle, so most applications never call the individual transitions. `prepare()`, `activate()`, and `deactivate()` — the transitions through `PAUSED` shown above — are covered in [Advanced state control](#advanced-state-control).

| Method | Description |
| - | - |
| `void execute()` | `start()`, `wait()`, `stop()` in sequence. The normal entry point. |
| `Pipeline &start()` | Sets the pipeline to `PLAYING`. Non-blocking: the state change is asynchronous and this returns without waiting for it. Throws `std::runtime_error` if the transition fails. |
| `Pipeline &wait()` | Blocks until the pipeline is done: `EOS` on the bus, `ERROR` on the bus, `stop()` from another thread, or `SIGINT`/`SIGTERM`. |
| `Pipeline &stop()` | Sets the pipeline to `NULL` and releases resources. Blocks until teardown completes. |
| `Pipeline &eos(bool enabled)` | When enabled, `stop()` first sends an end-of-stream event through the pipeline and waits for it to come back from the sinks, so that data already inside the pipeline is processed before teardown. Off by default. See [Draining the pipeline on shutdown](#draining-the-pipeline-on-shutdown). |

<Warning>
  `start()` is non-blocking, so it must be paired with `wait()` (or your own blocking logic). A `main()` that calls `start()` and returns destroys the pipeline before buffers flow.
</Warning>

To run the pipeline while the application does other work, split `execute()` apart:

```cpp theme={null}
pipeline.start();       // returns immediately

// ... application work, runtime property changes, snapshots ...

pipeline.wait();        // block until EOS / error / Ctrl+C
pipeline.stop();
```

**Draining the pipeline on shutdown**

By default `stop()` tears the pipeline down as fast as it can. Every element goes to `NULL` immediately and whatever buffers are still in flight — queued between elements, held inside an encoder, not yet written by a sink — are discarded. This is usually what you want: the pipeline stops without delay.

Some pipelines must not lose those buffers, and for them `eos(true)` changes the shutdown sequence. Instead of stopping straight away, `stop()` sends an end-of-stream event into the pipeline. The event travels downstream element by element; each element finishes processing the data it has already accepted, passes the event on, and the sinks report it back on the bus. Only once that end-of-stream notification comes back does `stop()` tear the pipeline down. Nothing that had already entered the pipeline is dropped.

The trade-off is shutdown latency: `stop()` now blocks until the last buffer has made it through the whole graph, which takes as long as the remaining data needs.

Use `eos(true)` when the tail of the stream is part of the result:

* **Muxed recordings.** Containers such as MP4 and MKV are written by a muxer (`mp4mux`, `matroskamux`) that writes its index and header metadata — the `moov` atom, in MP4's case — only on end-of-stream. Cut it off early and the file is left incomplete, often unplayable rather than merely short.
* **Encoders with buffered frames.** Encoders keep several frames in flight for B-frames, lookahead, and similar optimizations. End-of-stream is what makes the encoder flush those frames downstream; without it the tail of the recording is simply missing.
* **Any sink where losing the final buffered data is unacceptable** — file sinks, muxed outputs, and applications that expect to receive every frame through `AppSink`.

Not every sink needs draining, though. A live network stream can usually just be cut off, and a receiver will not care that the last few buffers never arrived. The question is not whether a sink writes somewhere, but whether anything downstream depends on the data still sitting in the pipeline.

Leave `eos()` off (the default) when the newest data is all that matters and stopping promptly is preferable — live preview to a display, a camera feed on screen, or writing self-contained files where each buffer is already complete on its own (JPEG frames through `multifilesink`, for example, which needs no finalization).

```cpp theme={null}
Pipeline pipeline("cam-encoder-pipeline");
pipeline
    .add("qtiqmmfsrc", "source")
    .add_stream_filter("vf", VideoFilter().format("NV12")
                                          .resolution(1920, 1080)
                                          .framerate(30))
    .add("v4l2h264enc", "encoder",
         "output-io-mode", "dmabuf-import", "capture-io-mode", "dmabuf")
    .add("h264parse", "parser")
    .add("mp4mux", "muxer")
    .add("filesink", "sink", "location", HOME_PATH + "/media/output.mp4")
    .eos(true)          // drain, so mp4mux can finalize the container
    .execute();
```

<Note>
  If the pipeline has already reached end-of-stream on its own — a file source that ran out of data, for instance — `stop()` detects it and tears down immediately instead of sending a second event and waiting again. Enabling `eos(true)` therefore costs nothing for pipelines that end naturally; it only matters when you stop a pipeline that is still running.
</Note>

**Advanced state control**

For reconfiguring a loaded pipeline — adding, removing, or relinking elements without returning to `NULL`.

| Method | Description |
| - | - |
| `Pipeline &prepare()` | `NULL` → `PAUSED`. Allocates resources and negotiates caps without running the clock. Blocking. |
| `Pipeline &activate()` | `PAUSED` → `PLAYING`. |
| `Pipeline &deactivate()` | `PLAYING` → `PAUSED`. Resources stay allocated. |

**Accessing elements at run time**

| Method | Description |
| - | - |
| `Element get(String name)` | Returns a wrapper for the named element, for runtime property changes and per-pad configuration. Throws `std::runtime_error` if the name is not in the pipeline. |
| `T get<T>(String name)` | Returns a typed wrapper, exposing that type's element-specific API — for example `get<CamSrc>("source")` for the image-capture calls. Use it when the element was added by factory name but its specialized interface is needed. |

**Runtime property and per-pad configuration** — `get()` reaches any element by name while the pipeline is loaded or playing:

```cpp theme={null}
// Change a property at run time.
pipeline.get("display").set("fullscreen", false);

// Per-pad properties on a request pad of qtivcomposer.
pipeline.get("composer").input(1).set("alpha", 0.5);
```

**Typed access to an element's own API** — `get<T>()` returns the specialized wrapper, so element-specific calls become available on an element that was added by factory name:

```cpp theme={null}
// qtiqmmfsrc was added as "source"; CamSrc exposes its capture API.
auto cam = pipeline.get<CamSrc>("source");
cam.image_capture();
```

See the sample applications for complete image-capture pipelines.

<Note>
  `get()` returns an element wrapper **by value**; each call yields a new wrapper referencing the same `GstElement`. Bind it with `auto`, not `auto&`.
</Note>

**Describing a pipeline in YAML**

The graph — elements, properties, stream filters, and links — can be described in YAML instead of in code. This separates *what the pipeline is* from *what the application does with it*: the same binary can run a different topology, a different model, or a different source by pointing it at another configuration file.

That makes it useful where the pipeline is not fixed at build time:

* **Deploying one binary across variants** — the same application against a built-in camera, a USB camera, or an RTSP stream, switching source and format per device.
* **Tuning without rebuilding** — resolution, framerate, model path, or delegate changed on the target, which matters when a rebuild means a full cross-compile and reflash.
* **Keeping topology under review** — the pipeline shape lives in a file that can be diffed and versioned on its own.

Loading from YAML does not give up application logic: the elements are ordinary pipeline members afterwards, so `get()` and `get<T>()` reach them as usual.

```yaml theme={null}
pipeline:
  elements:
    - type: filesrc
      name: src
      location: ~/media/video.mp4

    - type: qtdemux
      name: demux

    - type: h264parse
      name: parse

    - type: v4l2h264dec
      name: decoder
      output-io-mode: 4
      capture-io-mode: 4

    - type: filter
      name: videofilter
      video:
        format: NV12

    - type: waylandsink
      name: display
      fullscreen: true
```

Everything lives under a single root key, `pipeline:`, which takes three independent child keys:

```yaml theme={null}
pipeline:
  eos: true             # optional — EOS-on-shutdown, same as eos(true) in code
  elements:             # required — the elements, in add order
    - type: ...
  links:                # optional — explicit links, for non-linear graphs
    - [ ... ]
```

| Key | Required | Description |
| - | - | - |
| `eos` | No | Boolean. Equivalent to calling [`eos(bool)`](#draining-the-pipeline-on-shutdown). Defaults to false when omitted. |
| `elements` | Yes | Sequence of elements. Add order is link order, exactly as with `add()`. |
| `links` | No | Sequence of chains, each a list of element names. Equivalent to one `link()` call per entry. Omit it for linear pipelines. |

Within `elements:`, each entry needs `type` (the factory name) and `name` (the instance name); every remaining key is applied as a property of that element. An entry with `type: filter` inserts a stream filter instead, described by a `video:`, `image:`, `h264:`, `tensor:`, `text:`, `audio:`, or raw `caps:` block. Any of those blocks may carry an `add:` sequence of `key=value` strings, appended to the filter's caps through [`add()`](#videofilter) — the same escape-hatch the typed filters expose in code. It is accepted both inside the filter block and alongside it:

```yaml theme={null}
    - type: filter
      name: videofilter
      video:
        format: NV12
        add:
          - colorimetry=bt709
```

`links:` mirrors [`link()`](#branching-with-explicit-links) — one sequence per chain, so the branching pipeline shown earlier becomes:

```yaml theme={null}
  links:
    - [split, mlmuxer]
    - [source, videostream, split, q1, preprocessing, inferencing,
       postprocessing, mlf, mlmuxer, overlay, display]
```

The constructor takes the YAML content, so read the file first:

```cpp theme={null}
std::ifstream input(config_path);
std::ostringstream buffer;
buffer << input.rdbuf();

Pipeline pipeline("demo-pipeline", buffer.str());
pipeline.execute();
```

The names in the configuration are the handles the application uses afterwards. Given a config that declares a `qtimlpostprocess` element named `postprocessing`, the application can claim its post-processing without the topology being hard-coded:

```cpp theme={null}
Pipeline pipeline("demo-pipeline", buffer.str());

auto postprocessing = pipeline.get<MLPostprocess>("postprocessing");
postprocessing.set_handler([](const MLFrame& frame, const MLParam& params,
                              MLDetections& detections) {
  return decode_detection(frame, params, detections);
});

pipeline.execute();
```

### Element

**Purpose**

Generic wrapper around a `GstElement` with fluent, variadic property setting. Can be instantiated at create time or used at runtime to update a live pipeline. Serves as the base class for all specialized element types.

**Key Responsibilities**

* Create GStreamer elements from a factory name.
* Provide fluent, variadic property setting for all common types and `StreamFilter` (caps).
* Implement pad linking/unlinking with error checking.
* Support state publication to pipeline (`sync`, `deactivate`, `stop`).
* Expose per-pad configuration through `Port`.
* Serve as the base class for specialized wrappers.

```cpp theme={null}
Element display("waylandsink", "display");
display.set("fullscreen", true);
display.set("sync", false);
```

Any element can be constructed directly and handed to `Pipeline::add()`, or reached later by name with `Pipeline::get()`. Elements cannot be copied — only moved — so pass them by reference or move them where ownership has to change.

| `qti::Element` | Description |
| - | - |
| `Element(const std::string &factory, const std::string &name = {})` | Creates the element from a GStreamer factory name. The instance name is optional but recommended, since it is how the element is addressed in `link()`, `get()`, and log messages. |
| `Element &set(String prop, Value value, ...)` | Sets one or more properties. Accepts any number of key/value pairs in a single call and returns `*this`, so calls can be chained. |
| `Element &link(Element &downstream, String src_pad = "src", String sink_pad = "sink")` | Links this element's src pad to `downstream`'s sink pad. Throws `std::runtime_error` naming both pads if the link fails. |
| `Element &unlink(Element &downstream, String src_pad = "src", String sink_pad = "sink")` | Unlinks the named pad pair between this element and `downstream`. |
| `Element &unlink(String src_pad = "src")` | Unlinks whatever is connected to the given src pad, without naming the peer. |
| `Element &sync()` | Synchronizes the element's state with its parent pipeline's state. Call this after adding an element to a pipeline that is already running. |
| `Element &deactivate()` | Moves the element to `PAUSED`. |
| `Element &stop()` | Moves the element to `NULL`. |
| `Port input(unsigned int id)` | Returns the sink pad at the given index, for per-pad configuration. |
| `Port input(String name_or_type, unsigned int id)` | Returns the sink pad matching a pad-template name and index. |
| `Port output(unsigned int id)` | Returns the src pad at the given index. |
| `Port output(String name_or_type, unsigned int id)` | Returns the src pad matching a pad-template name and index. |

**Setting properties**

`set()` is variadic, so a whole configuration can go in one call, and property names are validated immediately — an unknown property throws `std::invalid_argument` naming the property and the element type rather than being silently ignored:

```cpp theme={null}
Element decoder("v4l2h264dec", "decoder");
decoder.set("output-io-mode", 4, "capture-io-mode", 4);
```

Besides the usual scalar types and strings, `set()` accepts a `StreamFilter` wherever an element takes a caps property, and plain enums (converted to their underlying integer):

```cpp theme={null}
AppSrc appsrc("appsrc");
appsrc.set("caps", VideoFilter().format("NV12")
                                .resolution(1920, 1080)
                                .framerate(30));
```

**Per-pad configuration with Port**

Some elements are configured per pad rather than per element — a compositor's inputs each have their own position, size, and alpha. `input()` and `output()` return a `Port` for one pad, which carries the same fluent, validated `set()`:

```cpp theme={null}
// Second input of qtivcomposer: blend at 50% alpha.
pipeline.get("composer").input(1).set("alpha", 0.5);
```

| `qti::Port` | Description |
| - | - |
| `Port &set(String prop, Value value, ...)` | Sets one or more pad properties. Validated like `Element::set()`, and additionally accepts `std::vector<int>` and `{a, b}` initializer lists for array-valued pad properties such as positions and dimensions. |

The `id` selects which pad, and `name_or_type` narrows it to a pad template when an element has several families of pads. A template containing `%u` is formatted with the index (`"image_%u"` with `id` 0 gives `image_0`); otherwise the name and index are joined with an underscore (`"sink"` with `id` 1 gives `sink_1`). Without a name, the pad is matched by numeric suffix first and by position among same-direction pads second. An unresolvable pad throws `std::runtime_error`.

<Note>
  A `Port` refers to a pad of a live element and cannot be copied, only moved. Get it, set the properties, and let it go — as in the one-liner above — rather than storing it beyond the lifetime of the element.
</Note>

**Runtime changes**

Because `Pipeline::get()` returns an `Element` for any element in the pipeline, the same `set()` works while the pipeline is playing:

```cpp theme={null}
pipeline.get("display").set("fullscreen", false);
```

`sync()`, `deactivate()`, and `stop()` exist for reconfiguring a live pipeline. An element added to an already-running pipeline starts in `NULL` while the pipeline is `PLAYING`, so it will not process data until its state is brought up to match — that is what `sync()` does. `deactivate()` and `stop()` take a single element down without touching the rest of the graph.

<Note>
  Runtime relinking is an advanced use of the API. Pad linking, unlinking, and per-element state changes on a live pipeline must respect GStreamer's rules about when pads may be changed; the wrappers report failures as exceptions but do not remove the need to get the sequence right.
</Note>

### AppSrc

**Purpose**

`AppSrc` wraps the `appsrc` element, the entry point for feeding application-produced buffers into a pipeline. It manages the `need-data`/`enough-data` signal plumbing so the application only has to supply buffers, either from a callback or by pushing them directly.

**Key Responsibilities**

* Validate that the wrapped element is indeed an `appsrc`.
* Register and dispatch the underlying `need-data` and `enough-data` signals.
* Let the application supply buffers on demand through `set_buffer_producer()`, or push them directly with `push_buffer()`.
* Signal end-of-stream on this element's output.
* Manage lifetime of the connected signal handlers.

```cpp theme={null}
AppSrc appsrc("src");
appsrc.set("is-live", true)
    .set("block", true)
    .set("format", AppSrc::Format::TIME)
    .set("do-timestamp", true)
    .set("caps", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30));
```

| `qti::AppSrc` | Description |
| - | - |
| `AppSrc(const std::string &name = {})` | Creates an `appsrc` element with an optional instance name. |
| `AppSrc &set(String prop, Value value, ...)` | Sets one or more properties, same as `Element::set()`. |
| `AppSrc &set_buffer_producer(std::function<bool(qti::Buffer &)> producer)` | Registers a callback invoked whenever appsrc needs more data. Fill the buffer and return `true` to push it, `false` to skip this round. |
| `AppSrc &set_enough_handler(std::function<void()> enough)` | Registers a callback invoked when appsrc reports it is holding enough queued data. |
| `bool push_buffer(qti::Buffer &buffer)` / `bool push_buffer(qti::Buffer &&buffer)` | Pushes a buffer into the stream directly, outside the `need-data` callback. Returns `true` once it is accepted downstream. |
| `void end_of_stream()` | Sends an end-of-stream event on this element's output. |

`AppSrc::Format` mirrors appsrc's own `format` property (`DEFAULT`, `BYTES`, `TIME`, `BUFFERS`, `PERCENT`); `TIME`, used above, is the usual choice for timestamped media.

<Note>
  `push_buffer()` transfers a buffer's underlying `GstBuffer` without copying when the `Buffer` already owns one — which is exactly the case for a buffer just pulled from an `AppSink`. A `Buffer` filled with raw data instead (as inside a `set_buffer_producer()` callback) is copied into a new `GstBuffer` on push.
</Note>

### AppSink

**Purpose**

`AppSink` wraps the `appsink` element, the exit point that delivers pipeline output back to application code. It pulls samples through appsink's action signals and hands them to the application as `Buffer`.

**Key Responsibilities**

* Validate that the wrapped element is indeed an `appsink`.
* Pull samples via appsink's action signals and convert them into `Buffer`.
* Deliver each buffer to the application through `set_buffer_consumer()`.
* Deliver the first (preroll) sample separately through `set_preroll_handler()`.
* Notify the application of end-of-stream through `set_eos_handler()`.
* Manage lifetime of the connected signal handlers.

```cpp theme={null}
AppSink appsink("sink");
appsink.set("emit-signals", true)
    .set("max-buffers", 5)
    .set("drop", true)
    .set_buffer_consumer([](Buffer buf) {
        // consume buf.data(), buf.size(), buf.pts()
    });
```

| `qti::AppSink` | Description |
| - | - |
| `AppSink(const std::string &name = {})` | Creates an `appsink` element with an optional instance name. |
| `AppSink &set(String prop, Value value, ...)` | Sets one or more properties, same as `Element::set()`. |
| `AppSink &set_buffer_consumer(std::function<void(qti::Buffer)> consumer)` | Registers a callback invoked for every buffer pulled from the pipeline. |
| `AppSink &set_preroll_handler(std::function<bool(qti::Buffer &&)> preroll)` | Registers a callback invoked for the first (preroll) sample only; return `false` to report an error instead of accepting it. |
| `AppSink &set_eos_handler(std::function<void()> eos)` | Registers a callback invoked when appsink receives end-of-stream. |

**Bridging two pipelines**

Because `AppSink` and `AppSrc` are ordinary elements, they can sit in two independent `Pipeline` objects and be wired together in application code — useful when one graph produces frames (say, from a camera) and another consumes them (say, encoding or display) on its own clock:

```cpp theme={null}
AppSrc appsrc("src");
appsrc.set("is-live", true)
    .set("block", true)
    .set("format", AppSrc::Format::TIME)
    .set("do-timestamp", true)
    .set("caps", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30));

AppSink appsink("sink");
appsink.set("emit-signals", true)
    .set("max-buffers", 5)
    .set("drop", true)
    .set_buffer_consumer([&](Buffer b) {
        appsrc.push_buffer(std::move(b));
    });

Pipeline pipeline1("p1");
pipeline1.add("videotestsrc", "testsrc", "is-live", true, "pattern", "ball")
    .add_stream_filter("videostream", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
    .add(appsink);

Pipeline pipeline2("p2");
pipeline2.add(appsrc)
    .add("waylandsink", "display", "fullscreen", true);
```

Each buffer handed to `set_buffer_consumer()` on `pipeline1` is pushed straight into `pipeline2` through `appsrc.push_buffer()`, without a copy.

### CamSrc

**Purpose**

`CamSrc` wraps the `qtiqmmfsrc` camera source element and exposes its still/burst image-capture action signals as ordinary method calls, so an application can trigger a snapshot without touching GStreamer signals directly.

**Key Responsibilities**

* Validate that the wrapped element is indeed a `qtiqmmfsrc`.
* Trigger still or burst image capture through `image_capture()`.
* Cancel any pending capture request through `cancel_capture()`.

| `qti::CamSrc` | Description |
| - | - |
| `CamSrc(const std::string &name = {})` | Creates a `qtiqmmfsrc` element with an optional instance name. |
| `bool image_capture(unsigned int count = 1)` | Captures `count` still images. |
| `bool image_capture(CaptureMode mode, unsigned int count = 1)` | Captures `count` images using `CaptureMode::kStill` or `CaptureMode::kBurst`. |
| `bool image_capture(CaptureMode mode, unsigned int count, const std::vector<void*> &metadata_ptrs)` | Same as above, attaching a per-request metadata payload to each capture. |
| `bool cancel_capture()` | Cancels any capture requests still pending. |

A running pipeline built from a plain factory name still has a `qtiqmmfsrc` element in it; use `Pipeline::get<CamSrc>()` to reach it as a `CamSrc` and call its capture API:

```cpp theme={null}
auto cam = pipeline.get<CamSrc>("source");
cam.image_capture();
```

See `examples/test_camera_and_capture` for a complete pipeline that branches a `qtiqmmfsrc` into a live display and a `multifilesink` for captured images.

### MLVConverter

**Purpose**

`MLVConverter` wraps the `qtimlvconverter` element, which converts raw video frames into the input tensor format expected by a model. Beyond the properties available on the generic `Element`, it lets the application replace the built-in conversion with its own implementation through a lambda.

**Key Responsibilities**

* Validate that the wrapped element is a `qtimlvconverter`.
* Convert incoming video frames into model input tensors.
* Optionally delegate the conversion to an application-supplied pre-process callback.
* Provide fluent, variadic property setting that returns `MLVConverter&` for chaining.

**Pre-process callback**

The callback receives the source frames as `MLVideoBlits` (each entry carries the image planes plus the source quadrilateral, destination rectangle, alpha and rotation) and must fill the tensors of the `MLFrame` output. Returning `false` marks the conversion as failed.

```cpp theme={null}
using TensorsPreprocessCallback =
    std::function<bool(const MLVideoBlits& blits, MLFrame& output)>;
```

<Note>
  Set `engine` to `none` to disable the internal pre-processing path when you supply your own callback — otherwise the built-in conversion runs instead.
</Note>

**Usage**

```cpp theme={null}
MLVConverter preprocessing("preprocessing");

// engine=none disables internal preprocessing path in qtimlvconverter.
preprocessing.set("engine", "none");

preprocessing.set_handler(
    [](const MLVideoBlits& blits, MLFrame& output) {
      if (blits.entries.empty() || output.tensors.empty())
        return false;

      const MLVideoBlit& blit = blits.entries.front();
      return convert_nv12_to_nhwc_i8(blit.image, &blit,
                                     output.tensors.front());
    });

pipeline.add(preprocessing);
```

| `qti::MLVConverter` | Description |
| - | - |
| `MLVConverter(const std::string &name = {})` | Creates a `qtimlvconverter` element with an optional instance name. |
| `MLVConverter &set(String prop, Value value, ...)` | Sets one or more properties, same as `Element::set()`. |
| `MLVConverter &set_handler(TensorsPreprocessCallback handler)` | Registers the external pre-process callback. Requires `engine=none` to take effect. |

### MLPostprocess

**Purpose**

`MLPostprocess` wraps the `qtimlpostprocess` element, which decodes model output tensors into ML metadata attached to the frame. It lets the application replace the built-in decoding module with its own implementation through a lambda.

**Key Responsibilities**

* Validate that the wrapped element is a `qtimlpostprocess`.
* Decode output tensors into ML metadata (classifications, detections, poses, depth maps, segmentations), or pass the raw tensors through for a further stage.
* Optionally delegate the decoding to an application-supplied post-process callback.
* Select the callback type automatically from the lambda's third parameter.
* Provide fluent, variadic property setting that returns `MLPostprocess&` for chaining.

**Post-process callbacks**

`set_handler()` is overloaded once per ML task. The overload is selected by the type of the third parameter of the callback, so the same method name registers any of the supported result types. Each callback receives the model output tensors as `MLFrame`, the element parameters as `MLParam`, and an output container to fill. Returning `false` marks the frame as failed.

| Callback type | Signature | Output |
| - | - | - |
| `ClassificationPostprocessCallback` | `bool(const MLFrame&, const MLParam&, MLClassifications&)` | Class name, confidence, color |
| `ObjectDetectionPostprocessCallback` | `bool(const MLFrame&, const MLParam&, MLDetections&)` | Bounding boxes, confidence, landmarks |
| `PoseEstimationPostprocessCallback` | `bool(const MLFrame&, const MLParam&, MLPoses&)` | Keypoints and keypoint links |
| `DepthEstimationPostprocessCallback` | `bool(const MLFrame&, const MLParam&, MLDepthMaps&)` | Per-cell depth values and colors |
| `SegmentationPostprocessCallback` | `bool(const MLFrame&, const MLParam&, MLSegmentations&)` | Per-cell labels and colors |
| `TensorsPostprocessCallback` | `bool(const MLFrame&, const MLParam&, MLFrame&)` | Raw output tensors, for daisy-chaining into a next stage |

<Tip>
  Use `MLParam::get()` to read element parameters inside the callback. For example `params.get("input-tensor-region", region)` returns the `Region` the frame was scaled into, which is needed to map model coordinates back to frame coordinates.
</Tip>

**Usage**

```cpp theme={null}
static const std::vector<LabelEntry> labels =
    load_labels(HOME_PATH + "/labels/resnet101.json");

MLPostprocess postprocessing("postprocessing");
postprocessing
    .set("results", 1)
    .set_handler([](const MLFrame& frame, const MLParam& params,
                    MLClassifications& classifications) {
      return decode_top1_classification(frame, params, labels,
                                        classifications,
                                        /*confidence_threshold=*/51.0f, "");
    });

pipeline.add(postprocessing);
```

| `qti::MLPostprocess` | Description |
| - | - |
| `MLPostprocess(const std::string &name = {})` | Creates a `qtimlpostprocess` element with an optional instance name. |
| `MLPostprocess &set(String prop, Value value, ...)` | Sets one or more properties, same as `Element::set()`. |
| `MLPostprocess &set_handler(ClassificationPostprocessCallback handler)` | Registers a callback that decodes tensors into classifications. |
| `MLPostprocess &set_handler(ObjectDetectionPostprocessCallback handler)` | Registers a callback that decodes tensors into detections. |
| `MLPostprocess &set_handler(PoseEstimationPostprocessCallback handler)` | Registers a callback that decodes tensors into poses. |
| `MLPostprocess &set_handler(DepthEstimationPostprocessCallback handler)` | Registers a callback that decodes tensors into depth maps. |
| `MLPostprocess &set_handler(SegmentationPostprocessCallback handler)` | Registers a callback that decodes tensors into segmentations. |
| `MLPostprocess &set_handler(TensorsPostprocessCallback handler)` | Registers a callback that produces raw output tensors, for chaining into another ML stage. |

### ML inference bins

**Purpose**

The ML video bins wrap a full pre-process → inference → post-process chain into a single element, so a whole AI branch can be added to the pipeline as one node instead of being assembled from separate elements and queues. Four bins are available, one per inference runtime:

| QIM SDK C++ class | GStreamer element | Runtime |
| - | - | - |
| `MLVideoONNXBin` | `qtimlvideoonnxbin` | ONNX Runtime |
| `MLVideoQNNBin` | `qtimlvideoqnnbin` | Qualcomm® AI Engine Direct (QNN) |
| `MLVideoSNPEBin` | `qtimlvideosnpebin` | Snapdragon® Neural Processing Engine (SNPE) |
| `MLVideoTFLiteBin` | `qtimlvideotflitebin` | LiteRT (TensorFlow Lite) |

All four expose an identical API and differ only in the wrapped element and the runtime-specific properties. Each inherits from both the pre-process and the post-process base, so it accepts a custom pre-process callback, a custom post-process callback, or both.

**Key Responsibilities**

* Validate that the wrapped element matches the expected bin factory.
* Run pre-processing, inference, and post-processing inside a single element.
* Optionally delegate pre-processing to an application-supplied callback via `set_preprocess_handler()`.
* Optionally delegate post-processing to an application-supplied callback via `set_postprocess_handler()`.
* Provide fluent, variadic property setting that returns the concrete bin type for chaining.

**Properties**

Because the bin embeds several stages, its properties are prefixed by the stage they configure — `preprocess-*`, `inference-*`, and `postprocess-*`. For example `inference-model` on the bin corresponds to `model` on a standalone inference element, and `postprocess-labels` corresponds to `labels` on a standalone `qtimlpostprocess`.

<Note>
  Set `preprocess-engine` to `none` to disable the internal pre-processing path when you supply your own pre-process callback.
</Note>

**Usage**

**Custom post-processing** — the bin handles pre-processing and inference internally, the application decodes the output tensors.

```cpp theme={null}
static const std::vector<LabelEntry> labels =
    load_labels(HOME_PATH + "/labels/yolov8.json");

MLVideoTFLiteBin mlbin("mlbin");
mlbin
    .set("inference-delegate", "external",
         "inference-external-delegate-path", "libQnnTFLiteDelegate.so",
         "inference-external-delegate-options",
         "QNNExternalDelegate,backend_type=htp;",
         "inference-model", HOME_PATH + "/models/yolov8_det_quantized.tflite")
    .set_postprocess_handler(
        [&](const MLFrame& frame, const MLParam& params,
            MLDetections& detections) {
          return decode_detection(frame, detections, params, labels,
                                  /*confidence_threshold=*/0.70f);
        });

Pipeline pipeline("mlbin-external-detection");
pipeline
    .add("filesrc", "src", "location", HOME_PATH + "/media/video.mp4")
    .add("qtdemux", "demux")
    .add("h264parse", "parse")
    .add("v4l2h264dec", "decoder", "output-io-mode", 4, "capture-io-mode", 4)
    .add_stream_filter("vf", VideoFilter().format("NV12"))
    .add(mlbin)
    .add("qtivoverlay", "overlay")
    .add("waylandsink", "display", "fullscreen", true);

pipeline.execute();
```

**Custom pre-processing** — the application converts frames into input tensors, the bin runs inference and its built-in post-process module.

```cpp theme={null}
MLVideoTFLiteBin mlbin("mlbin");

// preprocess-engine=none disables internal preprocessing path in mlbin.
mlbin.set("preprocess-engine", "none");
mlbin.set("inference-delegate", "external");
mlbin.set("inference-external-delegate-path", "libQnnTFLiteDelegate.so");
mlbin.set("inference-external-delegate-options",
          "QNNExternalDelegate,backend_type=htp;");
mlbin.set("inference-model",
          HOME_PATH + "/models/yolov8_det_quantized.tflite");
mlbin.set("postprocess-module", "yolov8");
mlbin.set("postprocess-labels", HOME_PATH + "/labels/yolov8.json");

mlbin.set_preprocess_handler(
    [](const MLVideoBlits& blits, MLFrame& output) {
      if (blits.entries.empty() || output.tensors.empty())
        return false;

      const MLVideoBlit& blit = blits.entries.front();
      return convert_nv12_to_nhwc_i8(blit.image, &blit,
                                     output.tensors.front());
    });
```

Swapping runtimes is a matter of changing the class and the runtime-specific properties; the callback registration is unchanged.

```cpp theme={null}
MLVideoQNNBin   qnn_bin("mlbin");     // qtimlvideoqnnbin
MLVideoSNPEBin  snpe_bin("mlbin");    // qtimlvideosnpebin
MLVideoONNXBin  onnx_bin("mlbin");    // qtimlvideoonnxbin
```

**APIs**

The table below applies to each of `MLVideoONNXBin`, `MLVideoQNNBin`, `MLVideoSNPEBin`, and `MLVideoTFLiteBin`; substitute the concrete class name for `<Bin>`.

| `qti::<Bin>` | Description |
| - | - |
| `<Bin>(const std::string &name = {})` | Creates the corresponding `qtimlvideo*bin` element with an optional instance name. |
| `<Bin> &set(String prop, Value value, ...)` | Sets one or more properties, same as `Element::set()`. Properties are stage-prefixed: `preprocess-*`, `inference-*`, `postprocess-*`. |
| `<Bin> &set_preprocess_handler(TensorsPreprocessCallback handler)` | Registers the external pre-process callback. Requires `preprocess-engine=none` to take effect. |
| `<Bin> &set_postprocess_handler(Callback handler)` | Registers the external post-process callback. Accepts the same task-specific callback types as `MLPostprocess::set_handler()`. |

<Note>
  The bins cannot be copied, only moved — the same as the other element wrappers.
</Note>

### Buffer

**Purpose**

Unified buffer abstraction over native GStreamer memory. Provides read/write access, timestamp metadata (pts, dts, duration), resize support, and zero-copy transfer between AppSink and AppSrc.

**Key Responsibilities**

* Own and manage mapping/unmapping of underlying `GstBuffer` or `GstSample`.
* Provide read/write access to buffer memory.
* Support resizing and writable allocation for AppSrc.
* Carry timestamps (PTS/DTS/duration) and apply them back to `GstBuffer`.
* Enable zero-copy extraction using `take_gst_buffer()`.

```cpp theme={null}
appsink.set_buffer_consumer([](Buffer buf) {
    auto* data = buf.data();
    auto  pts  = buf.pts();
});
```

A `Buffer` arrives either from an `AppSink` — wrapping the sample the pipeline just produced — or is allocated by the application to be filled and pushed through an `AppSrc`. Which of the two it is determines whether the payload is writable: samples coming out of the pipeline are read-only.

**APIs**

| `qti::Buffer` | Description |
| - | - |
| `Buffer()` | Creates an empty wrapper, referencing no data. `valid()` returns `false` until it is filled. |
| `Buffer(size_t size)` | Allocates a writable buffer with a payload of `size` bytes, for pushing into an `AppSrc`. |
| `static Buffer from_readable_sample(void *gst_sample_opaque)` | Wraps an existing `GstSample` and exposes its payload and metadata as a `Buffer`. |
| `uint8_t *data()` / `const uint8_t *data() const` | Returns a pointer to the payload. The non-const overload requires a writable buffer. |
| `size_t size() const` | Returns the payload size in bytes. |
| `void resize(size_t n)` | Resizes the payload storage. |
| `void set_pts(uint64_t ns)` / `void set_dts(uint64_t ns)` / `void set_duration(uint64_t ns)` | Sets the presentation timestamp, decode timestamp, or duration, in nanoseconds. |
| `uint64_t pts() const` / `uint64_t dts() const` / `uint64_t duration() const` | Reads the corresponding timestamp field, in nanoseconds. |
| `bool is_writable() const` | Returns `true` when the underlying memory can be modified in place. |
| `bool is_readonly() const` | Returns `true` when the buffer came from an `AppSink` and its memory cannot be modified. |
| `bool valid() const` | Returns `true` when the wrapper currently references valid data. |
| `void *take_gst_buffer()` | Transfers ownership of the underlying `GstBuffer` to the caller and clears this wrapper. This is the zero-copy hand-off path used by `AppSrc::push_buffer()`. |
| `void refill_for_appsrc(size_t n)` | Prepares the buffer's storage for reuse inside an `AppSrc` producer callback, resizing it to `n` bytes. |

<Note>
  Like the element wrappers, a `Buffer` cannot be copied, only moved — ownership of the underlying memory is unique. Pass it by reference, or `std::move()` it where ownership has to change, as in the `AppSink` → `AppSrc` bridge above.
</Note>

### StreamFilter

**Purpose**

StreamFilter describes the characteristics of a media stream between two pipeline stages so they can negotiate concrete parameters rather than relying on defaults. Without explicit parameters, adjacent components may fall back to suboptimal formats, trigger implicit conversions, or produce mismatched configurations.

Specialized subtypes provide fluent, type-safe setters for each domain. Raw caps strings are also accepted for cases not covered by typed setters or when copying directly from gst-launch output.

**When to Use StreamFilter**

* When two adjacent stages can interoperate in multiple ways, and you want to select explicit parameters instead of relying on defaults.
* When you need to optimize latency or performance by avoiding implicit conversions (e.g., choose a hardware-friendly pixel format).
* When you want predictable behavior across platforms (e.g., lock color range/primaries/transfer/matrix).
* When you integrate tensors for AI/ML and must precisely specify dimensions and element types.
* When you need a human-readable summary of the stream parameters for logs and debugging.

A `StreamFilter` is not an `Element` and is never added to a pipeline with `add()`. It is always attached between stages with `Pipeline::add_stream_filter(unique_name, filter)`, which internally creates a `capsfilter` element and applies the filter's caps to it. The base `StreamFilter` class itself only exposes a raw-caps-string constructor and `to_string()`; the fluent per-domain setters (`.format()`, `.resolution()`, `.framerate()`, ...) live on the typed subclasses such as `VideoFilter`.

**Usage**

**Typed subclass**

```cpp theme={null}
Pipeline pipeline("video-pipeline");
pipeline.add("videotestsrc", "src", "pattern", "ball")
        .add_stream_filter("filter", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
        .add("waylandsink", "sink")
        .execute();
```

**Raw caps string**

```cpp theme={null}
Pipeline pipeline("video-pipeline-streamfilter-string");
pipeline.add("videotestsrc", "src", "pattern", "ball")
        .add_stream_filter("filter", qti::StreamFilter(
            "video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1"))
        .add("waylandsink", "sink")
        .execute();
```

**APIs**

| `qti::StreamFilter` | Description |
| - | - |
| `StreamFilter()` | Creates an empty filter expression. |
| `StreamFilter(const std::string &caps_or_mediatype)` | Creates a filter from a raw caps or media-type string, e.g. `"video/x-raw,format=NV12"`. |
| `std::string to_string() const` | Serializes the current filter expression to a caps string, for logging/debugging. |

### VideoFilter

**Purpose**

Configures a raw video stream (YUV, RGB, Bayer) between a camera source and any downstream processing or display element. Supports format, resolution, framerate, colorimetry, interlace mode, and pixel aspect ratio.

**Key Responsibilities**

* Produce caps with standard raw video attributes: format, resolution, framerate, colorimetry, interlace mode, pixel aspect ratio, etc.

**APIs**

| Member | Description |
| - | - |
| `format(const std::string &fmt)` / `format(Format fmt)` | Sets the stream's pixel format, for example `NV12`, `YUY2`, `RGB`, `RGBA`. Determines how pixel data is laid out in memory; essential for hardware compatibility and color-conversion paths. Accepts a string or a `VideoFilter::Format` enumerator. |
| `resolution(int width, int height)` | Specifies the frame dimensions in pixels. |
| `framerate(int num, int den = 1)` / `framerate(float num)` | Sets the frame rate as a fraction, for example `30/1` or `30000/1001` frames-per-second. Pass a single value when you do not need a fractional frame rate, like `30.0` fps. |
| `colorimetry(const std::string &value)` | Defines the colorimetry standard, for example `bt601`, `bt709`, `bt2020`. |
| `range(const std::string &value)` | Selects the quantization range, `full` or `limited`. |
| `interlace(const std::string &mode)` | Sets the interlacing mode, for example `progressive` or `interleaved`. Progressive is standard for modern pipelines; interlaced is still used in some camera/legacy systems. |
| `pixel_aspect_ratio(int num, int den = 1)` | Specifies the pixel aspect ratio, also called PAR. Normally `1/1`; used for anamorphic or non-square-pixel formats. |
| `add(const std::string &expr)` | Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API. |

`VideoFilter::Format` enumerates the raw video formats (`video/x-raw`), `JPEG` (`image/jpeg`), and the Bayer patterns (`video/x-bayer`: `BGGR`, `RGGB`, `GBRG`, `GRBG`, `MONO`).

**Usage**

We can achieve the same results in three different ways:

**Typed (enum) API** - if you want compile-time safety, IDE autocompletion, and guardrails against typos. Ideal for SDK consumers and most production code.

```cpp theme={null}
auto v1 = VideoFilter()
                .format(VideoFilter::Format::NV12)
                .resolution(1920, 1080)
                .framerate(30)
                .colorimetry("bt709")
                .range("limited")
                .interlace("progressive");
```

**String API** - if you need quick iteration or to pass values not (yet) covered by enums — e.g., enabling a freshly added GStreamer value on a target platform without immediately updating headers.

```cpp theme={null}
auto v2 = VideoFilter()
                .format("NV12")
                .resolution(1920, 1080)
                .framerate(30)
                .colorimetry("bt709")
                .range("limited")
                .interlace("progressive");
```

**Raw caps string** - very useful for setting arbitrary stream (caps) filter or you want to lift an existing caps line from logs, gst-launch, or a pad template and drop it as-is — great for debugging, prototyping, or reproducing pipelines exactly.

```cpp theme={null}
auto v3 = StreamFilter("video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1,colorimetry=bt709,range=limited");
```

### H264Filter

**Purpose**

Configures an encoded H.264 stream between an encoder and a downstream parser or muxer. Supports resolution, framerate, stream-format, alignment, profile, and level.

**Key Responsibilities**

* Specify resolution, framerate, stream-format, alignment, profile, level.
* Insert `codec_data` buffers.
* Attach arbitrary name/value string properties.

**APIs**

| Member | Description |
| - | - |
| `resolution(int width, int height)` | Specifies the frame dimensions in pixels. |
| `framerate(int num, int den = 1)` / `framerate(float num)` | Sets the frame rate as a fraction, or as a single value when a fractional rate is not needed. |
| `profile(const std::string &p)` | Sets the H.264 profile caps field, for example `baseline`, `main`, `high`. |
| `level(const std::string &lvl)` | Sets the H.264 level caps field. |
| `stream_format(const std::string &f)` | Sets the H.264 stream-format caps field, for example `byte-stream`, `avc`. |
| `alignment(const std::string &a)` | Sets the H.264 alignment caps field, for example `au`, `nal`. |
| `codec_data(const std::string &bytes)` | Sets the H.264 `codec_data` caps field from a byte string. |
| `set(const std::string &key, const std::string &val)` | Shorthand for `add(key + "=" + val)`. |
| `add(const std::string &expr)` | Appends an arbitrary custom caps field using a `key=value` expression. |

**Usage**

```cpp theme={null}
auto h264f = H264Filter().resolution(1920, 1080)
                         .framerate(30)
                         .profile("baseline")
                         .stream_format("byte-stream")
                         .alignment("au");

Pipeline pipeline("h264-pipeline");
pipeline.add_stream_filter("h264f", h264f);
```

### TensorFilter

**Purpose**

Configures an ML tensor stream with explicit data type and shape (e.g., UINT8, NHWC). Not needed in a full inference pipeline — when preprocess, inference, and postprocess elements are all present, tensor parameters are negotiated automatically. Use TensorFilter when:

* The preprocessing stage does not have model dimension information

* You need exact tensor data output from the inference plugin

**Key Responsibilities**

* Declare element type (e.g., `UINT8`, `FLOAT32`) for tensors.
* Specify shape (dimensions). Supports:
  * Single tensor: a 1D list (e.g., `1, 520, 520, 3`).
  * Multiple tensors: array-of-arrays (e.g., `{ {1,3,H,W}, {1,32} }`) when needed.

**APIs**

| Member | Description |
| - | - |
| `type(const std::string &t)` / `type(Type t)` | Specifies the tensor data type, for example `UINT8` or `FLOAT32`. Accepts a string or a `TensorFilter::Type` enumerator (`UINT8`, `UINT16`, `UINT32`, `INT8`, `INT16`, `INT32`, `FLOAT16`, `FLOAT32`). |
| `dimensions(UInts... dims)` | Specifies a single tensor's dimensions as a variadic list of integers, for example `dimensions(1, 520, 520, 3)`. |
| `dimensions(const std::vector<int> &one_tensor_dims)` | Specifies a single tensor's dimensions as a vector. |
| `dimensions(const std::vector<std::vector<int>> &many_tensors_dims)` | Specifies several tensors at once, one inner vector per tensor. |
| `add(const std::string &expr)` | Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API. |

**Usage**

We can achieve the same results in three different ways:

**Typed (enum) API** - if you want compile-time safety, IDE autocompletion, and guardrails against typos. Ideal for SDK consumers and most production code.

```cpp theme={null}
auto t1 = TensorFilter()
                .type(TensorFilter::Type::UINT8)
                .dimensions(1, 520, 520, 3);      // NHWC, for example
```

**String API** - if you need quick iteration or to pass values not (yet) covered by enums — e.g., enabling a freshly added GStreamer value on a target platform without immediately updating headers.

```cpp theme={null}
auto t2 = TensorFilter()
                .type("FLOAT32")
                .dimensions(1, 3, 640, 640);      // NCHW, for example
```

**Raw caps string** - very useful for setting arbitrary stream (caps) filter or you want to lift an existing caps line from logs, gst-launch, or a pad template and drop it as-is — great for debugging, prototyping, or reproducing pipelines exactly.

```cpp theme={null}
auto t3 = StreamFilter("neural-network/tensors,type=UINT8,dimensions=<<1,520,520,3>>");
```

### TextFilter

**Purpose**

Configures a raw text stream (text/x-raw) for pipelines that carry textual data such as metadata, subtitles, or logs. Minimal by design — attach only the fields you need using .add("key=value").

**Key Responsibilities**

* Provide a lightweight caps descriptor for text (`text/x-raw`) without enforcing a schema.
* Let callers attach arbitrary properties (e.g., encoding/charset, language, packetization hints) using `.add("key=value")`.

**APIs**

| Member | Description |
| - | - |
| `add(const std::string &expr)` | Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API. |

**Usage**

```cpp theme={null}
auto txt = TextFilter().add("charset=utf8");
```

### ImageFilter

**Overview**

Used exclusively with CamSrc (qtiqmmfsrc) to enable and link its dedicated image capture pad. Without ImageFilter, the image pad is not requested, not activated, and produces no buffers. It is a structural enabler, not a processing stage — it is not intended for general image processing or frame extraction from video sources.

`ImageFilter` is used to:

* Enable the image output pad exposed by `CamSrc` (`qtiqmmfsrc`)
* Attach that pad to the pipeline so the source can activate image capture
* Provide a concrete sink target required by GStreamer for pad activation

It is not intended for:

* Generic image processing
* Frame extraction from arbitrary video sources
* Replacement of `VideoFilter` or `AppSink`

**Relationship to CamSrc (qtiqmmfsrc)**

* `CamSrc` (`qtiqmmfsrc`) exposes a dedicated image pad that remains inactive unless linked.
* `ImageFilter` explicitly exists to satisfy this requirement.
* Without `ImageFilter`, the image pad:
  * Will not be requested
  * Will not be activated
  * Will not produce buffers

This makes `ImageFilter` a structural enabler, not a processing stage.

**APIs**

| Member | Description |
| - | - |
| `format(const std::string &fmt)` | Sets the image format for the capture pad, for example `JPEG`. |
| `resolution(int width, int height)` | Specifies the captured image dimensions in pixels. |
| `framerate(int num, int den = 1)` / `framerate(float num)` | Sets the capture pad's frame rate. |
| `add(const std::string &expr)` | Appends an arbitrary custom caps field using a `key=value` expression. |

**Usage**

```cpp theme={null}
auto imagefilter1 = ImageFilter()
             .format("JPEG")
             .resolution(3840, 2160);
```

### AudioFilter

**Purpose**

Configures a raw PCM audio stream with sample format, channel count, sample rate, and layout. Use this to ensure audio converters and sinks negotiate exactly the format your hardware supports rather than inserting an implicit resampler.

**Key Responsibilities**

* Set audio sample format (e.g., `S16LE`, `F32LE`) to match HW/SW capabilities.
* Define channels and rate for correct resampling/mixing.
* Choose layout (`interleaved` vs `non-interleaved`) to align with downstream expectations.

**APIs**

| Member | Description |
| - | - |
| `format(const std::string &fmt)` / `format(Format f)` | Sets the audio sample format, for example `S16LE`, `F32LE`. Accepts a string or an `AudioFilter::Format` enumerator. |
| `channels(int n)` | Specifies the number of audio channels. |
| `rate(int hz)` | Specifies the audio sample rate in Hz. |
| `layout(const std::string &l)` / `layout(Layout l)` | Specifies the audio layout. Accepts a string or `AudioFilter::Layout::INTERLEAVED` / `NON_INTERLEAVED`. |
| `add(const std::string &expr)` | Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API. |

`AudioFilter::Format` covers the signed/unsigned integer formats (`S8` through `U32BE`, including the packed 18/20/24-bit variants) and the float formats `F32LE`, `F32BE`, `F64LE`, `F64BE`.

**Usage**

We can achieve the same results in three different ways:

**Typed (enum) API** - if you want compile-time safety, IDE autocompletion, and guardrails against typos. Ideal for SDK consumers and most production code.

```cpp theme={null}
auto a1 = AudioFilter()
                .format(AudioFilter::Format::S16LE)
                .rate(48000)
                .channels(2)
                .layout(AudioFilter::Layout::INTERLEAVED);
```

**String API** - if you need quick iteration or to pass values not (yet) covered by enums — e.g., enabling a freshly added GStreamer value on a target platform without immediately updating headers.

```cpp theme={null}
auto a2 = AudioFilter()
                .format("F32LE")
                .rate(44100)
                .channels(1)
                .layout("non-interleaved");
```

**Raw caps string** - very useful for setting arbitrary stream (caps) filter or you want to lift an existing caps line from logs, gst-launch, or a pad template and drop it as-is — great for debugging, prototyping, or reproducing pipelines exactly.

```cpp theme={null}
auto a3 = StreamFilter("audio/x-raw,format=S16LE,rate=48000,channels=2,layout=interleaved");
```

### Logging

The SDK can either simplify GStreamer's log stream into a compact, uniform format or step aside and let GStreamer log raw, with `GST_DEBUG` fully in control. Both modes, the verbosity levels, and the ordering constraint on configuring them are covered in [Logging and Diagnostics](/app-builder/cpp-logging).

## QIM SDK Nodes

All nodes are executed by GStreamer elements under the hood. Most elements are handled through the `Element` base class. However, elements that provide additional functionality — such as application buffer interfaces in appsink and appsrc, or image-capture signals in camera sources like `qtiqmmfsrc` — require extended interfaces.

If you do not need these extra capabilities (for example, image capture), you can still use these elements via the generic `Element` base class without any limitations for standard data processing.

Below is a table listing all supported elements and the corresponding QIM SDK wrapper classes used to work with them.

**Sources**

| GStreamer | QIM SDK C++ Class |
| - | - |
| `qtiqmmfsrc` | `qti::CamSrc` (also usable as `qti::Element`) |
| `v4l2src` | `qti::Element` |
| `filesrc` | `qti::Element` |
| `multifilesrc` | `qti::Element` |
| `appsrc` | `qti::AppSrc` |
| `appsink` | `qti::AppSink` |
| `qtisocketsrc` | `qti::Element` |

**AI**

| GStreamer | QIM SDK C++ Class |
| - | - |
| `qtimlaconverter` | `qti::Element` |
| `qtimlvconverter` | `qti::MLVConverter` (also usable as `qti::Element`) |
| `qtimlqnn` | `qti::Element` |
| `qtimlsnpe` | `qti::Element` |
| `qtimltflite` | `qti::Element` |
| `qtimlpostprocess` | `qti::MLPostprocess` (also usable as `qti::Element`) |
| `qtimlvideoonnxbin` | `qti::MLVideoONNXBin` (also usable as `qti::Element`) |
| `qtimlvideoqnnbin` | `qti::MLVideoQNNBin` (also usable as `qti::Element`) |
| `qtimlvideosnpebin` | `qti::MLVideoSNPEBin` (also usable as `qti::Element`) |
| `qtimlvideotflitebin` | `qti::MLVideoTFLiteBin` (also usable as `qti::Element`) |
| `qtimlaic` | `qti::Element` |
| `qtimetamux` | `qti::Element` |
| `qtimlmetaextractor` | `qti::Element` |
| `qtimlmetaparser` | `qti::Element` |
| `qtimldemux` | `qti::Element` |
| `qtimetatransform` | `qti::Element` |
| `mlonnx` | `qti::Element` |

**Multimedia**

| GStreamer | QIM SDK C++ Class |
| - | - |
| `qtiobjtracker` | `qti::Element` |
| `qtismartvencbin` | `qti::Element` |
| `qtivoverlay` | `qti::Element` |
| `qtivsplit` | `qti::Element` |
| `qtivcomposer` | `qti::Element` |
| `qtibatch` | `qti::Element` |
| `qticamreproc` | `qti::Element` |
| `qtic2adec` | `qti::Element` |
| `qtic2aenc` | `qti::Element` |
| `qtic2vdec` | `qti::Element` |
| `qtic2venc` | `qti::Element` |
| `qtijpegenc` | `qti::Element` |
| `qtivtransform` | `qti::Element` |
| `qticvimgpyramid` | `qti::Element` |
| `qticvoptclflow` | `qti::Element` |
| `qtidfs` | `qti::Element` |
| `v4l2h264dec` | `qti::Element` |
| `v4l2h264enc` | `qti::Element` |
| `v4l2h265dec` | `qti::Element` |
| `v4l2h265enc` | `qti::Element` |
| `tee` | `qti::Element` |
| `h264parse` | `qti::Element` |
| `rtpptdemux` | `qti::Element` |
| `rtph264depay` | `qti::Element` |
| `queue` | `qti::Element` |
| `qtdemux` | `qti::Element` |
| `tsdemux` | `qti::Element` |

## Common Application Design

### Create pipeline with config file

The SDK supports fully declarative pipeline construction using YAML. A YAML pipeline describes elements and links between them, which are translated at runtime into a GStreamer pipeline.

**TopLevel Structure**

A YAML pipeline file has a single top-level key:

```yaml theme={null}
pipeline:
  elements: [...]
  links: [...]
```

**Elements Section**

`elements` is an ordered list of pipeline elements. Each element entry defines:

* The GStreamer element type
* A unique name used for linking
* Optional properties
* Optional filter configuration

**Common Fields**

```yaml theme={null}
- type: <element-type>
  name: <unique-name>
  <property>: <value>
```

* `type`: GStreamer element factory name, or `filter`
* `name`: Unique identifier within the pipeline
* properties: Any remaining key/value pairs are passed directly to the element as GStreamer properties

**Filter Elements**

Filters are declared using `type: filter` and are further specialized by their configuration block.

**Stream Filter**

```yaml theme={null}
- type: capsfilter
  name: raw_caps
  caps: "video/x-raw,format=NV12,width=1280,height=720,framerate=30/1"
```

**Video Filter**

```yaml theme={null}
- type: filter
  name: vf
  video:
    format: NV12
    width: 1920
    height: 1080
    framerate: 30
```

**Links Section**

`links` defines explicit connection paths between elements.

Each entry is an ordered list of element names. Elements are linked sequentially from left to right.

```yaml theme={null}
links:
  - [src, demux, parse, decoder, vf, split]
```

**Example yaml files**

**YAML config file**

```yaml theme={null}
pipeline:
  elements:
    - type: qtiqmmfsrc
      name: source
      camera: "0"

    - type: filter
      name: filter
      video:
        format: NV12
        width: 1920
        height: 1080
        framerate: 30.0

    - type: tee
      name: split

    - type: qtimlvconverter
      name: preprocessing

    - type: qtimltflite
      name: inferencing
      delegate: gpu
      model: /etc/models/model.tflite

    - type: qtimlpostprocess
      name: postprocessing

    - type: qtimetamux
      name: mlmuxer

    - type: qtivoverlay
      name: overlay

    - type: waylandsink
      name: display
      sync: true
      async: true
      fullscreen: true

  links:
    - [source, filter, split, preprocessing, inferencing, postprocessing, mlmuxer, overlay, display]
    - [split, mlmuxer]
```

### Multi-Model Daisy Chain

Below is shown a logical diagram of an AI daisy chain pipeline with sequential and parallel inferencing.

The same could be handled by a straight sequential pipeline of inference bins. But we must set manually the source for every inference bin.

<img alt="Introduction" />

<Accordion title="Try me">
  <Note>
    Several `.add(...)` calls in this example had missing/mismatched quotes in the source PDF (e.g. a stray closing paren instead of a closing quote, or a missing leading quote). These have been corrected below where unambiguous. One inline comment truncated at the page margin was completed using the identical sentence found in the prose above.
  </Note>

  ```cpp theme={null}
  Element ml_car("qtimlvideotflitebin", "car")
    .add("performance-profile", "high-performance")
    .add("inference-delegate", "dsp")
    .add("inference-model", "/opt/car_detect.dlc")
    .add("inference-layers", "< /model.22/Mul_2, /model.22/Sigmoid >")
    .add("postprocess-threshold", 60.0)
    .add("postprocess-results", 4)
    .add("postprocess-module", "yolov8")
    .add("postprocess-labels", "/opt/car_detect_model.labels");
    // The same could be handle by straight sequential pipeline of inference bins. But we must set manually source to every inference bin.

  Element ml_plate("qtimlvideotflitebin", "plated")
    .add("source", 1)
    .add("image-disposition", "centre")
    .add("performance-profile", "high-performance")
    .add("inference-delegate", "dsp")
    .add("inference-model", "/opt/plate_detect.dlc")
    .add("inference-layers", "< /model.22/Mul_2, /model.22/Sigmoid >")
    .add("postprocess-threshold", 40.0)
    .add("postprocess-results", 1)
    .add("postprocess-module", "yolov8")
    .add("postprocess-labels", "/opt/plate_detect_model.labels");

  Element ml_anpr("qtimlvideotflitebin", "anpr")
    .add("source", 2)
    .add("image-disposition", "centre")
    .add("inference-delegate", "dsp")
    .add("inference-model", "/opt/anpr.dlc")
    .add("inference-layers", "< /decode_head2/classifier/Conv, /decode_head/classifier/Conv >")
    .add("postprocess-threshold", 15.0)
    .add("postprocess-results", 1)
    .add("postprocess-module", "anpr")
    .add("postprocess-labels", "/opt/anpr_model.labels");

  Element ml_siglip("qtimlvideotflitebin", "siglip")
    .add("source", 1)
    .add("image-disposition", "centre");
    .add("inference-delegate", "dsp");
    .add("inference-model", "/opt/car_make_model.dlc");
    .add("inference-layers", "< /classifier/Gemm >");
    .add("postprocess-threshold", 15.0);
    .add("postprocess-results", 1);
    .add("postprocess-module", "mobilenet");
    .add("postprocess-labels", "/opt/car_make_model.labels");

  Pipeline pipeline("SmartVehicleIdentification");
  pipeline.add("camsrc", "source", "camera", "0")
              .add_stream_filter("filter", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
              .add(ml_car)
              .add(ml_plate)
              .add(ml_anpr)
              .add(ml_siglip)
              .add("qtivoverlay", "overlay")
              .add("waylandsink", "display", "sync", "true", "async", "true", "fullscreen")
              .execute();
  ```
</Accordion>

### Custom Postprocessing with YOLOv8 (Detection)

QIM SDK implements AI model post-processing through dedicated modules that take tensors as input and produce predictions. All remaining complexity — such as batching, daisy-chaining, image mask support — is encapsulated within the post-processing plugin. This makes the post-processing module API an ideal integration point for application-specific post-processing logic. As a result, applications do not need to handle batching, chaining, image masks, or ML metadata themselves. We will implement a specialized post-processing module to manage bindings with the application.

<img alt="Introduction" />

Custom postprocessing callback is attached to a [`qtimlpostprocess`](/plugin-reference/qtimlpostprocess) element via [`MLPostprocess::set_handler(...)`](/plugin-reference/qtimlpostprocess).

Typical pipeline segment:

1. Preprocess frame/tensors ([`qtimlvconverter`](/plugin-reference/qtimlvconverter))
2. Run ML inference ([`qtimltflite`](/plugin-reference/qtimltflite))
3. Convert raw tensors in custom callback ([`qtimlpostprocess`](/plugin-reference/qtimlpostprocess))

**Supported callback types**

* `ClassificationPostprocessCallback`
* `ObjectDetectionPostprocessCallback`
* `PoseEstimationPostprocessCallback`
* `DepthEstimationPostprocessCallback`
* `SegmentationPostprocessCallback`
* `TensorsPostprocessCallback`

**Minimal usage**

```cpp theme={null}
MLPostprocess postprocessing("postprocessing");
postprocessing.set_handler(my_callback);

pipeline.add("qtimltflite", "inferencing", ...)
 .add(postprocessing)
 .add_stream_filter("mlf", TextFilter())
 ...;
```

**Input and output explained**

`MLFrameView` (read-only model tensors)

Used by classification/detection/pose/depth/segmentation callbacks. Each `TensorView` contains:

* `type`: tensor element type (`MLTensorType`)
* `dimensions`: tensor shape
* `data`: raw tensor memory (`const uint8_t*`)
* `size`: tensor bytes

Common access pattern:

```cpp theme={null}
const auto& t = frame.tensors[0];
const float* data = static_cast<const float*>(t.data);
```

`MLParam` (runtime metadata and context)

`MLParam` carries key/value fields provided by pipeline and preprocessing/postprocess context. Commonly used keys:

* `input-tensor-width`
* `input-tensor-height`
* `input-tensor-region` (mapped to `Region {x, y, width, height}`)

Example:

```cpp theme={null}
float source_width = 0.0f;
float source_height = 0.0f;
Region region;

params.get("input-tensor-width", source_width);
params.get("input-tensor-height", source_height);
params.get("input-tensor-region", region);
```

**Callback signatures**

* Callbacks (`ClassificationPostprocessCallback`, `ObjectDetectionPostprocessCallback`, `PoseEstimationPostprocessCallback`, `DepthEstimationPostprocessCallback`, `SegmentationPostprocessCallback`) fill the provided output metadata container (`MLClassifications`, `MLDetections`, `MLPoses`, `MLDepthMaps`, `MLSegmentations`).
* Tensor callback (`TensorsPostprocessCallback`) fills `MLFrame& output` for downstream tensor stages.
* Input `MLFrameView` is read-only and must not be modified.

**Classification callback**

```cpp theme={null}
using ClassificationPostprocessCallback =
 std::function<bool(const MLFrameView&, const MLParam&, MLClassifications&)>;
```

Output: fill `MLClassifications` with `MLClassification` items. Field meanings:

* `name`: class label text
* `confidence`: prediction confidence
* `color`: optional UI/overlay color
* `extra`: optional metadata map

**Object detection callback**

```cpp theme={null}
using ObjectDetectionPostprocessCallback =
 std::function<bool(const MLFrameView&, const MLParam&, MLDetections&)>;
```

Output: fill `MLDetections` with `MLDetection` items. Field meanings:

* `left`, `top`, `right`, `bottom`: normalized box coordinates
* `name`, `confidence`, `color`: class info
* `landmarks`: optional keypoints
* `extra`: optional metadata map

Implemented example includes confidence thresholding, region coordinate transform, and NMS.

**Segmentation callback**

```cpp theme={null}
using SegmentationPostprocessCallback =
 std::function<bool(const MLFrameView&, const MLParam&, MLSegmentations&)>;
```

Input parameters

* `frame` (`const MLFrameView&`): read-only model output tensors.
* `params` (`const MLParam&`): runtime metadata (for example: `input-tensor-width`, `input-tensor-height`, `input-tensor-region`).
* `segmentations` (`MLSegmentations&`): output container to fill.

Output contract (`MLSegmentation`)

* `labels`: per-pixel semantic labels.
* `colors`: per-pixel colors; index-aligned with `labels`.
* `n_rows`, `n_columns`: segmentation map dimensions.
* `extra`: optional metadata fields.

**Pose estimation callback**

```cpp theme={null}
using PoseEstimationPostprocessCallback =
 std::function<bool(const MLFrameView&, const MLParam&, MLPoses&)>;
```

Output: fill `MLPoses` with `MLPose` items. Field meanings:

* `keypoints`: list of `MLKeypoint` (`name`, `x`, `y`, `confidence`, `color`)
* `links`: list of `MLKeypointLink` describing skeleton connections
* `name`, `confidence`, `extra`: pose-level metadata

**Depth estimation callback**

```cpp theme={null}
using DepthEstimationPostprocessCallback =
 std::function<bool(const MLFrameView&, const MLParam&, MLDepthMaps&)>;
```

Output: fill `MLDepthMaps` with one or more `MLDepthMap` items. Field meanings:

* `values`: per-pixel depth values (`double`)
* `colors`: per-pixel pseudo-color values (`uint32_t`)
* `n_rows`, `n_columns`: depth map shape
* `extra`: optional metadata map

**Tensor-to-tensor callback**

```cpp theme={null}
using TensorsPostprocessCallback =
 std::function<bool(const MLFrameView&, const MLParam&, MLFrame&)>;
```

output is owned and writable; callback typically:

1. validates input tensors from `frame`
2. copies/selects required tensors into output
3. updates output tensor dimensions/order if needed

**Semantic callbacks** (`classification`, `detection`, `pose`, `depth`, `segmentation`)

```cpp theme={null}
bool callback(const MLFrameView& frame,
                         const MLParam& params,
                         OutputType& output)
```

* `frame`: input tensors from inference (read-only)
* `params`: runtime metadata (input sizes, region, etc.)
* `output`: semantic objects to be filled by callback
* return `true` on valid output
* return `false` on validation/decode failure

**Tensor callback**

```cpp theme={null}
bool callback(const MLFrameView& frame,
                         const MLParam& params,
                         MLFrame& output)
```

* `frame`: input tensors from inference (read-only)
* `params`: runtime metadata/context
* `output`: writable output tensors for downstream tensor stage
* return `true` on valid output
* return `false` on validation/decode failure

**Copy-ready callback skeletons**

Classification

```cpp theme={null}
ClassificationPostprocessCallback cb =
         [](const MLFrameView& frame, const MLParam& params,
              MLClassifications& out) {
              // decode tensors -> classes
              return true;
         };
```

Detection

```cpp theme={null}
ObjectDetectionPostprocessCallback cb =
         [](const MLFrameView& frame, const MLParam& params,
              MLDetections& out) {
              // decode tensors -> detections
              return true;
         };
```

Pose estimation

```cpp theme={null}
PoseEstimationPostprocessCallback cb =
         [](const MLFrameView& frame, const MLParam& params,
              MLPoses& out) {
              // decode tensors -> poses
              return true;
         };
```

Depth estimation

```cpp theme={null}
DepthEstimationPostprocessCallback cb =
         [](const MLFrameView& frame, const MLParam& params,
              MLDepthMaps& out) {
              // decode tensors -> depthmaps
              return true;
         };
```

Segmentation

```cpp theme={null}
SegmentationPostprocessCallback cb =
         [](const MLFrameView& frame, const MLParam& params,
              MLSegmentations& out) {
              // decode tensors -> segmentations
              return true;
         };
```

Tensors

```cpp theme={null}
TensorsPostprocessCallback cb =
         [](const MLFrameView& frame, const MLParam& params,
              MLFrame& output) {
              // map/copy selected tensors from frame -> output
              return true;
         };
```

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_detection_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_detection_yolov8)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_detection_yolov8`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
  | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4          <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_external_postprocess_detection_yolov8
      ```

      The application runs inference on the video file and uses the custom, application-defined postprocessing callback to decode detections, which are then overlaid and displayed fullscreen.

      To stop the application, press **CTRL + C**.
    </Step>
  </Steps>
</Accordion>


# Coding Agent
Source: https://imsdkdocs.qualcomm.com/app-builder/cpp-coding-agent

Use the QIM SDK C++ App Builder skill to generate C++ pipeline apps with an AI coding agent.

## What is the C++ App Builder Skill?

The **QIM SDK C++ App Builder** is an AI coding skill that generates C++ applications using the `QIM SDK C++ API`. You describe the pipeline behavior and configuration in natural language, and the agent produces a ready-to-build `main.cc` and `CMakeLists.txt` built on the `QIM SDK C++ API` — along with a `README.md` that documents the information about the generated application.

**Flow**

<img alt="QIM SDK coding agent workflow" />

**What it generates**

For each request, the agent produces an artifact folder named `qimsdk-cpp-<name>/` containing:

* **`main.cc`** — a complete, buildable C++ application using the QIM SDK C++ API, including pipeline construction, element wiring, stream filters, and ML inference configuration
* **`CMakeLists.txt`** — the build script that compiles `main.cc` and links against `qimsdk-app-builder`
* **`README.md`** — a detailed document covering:
  * Purpose and pipeline behavior summary
  * Configuration placeholders (input, model, labels, output paths)
  * A step-by-step pipeline flow (text summary + Mermaid diagram)
  * Steps to compile and run the app on device
* **`A YAML config file`** — only when you request declarative YAML pipeline mode

**Supported use cases:**

| Category | Details |
| - | - |
| **AI pipelines** | Object detection, image classification, semantic segmentation, face detection, pose estimation, super resolution, depth estimation, audio classification, object tracking, gesture recognition |
| **Inputs** | ISP camera, USB camera, file source (MP4/H.264), RTSP stream |
| **Outputs** | HDMI display, encoded file (MP4), RTSP stream, Redis/MQTT metadata |
| **ML backends** | TFLite (HTP/NPU), QNN, SNPE |
| **Pipeline patterns** | Single-stream, multi-stream, daisy-chained models, ML-bin fused inference, custom pre/post-processing, AppSrc/AppSink bridges, YAML config mode |

## Prerequisites

### For generating code

<Note>
  Qualcomm dev kits are not needed for code generation.
</Note>

**Host Machine**: Any PC with internet access

**Coding Agent**: Any AI coding agent installed (Claude Code, Cursor, Codex, etc.) which supports skills — the skills provided are agent-agnostic.

<Note>
  This skill is designed to work well even with low-reasoning / smaller models.

  Eg: A Sonnet 4.5 model is sufficient.
</Note>

### For deploying and running apps

The target devices where QIM SDK is supported (see [QIM SDK Installation Guide](/installation))

## How to Use

### Step 1: Get the skill

Choose one installation method:

<Tabs>
  <Tab title="Install Skill">
    Clone the skill repository and copy the **QIM SDK C++ App Builder** and **qimsdk-deploy** skills into your coding agent's skills directory:

    ```bash theme={null}
    git clone https://github.com/qualcomm/qimsdk-agentic-skills

    # The following commands are for Claude Code.
    # Set this to your coding agent's skills directory.
    cp -r qimsdk-agentic-skills/skills/qimsdk-cpp-app-builder ~/.claude/skills
    cp -r qimsdk-agentic-skills/skills/qimsdk-deploy ~/.claude/skills
    ```

    The skills directory is agent-specific. Check your coding agent's documentation for its skills location.

    Restart or reload your coding agent after installation. Verify that `qimsdk-cpp-app-builder` and `qimsdk-deploy` are available with `/skills`.
  </Tab>

  <Tab title="Install Plugin">
    **Claude Code** — run inside a Claude Code session:

    ```text theme={null}
    /plugin marketplace add https://github.com/qualcomm/qimsdk-agentic-skills.git
    /plugin install qimsdk-agentic-skills@qimsdk-marketplace
    /reload-plugins
    ```

    **Codex** — run in your terminal:

    ```bash theme={null}
    codex plugin marketplace add https://github.com/qualcomm/qimsdk-agentic-skills.git
    codex plugin add qimsdk-agentic-skills@qimsdk-marketplace
    ```
  </Tab>
</Tabs>

***

### Step 2: Generate the application

Describe your pipeline to the agent. It will automatically activate the skill and generate a complete `main.cc`, `CMakeLists.txt`, and `README.md`.

#### Sample Prompt

**Prompt format — describe pipeline behavior and configuration:**

```
Create a QIM SDK C++ app for single-stream YOLOX object detection from an
MP4 file, encoding the result to a file.

## Pipeline Behavior
- Decode an MP4 file using Qualcomm hardware decoder
- Run YOLOX object detection on full frames using TFLite external delegate on HTP/NPU
- Merge metadata with the video stream before overlay
- Overlay bounding boxes and labels
- Encode the overlaid video and save to an output MP4 file

## Configuration
- Input:   ~/Downloads/qimsdk_samples/media/ai_demo_sample.mp4
- Model:   ~/Downloads/qimsdk_samples/models/yolox_w8a8.tflite
- Labels:  ~/Downloads/qimsdk_samples/labels/yolov8.json
- Backend: TFLite external delegate, HTP/NPU
- Confidence: 51.0
- Output:  ~/Downloads/qimsdk_samples/media/obj_detect_out.mp4
```

More sample prompts can be found in the [Available Sample Prompts](#available-sample-prompts) section below.

#### Generated `README.md`

The README documents the generated app in full:

* **Purpose** — a plain-English summary of what the pipeline does
* **Files** — lists `main.cc`, `CMakeLists.txt`, and `README.md` with descriptions
* **Assumptions** — codec format, quantization requirements, camera defaults, output directory pre-conditions
* **Configuration** — all user-supplied paths (input, model, labels, output) with instructions on where to change them
* **Placeholders to Fill** — any values you still need to supply, or a note that none remain when the request was fully specified
* **Pipeline Flow** — a `Text Summary` walkthrough of every element plus a `Mermaid Diagram` of the full pipeline
* **Steps to compile** — how to build the app against the QIM SDK with CMake
* **Steps to Run** — exact commands to run on device, including any setup (e.g. `mkdir -p` for output directories)

<Accordion title="README.md">
  # QIM SDK C++ App — Single-Stream YOLOX Object Detection (MP4 → MP4)

  ## Purpose

  Decode an MP4 file with the Qualcomm hardware decoder, run **YOLOX** object
  detection on full frames using the **TFLite external delegate on the HTP/NPU**,
  merge the detection metadata with the video stream, overlay bounding boxes and
  class labels, then hardware-encode the annotated video to an output MP4 file.
  Headless (no display branch).

  ## Files

  * `main.cc` — the qimsdk C++ pipeline app
  * `CMakeLists.txt` — build target (`qimsdk-cpp-yolox-obj-detect-encode`)
  * `README.md` — this file

  ## Assumptions

  * Input is H.264-in-MP4, decoded with `v4l2h264dec` (`capture-io-mode=4`,
    `output-io-mode=4` — file source decoded through the hardware decoder).
  * YOLOX (`yolox_w8a8.tflite`) uses the `yolov8` postprocess module with the
    `yolov8.json` labels, per the model catalog.
  * Confidence threshold is applied as inline postprocess settings
    `{"confidence": 51.0}`.
  * No display: the annotated stream is normalized back to NV12
    (`render_vf`), encoded (`v4l2h264enc`, `capture/output-io-mode=4/4`, the
    correct pairing for a transform/decoder-produced NV12 buffer), parsed,
    muxed (`mp4mux`), and written to the output MP4 (`filesink`).
  * `pipeline.eos(true)` is set so `mp4mux` finalizes the container on EOS.
  * `HOME` is resolved once in C++ (`std::getenv("HOME")` with an unset/empty
    check) and prepended to every file path, because C++ string literals do not
    expand `$HOME`.

  ## Configuration (fixed constants in `main.cc`)

  | Value | Path |
  | - | - |
  | Input | `$HOME/Downloads/qimsdk_samples/media/ai_demo_sample.mp4` |
  | Model | `$HOME/Downloads/qimsdk_samples/models/yolox_w8a8.tflite` |
  | Labels | `$HOME/Downloads/qimsdk_samples/labels/yolov8.json` |
  | Output | `$HOME/Downloads/qimsdk_samples/media/obj_detect_out.mp4` |

  ## Placeholders to Fill

  None. All paths, model, labels, confidence, and delegate options are concrete.
  `${QIMSDK_BINDIR}` is expected to be defined by the parent SDK build.

  ## Pipeline Flow

  ### Text Summary

  `filesrc` reads the MP4 and `qtdemux` extracts the H.264 elementary stream,
  which `h264parse` prepares and `v4l2h264dec` hardware-decodes. A `queue`
  (`q_dec`) decouples the decoder thread, then a `VideoFilter` (`vf`) constrains
  the stream to NV12. A `tee` (`split`) branches:

  * **Passthrough / video branch** → `qtimetamux` (`metamux`).
  * **AI branch** → `queue` (`q_ai`) → `qtimlvconverter` (`pre`, preprocess to
    model tensor) → `qtimltflite` (`infer`, YOLOX, external delegate, HTP/NPU) →
    `qtimlpostprocess` (`post`, `module=yolov8`, labels, `{"confidence": 51.0}`) →
    `TextFilter` (`mlf`) → `qtimetamux` (`metamux`).

  `qtimetamux` merges the detection metadata back onto the video frames,
  `qtivoverlay` draws the bounding boxes and class labels, a `VideoFilter`
  (`render_vf`) normalizes the overlaid frames to NV12 for the encoder, and the
  stream is hardware-encoded by `v4l2h264enc`, parsed by `h264parse`
  (`h264parser`), muxed by `mp4mux`, and written by `filesink` to the output MP4.

  ### Mermaid Diagram

  ```mermaid theme={null}
  flowchart TD
    SRC[filesrc] --> DEMUX[qtdemux]
    DEMUX --> PARSE[h264parse]
    PARSE --> DEC[v4l2h264dec]
    DEC --> QDEC[queue q_dec]
    QDEC --> NV12[NV12 VideoFilter vf]
    NV12 --> SPLIT[tee split]
    SPLIT -->|passthrough| MUX[qtimetamux metamux]
    SPLIT -->|AI| QAI[queue q_ai]
    QAI --> PRE[qtimlvconverter pre]
    PRE --> INFER[qtimltflite YOLOX HTP]
    INFER --> POST[qtimlpostprocess module=yolov8]
    POST --> TEXT[TextFilter mlf]
    TEXT --> MUX
    MUX --> OVL[qtivoverlay]
    OVL --> RENDER[NV12 VideoFilter render_vf]
    RENDER --> ENC[v4l2h264enc]
    ENC --> PARSEOUT[h264parse h264parser]
    PARSEOUT --> MP4[mp4mux]
    MP4 --> SINK[filesink]
  ```

  ## Steps to Compile

  Yocto: [https://imsdkdocs.qualcomm.com/advanced/yocto-build#steps-to-build-custom-application](https://imsdkdocs.qualcomm.com/advanced/yocto-build#steps-to-build-custom-application)

  ## Steps to Run

  First ensure all referenced files — the input MP4, the `yolox_w8a8.tflite`
  model, and the `yolov8.json` labels — are already present on the device at the
  paths configured in `main.cc`.

  ```bash theme={null}
  # Copy the app to device
  scp qimsdk-cpp-yolox-obj-detect-encode <user>@<device-ip>:~/

  # SSH into device and run
  ssh <user>@<device-ip>
  chmod +x ~/qimsdk-cpp-yolox-obj-detect-encode
  ./qimsdk-cpp-yolox-obj-detect-encode
  ```

  The input MP4 must exist at
  `$HOME/Downloads/qimsdk_samples/media/ai_demo_sample.mp4` on the device, and the
  output directory (`$HOME/Downloads/qimsdk_samples/media/`) must be writable so
  the app can create `obj_detect_out.mp4`. The app runs to end-of-file, finalizes
  the MP4, and exits.
</Accordion>

#### Generated `main.cc`

* The generated application constructs a tee-split pipeline: decoded frames are split into a passthrough branch and an AI branch.
* The AI branch runs `qtimlvconverter` (preprocess) → `qtimltflite` (YOLOX inference via HTP/NPU) → `qtimlpostprocess` → `TextFilter`, then merges back into `qtimetamux` with the passthrough video.
* `qtivoverlay` draws bounding boxes, and the result is hardware-encoded and written to an MP4 file via `filesink`.
* Pipeline construction and `.execute()` live inside `create_and_execute_pipeline()`; `main()` sets up logging, wraps the call in a try/catch, and returns a non-zero exit code on error.
* All paths are `$HOME`-relative and resolved once via a checked `std::getenv("HOME")` helper — C++ string literals do not expand shell variables.

<Accordion title="main.cc">
  ```cpp theme={null}
  #include <iostream>
  #include <cstdlib>
  #include <stdexcept>

  #include <qti/qimsdk.h>

  using namespace qti;

  // C++ string literals never expand $HOME; resolve it once at startup with an
  // explicit unset/empty check so HOME-relative paths fail loudly if HOME is unset.
  const std::string HOME_PATH = [] {
    const char* home = std::getenv("HOME");
    if (home == nullptr || *home == '\0') {
      throw std::runtime_error("HOME is not set; cannot resolve filesystem paths");
    }
    return std::string(home);
  }();

  //  Example pipeline:
  //
  //    filesrc -> qtdemux -> h264parse -> v4l2h264dec -> [vf NV12] -> tee
  //      tee (passthrough) -> qtimetamux -> qtivoverlay -> [render vf NV12] -> encoder -> parser -> muxer -> filesink
  //      tee (AI branch)   -> qtimlvconverter -> qtimltflite (YOLOX, HTP external delegate)
  //                        -> qtimlpostprocess (module=yolov8, labels, confidence=51.0)
  //                        -> [TextFilter] -> qtimetamux
  //
  //  The pipeline reads an MP4/H.264 file, decodes it with the hardware decoder,
  //  runs YOLOX object detection on full frames via the HTP/NPU (TFLite external
  //  delegate), merges the detection metadata back with the video, overlays
  //  bounding boxes and class labels on each frame, encodes the overlaid video
  //  with the hardware H.264 encoder, and writes the result to an output MP4 file.

  void create_and_execute_pipeline() {

    // Reads the input media file as raw bytes.
    Element src("filesrc", "src");
    src.set("location", HOME_PATH + "/Downloads/qimsdk_samples/media/ai_demo_sample.mp4");

    // Extracts elementary streams from the MP4 container.
    // Dynamic pad-added linking is handled internally by the SDK.
    Element demux("qtdemux", "demux");

    // Prepares the H.264 bitstream for the decoder.
    Element parse("h264parse", "parse");

    // Decodes the compressed H.264 stream into raw video frames.
    //
    // The I/O mode is configured to enforce DMA buffer usage,
    // avoiding unnecessary buffer copies.
    Element decoder("v4l2h264dec", "decoder");
    decoder.set("output-io-mode", 4);
    decoder.set("capture-io-mode", 4);

    // Queue immediately after the hardware decoder before any filter/tee.
    Element q_dec("queue", "q_dec");

    // Normalizes decoded output to NV12 before branching/AI preprocessing.
    auto vf = VideoFilter().format("NV12");

    // Splits the normalized video into a passthrough (main video) branch and
    // an AI inference branch.
    Element split("tee", "split");

    // Queue to isolate the AI branch from the tee.
    Element q_ai("queue", "q_ai");

    // Converts raw video frames into normalized tensors for inference.
    Element preprocess("qtimlvconverter", "pre");

    // Runs YOLOX object detection via the TFLite HTP/NPU external delegate.
    Element infer("qtimltflite", "infer");
    infer.set("delegate", "external");
    infer.set("external-delegate-path", "libQnnTFLiteDelegate.so");
    infer.set("external-delegate-options",
               "QNNExternalDelegate,backend_type=htp,log_level=(string)1;");
    infer.set("model", HOME_PATH + "/Downloads/qimsdk_samples/models/yolox_w8a8.tflite");

    // Decodes YOLOX output tensors into bounding boxes + class labels.
    // module=yolov8 is the documented compatibility mapping for YOLOX detection.
    // The confidence threshold (51.0) is applied as inline JSON settings.
    Element post("qtimlpostprocess", "post");
    post.set("module", "yolov8");
    post.set("labels", HOME_PATH + "/Downloads/qimsdk_samples/labels/yolov8.json");
    post.set("settings", "{\"confidence\": 51.0}");

    // Serialized detection metadata bus feeding qtimetamux.
    auto mlf = TextFilter();

    // Synchronizes AI detection metadata with the original video buffer.
    Element metamux("qtimetamux", "metamux");

    // Draws bounding boxes and class labels on the video frame.
    Element overlay("qtivoverlay", "overlay");

    // Encoder requires NV12 raw video input.
    auto render_vf = VideoFilter().format("NV12");

    // Encodes the overlaid raw video frames into H.264.
    //
    // File/decode source path: driver manages both encoder input and output
    // buffers via dmabuf, so both io-modes are 4.
    Element encoder("v4l2h264enc", "encoder");
    encoder.set("output-io-mode", 4);
    encoder.set("capture-io-mode", 4);

    // Parses the encoded H.264 bitstream for muxing.
    Element h264parser("h264parse", "h264parser");

    // Muxes the encoded stream into an MP4 container.
    Element muxer("mp4mux", "muxer");

    // Writes the muxed MP4 stream to the output file.
    Element sink("filesink", "sink");
    sink.set("location", HOME_PATH + "/Downloads/qimsdk_samples/media/obj_detect_out.mp4");

    // Creates the pipeline, adds and links elements, and executes it.
    //
    // Explicit linking is applied for the tee/metamux branches.
    // eos(true) lets mp4mux finalize the container correctly on EOS.
    Pipeline pipeline("qimsdk-cpp-yolox-obj-detect-encode");
    pipeline.add(src)
            .add(demux)
            .add(parse)
            .add(decoder)
            .add(q_dec)
            .add_stream_filter("vf", vf)
            .add(split)
            .add(metamux)
            .add(q_ai)
            .add(preprocess)
            .add(infer)
            .add(post)
            .add_stream_filter("mlf", mlf)
            .add(overlay)
            .add_stream_filter("render_vf", render_vf)
            .add(encoder)
            .add(h264parser)
            .add(muxer)
            .add(sink)
            .eos(true)
            .link("src", "demux", "parse", "decoder", "q_dec", "vf", "split")
            .link("split", "metamux")
            .link("split", "q_ai", "pre", "infer", "post", "mlf", "metamux")
            .link("metamux", "overlay", "render_vf", "encoder", "h264parser", "muxer", "sink")
            .execute();
  }

  int main() {
    // Route GStreamer logs through the IMSDK logger and enable debug output.
    qti::SetImsdkGstLogMode(qti::ImsdkGstLogMode::ImsdkLog);
    qti::SetImsdkLogLevel(qti::ImsdkLogLevel::Debug);

    try {
      create_and_execute_pipeline();
    } catch (const std::exception &ex) {
      std::cerr << "Exception: " << ex.what() << std::endl;
      return 1;
    }

    return 0;
  }
  ```
</Accordion>

#### Generated `CMakeLists.txt`

* Defines a single executable target (`qimsdk-cpp-yolox-obj-detect-encode`) built from `main.cc`.
* Links against `qimsdk-app-builder` — no direct GStreamer package dependency.
* Installs the binary to `${QIMSDK_BINDIR}` (defined by the parent SDK build) with standard executable permissions.

<Accordion title="CMakeLists.txt">
  ```cmake theme={null}
  cmake_minimum_required(VERSION 3.8.2)

  set(TEST_TARGET qimsdk-cpp-yolox-obj-detect-encode)

  add_executable(${TEST_TARGET}
    main.cc
  )

  target_link_libraries(${TEST_TARGET} PRIVATE
    qimsdk-app-builder
  )

  install(
    TARGETS ${TEST_TARGET}
    RUNTIME DESTINATION ${QIMSDK_BINDIR}
    PERMISSIONS OWNER_EXECUTE OWNER_WRITE OWNER_READ
                GROUP_EXECUTE GROUP_READ
                WORLD_EXECUTE WORLD_READ
  )
  ```
</Accordion>

***

### Step 3: Running the applications

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy the assets to the device">
    The input video, model, and labels must be present on the device before running — regardless of how you build and run the app. Copy them to the paths `main.cc` expects:

    ```bash theme={null}
    # Replace <user> and <device-ip> with your device credentials.
    ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"

    # Copy the assets to device (paths must match main.cc)
    scp ai_demo_sample.mp4 <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
    scp yolox_w8a8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
    scp yolov8.json <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
    ```
  </Step>

  <Step title="Build and run the application">
    The C++ app must be compiled before it can run. With the assets in place, you have two options to build and run the agent-generated app:

    **Option A — Use the QIM SDK Deploy skill (Recommended):**

    The `qimsdk-deploy` skill builds the app on your host against the Yocto SDK, then pushes the compiled binary to the device and runs it over SSH. It handles only the app build and binary transfer — the assets from the previous step must already be on the device.

    **Option B — Follow the generated README:**

    Follow the steps in the [`README.md`](#generated-readme-md) to compile the app, then push the binary to the device and run it:

    1. **Compile the app** by following the steps at [Yocto Build](/advanced/yocto-build).

    2. **Push the compiled binary to the device and run it:**

       ```bash theme={null}
       # Copy the built binary to device
       scp qimsdk-cpp-yolox-obj-detect-encode <user>@<device-ip>:~/

       # SSH into device and run
       ssh <user>@<device-ip>
       chmod +x ~/qimsdk-cpp-yolox-obj-detect-encode
       ./qimsdk-cpp-yolox-obj-detect-encode
       ```

    On completion (EOS), the annotated video is written to `~/Downloads/qimsdk_samples/media/obj_detect_out.mp4`.
  </Step>
</Steps>

***

## Available Sample Prompts

| # | Prompt |
| - | - |
| 1 | [Camera YOLOv8 object detection using fused TFLite ML-bin](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-cpp-app-builder/AI_01_mlbin_yolov8_camera_overlay.md) |
| 2 | [Two-stage PPE daisy-chain — person detection → PPE detection → overlay display](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-cpp-app-builder/AI_02_mlbin_ppe_daisy_chain_display.md) |
| 3 | [Object detection with custom C++ postprocess placeholder](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-cpp-app-builder/AI_03_custom_postprocess_detection_placeholder.md) |
| 4 | [Pose estimation with custom C++ postprocess placeholder](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-cpp-app-builder/AI_04_custom_postprocess_pose_placeholder.md) |
| 5 | [Inference app with tensor output via custom callback placeholder](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-cpp-app-builder/AI_05_tensor_dump_callback_placeholder.md) |
| 6 | [YOLOv8 object detection pipeline with generated YAML config](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-cpp-app-builder/AI_06_yaml_yolov8_generated_config.md) |
| 7 | [Object detection with custom C++ preprocess placeholder](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-cpp-app-builder/AI_07_custom_preprocess_detection_placeholder.md) |
| 8 | [Object detection with TFLite ML-bin custom preprocess placeholder](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-cpp-app-builder/AI_08_mlbin_custom_preprocess_detection_placeholder.md) |
| 9 | [Load existing YOLOv8 detection pipeline from external YAML config](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-cpp-app-builder/AI_09_yaml_yolov8_external_config.md) |
| 10 | [Single-stream YOLOX object detection from MP4, encoded to file](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-cpp-app-builder/AI_10_single_stream_object_detection.md) |
| 11 | [AI wall — four parallel AI inference streams composed into a 2×2 grid display](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-cpp-app-builder/AI_11_ai_wall.md) |
| 12 | [Four-stage gesture recognition daisy-chain pipeline](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-cpp-app-builder/AI_12_gesture_recognition.md) |
| 13 | [Audio classification — classify audio events from video and overlay results](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-cpp-app-builder/AI_13_audio_classification.md) |
| 14 | [Two pipelines bridged through application-managed buffers (AppSrc/AppSink)](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-cpp-app-builder/MM_01_appsrc_appsink_bridge.md) |


# Demo Applications
Source: https://imsdkdocs.qualcomm.com/app-builder/cpp-demo-applications

Ready-to-run C++ App Builder demo applications

The table below lists the C++ App Builder demo applications built on top of the [C++ App Builder](/app-builder/cpp-app-builder) SDK.

| # | Application | Source Code | Description |
| - | - | - | - |
| 1 | [Product Counting Application](/blogs/product-counting-application) | [demo\_product\_counting/main.cc](https://github.com/qualcomm/qimsdk/blob/main/cpp/examples/demo-apps/demo_product_counting/main.cc) | A YOLOv8-based object detection pipeline that counts products crossing a center region of interest in real time. |
| 2 | [Smart Board Application](/blogs/smart-board-application) | [demo\_smartboard/main.cc](https://github.com/qualcomm/qimsdk/blob/main/cpp/examples/demo-apps/demo_smartboard/main.cc) | A touchless virtual whiteboard controlled entirely by hand gestures, supporting freehand drawing, shape recognition, erasing, and dragging content via a two-stage palm detection and hand landmark pipeline. |
| 3 | [Touchless Wrist Steering Car Game](/blogs/touchless-wrist-steering-car-game) | [demo\_car\_game/main.cc](https://github.com/qualcomm/qimsdk/blob/main/cpp/examples/demo-apps/demo_car_game/main.cc) | A two-stage palm detection and hand landmark pipeline turns wrist movement into a virtual steering wheel for a real-time, touchless split-screen driving game. |


# Quick start guide
Source: https://imsdkdocs.qualcomm.com/app-builder/cpp-quickstart

A Quick demonstration of C++ SDK for building AI application on Qualcomm platforms

## What is the QIM SDK C++ App Builder?

### Overview

The **QIM SDK C++ App Builder** is a modern C++ SDK for building, running, and managing multimedia and AI pipelines on Qualcomm platforms. It sits as a high-level abstraction layer on top of GStreamer, taking care of framework-level concerns such as element negotiation, pad linking, signal management, event loops, and state transitions, so developers can stay focused on application logic instead of framework boilerplate.

<img alt="Introduction" />

### Key Capabilities

* **No GStreamer expertise required** — the SDK manages all underlying GStreamer mechanics internally, with no compile-time dependency on GStreamer headers.
* **Less code, no GStreamer boilerplate** — a fluent, high-level API bundles away the pipeline construction, element linking, and state management that GStreamer's native API would otherwise require, so a functional camera-to-display pipeline can be written in just a few lines of C++.
* **Custom pre/post-processing** — plug in your own pre- and post-processing logic around the built-in ML bins without writing a custom GStreamer element.
* **AI inference support** — built-in ML bins for TFLite, QNN, SNPE, and ONNX models.
* **Broad multimedia support** — video capture, signal processing, ML inference, overlay rendering, and display output.

### API Reference

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>API</th>
      <th>Description</th>
      <th>API Code Reference</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Pipeline**
      </td>

      <td>
        Builds, links, and manages the full lifecycle of a GStreamer pipeline. Supports both programmatic (C++ fluent API) and declarative (YAML) pipeline definition. Key methods: `add()`, `link()`, `add_stream_filter()`, `execute()`, `prepare()`, `start()`, `stop()`, `get()`.
      </td>

      <td>
        ```cpp theme={null}
        Pipeline p("cam-pipeline");
        p.add("qtiqmmfsrc", "src").add("waylandsink", "sink").execute();
        ```
      </td>
    </tr>

    <tr>
      <td>
        **Element**
      </td>

      <td>
        Generic wrapper around a GStreamer element. Supports fluent property setting, either at creation time or at runtime, to update a live pipeline.
      </td>

      <td>
        ```cpp theme={null}
        Element sink("waylandsink", "display");
        sink.set("fullscreen", true);
        sink.set("sync", false);
        ```
      </td>
    </tr>

    <tr>
      <td>
        **Buffer**
      </td>

      <td>
        Unified buffer abstraction over native GStreamer memory. Provides read/write access, timestamp metadata, and zero-copy transfer paths. Key methods: `data()`, `size()`, `resize()`, `pts()`, `dts()`, `duration()`.
      </td>

      <td>
        ```cpp theme={null}
        sink.setBufferConsumer([](Buffer buf) {
          auto* data = buf.data();
          auto pts = buf.pts();
        });
        ```
      </td>
    </tr>

    <tr>
      <td>
        **AppSrc**
      </td>

      <td>
        App-facing source. Feeds buffers from application code into the pipeline using lambda callbacks or a push API.
      </td>

      <td>
        ```cpp theme={null}
        AppSrc src(pipeline.get("src"));
        src.setBufferProducer([](Buffer& buf) {
          // fill buf.data()
          return true;
        });
        ```
      </td>
    </tr>

    <tr>
      <td>
        **AppSink**
      </td>

      <td>
        App-facing sink. Delivers pipeline output buffers to application code with zero-copy.
      </td>

      <td>
        ```cpp theme={null}
        AppSink sink(pipeline.get("sink"));
        sink.setBufferConsumer([](Buffer buf) {
          // consume buf.data()
        });
        ```
      </td>
    </tr>

    <tr>
      <td>
        **CamSrc**
      </td>

      <td>
        Built-in ISP camera source. Exposes an API for runtime image capture and snapshots.
      </td>

      <td>
        ```cpp theme={null}
        CamSrc cam(pipeline.get("source"));
        cam.captureImage("snapshot.jpg");
        ```
      </td>
    </tr>

    <tr>
      <td>
        **MLPreProcess**
      </td>

      <td>
        Attaches a custom C++ lambda for pre-processing input frames before ML inference.
      </td>

      <td>
        ```cpp theme={null}
        MLPreProcess pre(pipeline.get("preproc"));
        pre.set_handler([](Buffer& buf) {
          // transform input frame
        });
        ```
      </td>
    </tr>

    <tr>
      <td>
        **MLPostProcess**
      </td>

      <td>
        Attaches a custom C++ lambda for post-processing raw inference tensor output.
      </td>

      <td>
        ```cpp theme={null}
        MLPostProcess pp(pipeline.get("postproc"));
        pp.set_handler([](Buffer& buf) {
          // parse inference output
        });
        ```
      </td>
    </tr>

    <tr>
      <td>
        **MLVideoTFLiteBin** / **MLVideoQNNBin** / **MLVideoSNPEBin** / **MLVideoONNXBin**
      </td>

      <td>
        All-in-one preprocess + inference + postprocess bins. Supports TFLite, QNN, SNPE, and ONNX models.
      </td>

      <td>
        ```cpp theme={null}
        pipeline.add("MLVideoTFLiteBin", "ml", "model", "/models/model.tflite", "delegate", "htp");
        ```
      </td>
    </tr>

    <tr>
      <td>
        **VideoFilter** / **AudioFilter** / **ImageFilter** / **H264Filter** / **TensorFilter** / **TextFilter**
      </td>

      <td>
        Fluent caps builders for `add_stream_filter()`. Describes stream characteristics for caps negotiation between pipeline stages.
      </td>

      <td>
        ```cpp theme={null}
        pipeline.add_stream_filter("vf", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30));
        ```
      </td>
    </tr>
  </tbody>
</table>

***

## Build your application

<Note>
  Ensure you have QIM SDK installed. See [QIM SDK Installation Guide](../installation)
</Note>

Before running any of the applications below, create the sample directory tree on the target device:

```bash theme={null}
# ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{media,models,labels}"
```

This is required even for apps that don't download any media, model, or label files (such as the camera capture and recording apps), since they still write their output under this path.

### USB Camera Source

USB (UVC) cameras are handled by the `v4l2src` node. Each USB camera comes with its own format and frame constraints. Because of this, we always place `qtivtransform` right after the USB camera to ensure that the rest of the pipeline receives a hardware-friendly NV12 (Semi-planar YUV420) video format. `qtivtransform` automatically operates in passthrough mode if the USB camera already supports NV12, so you don't need to worry about any performance overhead.

<img alt="Introduction" />

<Tabs>
  <Tab title="Explicit coding style">
    <Info>
      Check application source code on GitHub: [`qimsdk_ref_usb_camera`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_usb_camera)

      Pre-built application on device: `/usr/bin/qimsdk_ref_usb_camera`
    </Info>

    ```cpp theme={null}
    #include <iostream>
    #include <qti/qimsdk.h>

    using namespace qti;

    //  Example pipeline:
    //
    //    source → transform → [videofilter] → display
    //
    //  The pipeline reads frames from a USB (V4L2) camera, rotates them, restricts
    //  the stream to NV12/1080p/30fps, and displays the result through Wayland.
    void create_and_execute_pipeline(const std::string &device) {
      // Captures frames from the USB camera source.
      Element source("v4l2src", "source");
      source.set("device", device);

      // Applies geometric transforms to video frames.
      Element transform("qtivtransform", "transform");

      // Stream filters used in branch links.
      // They define specific stream characteristics from the supported options.
      auto videofilter = VideoFilter().format("NV12").resolution(1920, 1080).framerate(30);

      // Render video stream on display.
      //
      // sync=false disables strict rendering synchronization to the pipeline clock.
      // fullscreen=true renders the output fullscreen on the target display.
      Element display("waylandsink", "display");
      display.set("sync", false);
      display.set("fullscreen", true);

      // Creates the pipeline, adds elements, links them explicitly, and executes it.
      Pipeline pipeline("usb-cam-pipeline");
      pipeline.add(source)
        .add(transform)
        .add_stream_filter("videofilter", videofilter)
        .add(display)
        .link("source", "transform", "videofilter", "display")
        .execute();
    }

    int main(int argc, char **argv) {
      // Route GStreamer logs through the QIM SDK logger and enable debug output.
      qti::SetImsdkGstLogMode(qti::ImsdkGstLogMode::ImsdkLog);
      qti::SetImsdkLogLevel(qti::ImsdkLogLevel::Debug);

      // V4L2 device node to capture from, defaults to /dev/video2. Pass -d/--device to override.
      std::string device = "/dev/video2";
      for (int i = 1; i < argc; ++i) {
        std::string arg = argv[i];
        if ((arg == "-d" || arg == "--device") && i + 1 < argc) {
          device = argv[++i];
        }
      }

      try {
        create_and_execute_pipeline(device);
      } catch (const std::exception &ex) {
        std::cerr << "Exception: " << ex.what() << std::endl;
        return 1;
      }

      return 0;
    }
    ```

    <Steps>
      <Step title="Connect a USB camera">
        Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the pipeline's `device` property if your camera is exposed on a different node).
      </Step>

      <Step title="Run the application">
        ```bash theme={null}
        /usr/bin/qimsdk_ref_usb_camera --input-config /dev/video2
        ```
      </Step>

      <Step title="Expected Output">
        The live camera feed is rotated and rendered fullscreen on the display in NV12 format at 1080p/30fps.

        <img alt="Expected Output" />

        To stop the application, press **CTRL + C**.
      </Step>

      <Step title="Customize application">
        * **Source code:** [`qimsdk_ref_usb_camera`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_usb_camera)
        * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
      </Step>
    </Steps>
  </Tab>

  <Tab title="Implicit coding style">
    <Info>
      Check application source code on GitHub: [`qimsdk_ref_usb_camera_impl`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_usb_camera_impl)

      Pre-built application on device: `/usr/bin/qimsdk_ref_usb_camera_impl`
    </Info>

    ```cpp theme={null}

    #include <iostream>

    #include <qti/qimsdk.h>

    using namespace qti;

    //  Example pipeline:
    //
    //    source → transform → [videofilter] → display
    //
    //  The pipeline reads frames from a USB (V4L2) camera, rotates them, restricts
    //  the stream to NV12/1080p/30fps, and displays the result through Wayland.

    void create_and_execute_pipeline(const std::string &device) {

      // Creates the pipeline, adds and links elements, and executes it.
      //
      // Elements are created on the fly as they are added, and linking is
      // implicit, following the order in which elements are added.
      //
      // sync=false disables strict rendering synchronization to the pipeline clock.
      // fullscreen=true renders the output fullscreen on the target display.
      Pipeline pipeline("usb-cam-pipeline");
      pipeline.add("v4l2src", "source", "device", device)
              .add("qtivtransform", "transform")
              .add_stream_filter("videofilter", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
              .add("waylandsink", "display", "sync", false, "fullscreen", true)
              .execute();
    }

    int main(int argc, char **argv) {
      // Route GStreamer logs through the QIM SDK logger and enable debug output.
      qti::SetImsdkGstLogMode(qti::ImsdkGstLogMode::ImsdkLog);
      qti::SetImsdkLogLevel(qti::ImsdkLogLevel::Debug);

      // V4L2 device node to capture from, defaults to /dev/video2. Pass -d/--device to override.
      std::string device = "/dev/video2";
      for (int i = 1; i < argc; ++i) {
        std::string arg = argv[i];
        if ((arg == "-d" || arg == "--device") && i + 1 < argc) {
          device = argv[++i];
        }
      }

      try {
        create_and_execute_pipeline(device);
      } catch (const std::exception &ex) {
        std::cerr << "Exception: " << ex.what() << std::endl;
        return 1;
      }

      return 0;
    }
    ```

    <Steps>
      <Step title="Connect a USB camera">
        Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the pipeline's `device` property if your camera is exposed on a different node).
      </Step>

      <Step title="Run the application">
        ```bash theme={null}
        /usr/bin/qimsdk_ref_usb_camera_impl --input-config /dev/video2
        ```
      </Step>

      <Step title="Expected Output">
        The live camera feed is rotated and rendered fullscreen on the display in NV12 format at 1080p/30fps.

        <img alt="Expected Output" />

        To stop the application, press **CTRL + C**.
      </Step>

      <Step title="Customize application">
        * **Source code:** [`qimsdk_ref_usb_camera_impl`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_usb_camera_impl)
        * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
      </Step>
    </Steps>
  </Tab>

  <Tab title="YAML config style">
    <Info>
      See the full YAML configuration here: [qimsdk\_ref\_usb\_camera.yaml](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_usb_camera.yaml)

      Check application source code on GitHub: [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)

      Pre-built application on device: `/usr/bin/qimsdk_ref_yml`

      YAML configuration saved on device: `/etc/qimsdk/qimsdk_ref_usb_camera.yaml`
    </Info>

    ```yaml theme={null}
    pipeline:
      elements:
        - type: v4l2src
          name: source
          device: /dev/video2

        - type: qtivtransform
          name: transform

        - type: filter
          name: videostream
          video:
            format: NV12
            width: 1920
            height: 1080
            framerate: 30

        - type: waylandsink
          name: display
          sync: false
          fullscreen: true
    ```

    <Steps>
      <Step title="Connect a USB camera">
        Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the pipeline's `device` property if your camera is exposed on a different node).
      </Step>

      <Step title="Run the application">
        ```bash theme={null}
        qimsdk_ref_yml /etc/qimsdk/qimsdk_ref_usb_camera.yaml
        ```
      </Step>

      <Step title="Expected Output">
        The live camera feed is rotated and rendered fullscreen on the display in NV12 format at 1080p/30fps.

        <img alt="Expected Output" />

        To stop the application, press **CTRL + C**.
      </Step>

      <Step title="Customize application">
        * **Source code:** [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)
        * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
      </Step>
    </Steps>
  </Tab>
</Tabs>

### Object Detection

This pipeline captures from a built-in camera, runs an object detection model through an ML inference bin, renders bounding box overlays on the output frames, and displays the result. The ML bin handles preprocessing, inference, and postprocessing internally. The stream filter pins the input video format expected by the model. The overlay element draws detection results on each frame before display.

<img alt="Introduction" />

<Tabs>
  <Tab title="Explicit coding style">
    <Info>
      Check application source code on GitHub: [`qimsdk_ref_camera_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_camera_yolov8)

      Pre-built application on device: `/usr/bin/qimsdk_ref_camera_yolov8`
    </Info>
  </Tab>
</Tabs>

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOv8 detection model | [Qualcomm AI Hub — YOLOv8](https://aihub.qualcomm.com/iot/models/yolov8_det) | `yolov8_det_quantized.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |

    <Note>
      If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

      Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
    </Note>
  </Step>

  <Step title="Copy the assets to the device">
    The model and labels must be present on the device before running. Copy them to the paths the application expects:

    ```bash theme={null}
    # Replace <user> and <device-ip> with your device credentials.
    # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

    ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"

    # Copy the assets to device
    scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
    scp yolov8.json <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
    ```
  </Step>

  <Step title="Connect an IMX camera">
    This application uses the built-in ISP camera. On platforms without an onboard ISP camera (e.g. IQ9), attach an external IMX camera (e.g. from RB3 Gen2) before running the application — see [ISP Camera (Config #2 / qticamsrc)](/advanced/debugging#isp-camera-config-2--qticamsrc) for the procedure to switch from `libcamera` to `qticamsrc`.
  </Step>

  <Step title="Run the application">
    ```bash theme={null}
    /usr/bin/qimsdk_ref_camera_yolov8
    ```
  </Step>

  <Step title="Expected Output">
    The live camera feed is rendered fullscreen on the display with YOLOv8 bounding boxes and class labels overlaid on each detected object in real time.

    <img alt="Expected Output" />

    To stop the application, press **CTRL + C**.
  </Step>

  <Step title="Customize application">
    * **Source code:** [`qimsdk_ref_camera_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_camera_yolov8)
    * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
  </Step>
</Steps>

### Custom Post-Processing

The QIM SDK implements AI model post-processing through dedicated modules that take tensors as input and produce predictions. All underlying complexity — batching, daisy-chaining, image mask support, and ML metadata handling — is encapsulated within the post-processing plugin. This makes `MLPostProcess` the ideal integration point for application-specific logic, since the application does not need to handle any of that complexity directly. A specialized post-processing module manages the bindings between the pipeline and the application.

<img alt="Introduction" />

<Tabs>
  <Tab title="Explicit coding style">
    <Info>
      Check application source code on GitHub: [`qimsdk_ref_external_postprocess_mlbin_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_mlbin_yolov8)

      Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_mlbin_yolov8`
    </Info>
  </Tab>
</Tabs>

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOv8 detection model | [Qualcomm AI Hub — YOLOv8](https://aihub.qualcomm.com/iot/models/yolov8_det) | `yolov8_det_quantized.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

      Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
    </Note>
  </Step>

  <Step title="Copy the assets to the device">
    The input video, model, and labels must be present on the device before running. Copy them to the paths the application expects:

    ```bash theme={null}
    # Replace <user> and <device-ip> with your device credentials.
    # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

    ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"

    # Copy the assets to device
    scp ai_demo_sample.mp4 <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
    scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
    scp yolov8.json <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
    ```
  </Step>

  <Step title="Run the application">
    ```bash theme={null}
    /usr/bin/qimsdk_ref_external_postprocess_mlbin_yolov8
    ```
  </Step>

  <Step title="Expected Output">
    The video plays back with YOLOv8 bounding boxes and class labels decoded by the custom postprocessing callback, overlaid and rendered fullscreen on the display.

    <img alt="Expected Output" />

    To stop the application, press **CTRL + C**.
  </Step>

  <Step title="Customize application">
    * **Source code:** [`qimsdk_ref_external_postprocess_mlbin_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_mlbin_yolov8)
    * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
  </Step>
</Steps>

***

## Benefits of QIM SDK C++ App Builder

* Significantly reduced source code
* Improved readability
* Internal GStreamer state transition handling
* Hidden End-of-Stream handling
* Automatic error and state event handling
* Automatic resource cleanup
* Simplified AppSink/AppSrc interaction using lambda functions
* SDK-managed buffer mapping and queries
* Simplified buffer access anywhere in the pipeline using wrapped GST probes
* YAML-based pipeline description


# Reference Applications
Source: https://imsdkdocs.qualcomm.com/app-builder/cpp-reference-applications

Reference C++ App Builder pipeline examples

## Multimedia Pipeline

Before running any of the applications below, create the sample directory tree on the target device:

```bash theme={null}
# ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{media,models,labels}"
```

This is required even for apps that don't download any media, model, or label files (such as the camera capture and recording apps), since they still write their output under this path.

### USB Camera Preview

USB (UVC) cameras are handled by the `v4l2src` node. Each USB camera comes with its own format and frame constraints. Because of this, we always place `qtivcomposer` right after the USB camera to ensure that the rest of the pipeline receives a hardware-friendly NV12 (Semi-planar YUV420) video format. However, `qtivcomposer` will automatically operate in passthrough mode if the USB camera already supports NV12, so you don't need to worry about any performance overhead.

<Note>
  Some USB webcams (e.g. Logitech Brio 4K) may render a green/garbled frame at 30fps depending on platform/build. If this happens, lower the pipeline's framerate (e.g. to 15fps) as a workaround.
</Note>

<img alt="Introduction" />

<Accordion title="Try me">
  <Tabs>
    <Tab title="Explicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_usb_camera`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_usb_camera)

        Pre-built application on device: `/usr/bin/qimsdk_ref_usb_camera`
      </Info>

      ```cpp theme={null}
      #include <iostream>
      #include <qti/qimsdk.h>
      using namespace qti;
      //  Example pipeline:
      //
      //    source → transform → [videofilter] → display
      //
      //  The pipeline reads frames from a USB (V4L2) camera, rotates them, restricts
      //  the stream to NV12/1080p/30fps, and displays the result through Wayland.
      void create_and_execute_pipeline(const std::string &device) {
      // Captures frames from the USB camera source.
      Element source("v4l2src", "source");
      source.set("device", device);

      // Applies geometric transforms to video frames.
      Element transform("qtivtransform", "transform");

      // Stream filters used in branch links.
      // They define specific stream characteristics from the supported options.
      auto videofilter = VideoFilter().format("NV12").resolution(1920, 1080).framerate(30);

      // Render video stream on display.
      //
      // sync=false disables strict rendering synchronization to the pipeline clock.
      // fullscreen=true renders the output fullscreen on the target display.
      Element display("waylandsink", "display");
      display.set("sync", false);
      display.set("fullscreen", true);

      // Creates the pipeline, adds elements, links them explicitly, and executes it.
      Pipeline pipeline("usb-cam-pipeline");
      pipeline.add(source)
      .add(transform)
      .add_stream_filter("videofilter", videofilter)
      .add(display)
      .link("source", "transform", "videofilter", "display")
      .execute();
      }
      int main(int argc, char **argv) {
      // Route GStreamer logs through the QIM SDK logger and enable debug output.
      qti::SetImsdkGstLogMode(qti::ImsdkGstLogMode::ImsdkLog);
      qti::SetImsdkLogLevel(qti::ImsdkLogLevel::Debug);
      // V4L2 device node to capture from, defaults to /dev/video2. Pass -d/--device to override.
      std::string device = "/dev/video2";
      for (int i = 1; i < argc; ++i) {
        std::string arg = argv[i];
        if ((arg == "-d" || arg == "--device") && i + 1 < argc) {
          device = argv[++i];
        }
      }
      try {
      create_and_execute_pipeline(device);
      } catch (const std::exception &ex) {
      std::cerr << "Exception: " << ex.what() << std::endl;
      return 1;
      }
      return 0;
      }
      ```

      <Steps>
        <Step title="Connect a USB camera">
          Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the pipeline's `device` property if your camera is exposed on a different node).
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          /usr/bin/qimsdk_ref_usb_camera --input-config /dev/video2
          ```

          The application connects to the USB camera, restricts the stream to NV12/1080p/30fps, and renders it fullscreen on the display.
        </Step>

        <Step title="Expected Output">
          The live USB camera feed is rendered fullscreen on the display.

          <img alt="Expected Output" />

          To stop the application, press **CTRL + C**.
        </Step>

        <Step title="Customize application">
          * **Source code:** [`qimsdk_ref_usb_camera`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_usb_camera)
          * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
        </Step>
      </Steps>
    </Tab>

    <Tab title="Implicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_usb_camera_impl`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_usb_camera_impl)

        Pre-built application on device: `/usr/bin/qimsdk_ref_usb_camera_impl`
      </Info>

      ```cpp theme={null}
      #include <iostream>
      #include <qti/qimsdk.h>
      using namespace qti;
      //  Example pipeline:
      //
      //    source → transform → [videofilter] → display
      //
      //  The pipeline reads frames from a USB (V4L2) camera, rotates them, restricts
      //  the stream to NV12/1080p/30fps, and displays the result through Wayland.
      void create_and_execute_pipeline(const std::string &device) {
      // Creates the pipeline, adds and links elements, and executes it.
      //
      // Elements are created on the fly as they are added, and linking is
      // implicit, following the order in which elements are added.
      //
      // sync=false disables strict rendering synchronization to the pipeline clock.
      // fullscreen=true renders the output fullscreen on the target display.
      Pipeline pipeline("usb-cam-pipeline");
      pipeline.add("v4l2src", "source", "device", device)
      .add("qtivtransform", "transform")
      .add_stream_filter("videofilter", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
      .add("waylandsink", "display", "sync", false, "fullscreen", true)
      .execute();
      }
      int main(int argc, char **argv) {
      // Route GStreamer logs through the QIM SDK logger and enable debug output.
      qti::SetImsdkGstLogMode(qti::ImsdkGstLogMode::ImsdkLog);
      qti::SetImsdkLogLevel(qti::ImsdkLogLevel::Debug);
      // V4L2 device node to capture from, defaults to /dev/video2. Pass -d/--device to override.
      std::string device = "/dev/video2";
      for (int i = 1; i < argc; ++i) {
        std::string arg = argv[i];
        if ((arg == "-d" || arg == "--device") && i + 1 < argc) {
          device = argv[++i];
        }
      }
      try {
      create_and_execute_pipeline(device);
      } catch (const std::exception &ex) {
      std::cerr << "Exception: " << ex.what() << std::endl;
      return 1;
      }
      return 0;
      }
      ```

      <Steps>
        <Step title="Connect a USB camera">
          Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the pipeline's `device` property if your camera is exposed on a different node).
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          /usr/bin/qimsdk_ref_usb_camera_impl --input-config /dev/video2
          ```

          The application connects to the USB camera, restricts the stream to NV12/1080p/30fps, and renders it fullscreen on the display.
        </Step>

        <Step title="Expected Output">
          The live USB camera feed is rendered fullscreen on the display.

          <img alt="Expected Output" />

          To stop the application, press **CTRL + C**.
        </Step>

        <Step title="Customize application">
          * **Source code:** [`qimsdk_ref_usb_camera_impl`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_usb_camera_impl)
          * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
        </Step>
      </Steps>
    </Tab>

    <Tab title="YAML config style">
      <Info>
        See the full YAML configuration here: [qimsdk\_ref\_usb\_camera.yaml](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_usb_camera.yaml)

        Check application source code on GitHub: [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)

        Pre-built application on device: `/usr/bin/qimsdk_ref_yml`

        YAML configuration saved on device: `/etc/qimsdk/qimsdk_ref_usb_camera.yaml`
      </Info>

      ```yaml theme={null}
      pipeline:
        elements:
          - type: v4l2src
            name: source
            device: /dev/video2

          - type: qtivtransform
            name: transform

          - type: filter
            name: videostream
            video:
              format: NV12
              width: 1920
              height: 1080
              framerate: 30

          - type: waylandsink
            name: display
            sync: false
            fullscreen: true
      ```

      <Steps>
        <Step title="Connect a USB camera">
          Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the pipeline's `device` property if your camera is exposed on a different node).
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          qimsdk_ref_yml /etc/qimsdk/qimsdk_ref_usb_camera.yaml
          ```

          The application connects to the USB camera, restricts the stream to NV12/1080p/30fps, and renders it fullscreen on the display.
        </Step>

        <Step title="Expected Output">
          The live USB camera feed is rendered fullscreen on the display.

          <img alt="Expected Output" />

          To stop the application, press **CTRL + C**.
        </Step>

        <Step title="Customize application">
          * **Source code:** [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)
          * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
        </Step>
      </Steps>
    </Tab>
  </Tabs>
</Accordion>

### Built-in Camera Preview

This pipeline includes a built-in camera node, a stream filter, and a display node. If the device has more than one built-in camera, you can specify which one to use by providing the camera's unique ID. The stream filter defines the camera's output format, resolution, and framerate. You can also configure the display settings, such as enabling fullscreen rendering.

<Note>
  IQ9 has no onboard ISP camera. You can attach an external IMX camera (e.g. from RB3 Gen2) instead — see [ISP Camera (Config #2 / qticamsrc)](/advanced/debugging#isp-camera-config-2--qticamsrc) for the procedure to switch from `libcamera` to `qticamsrc`.
</Note>

<img alt="Introduction" />

<Accordion title="Try me">
  <Tabs>
    <Tab title="Explicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_camera`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_camera)

        Pre-built application on device: `/usr/bin/qimsdk_ref_camera`
      </Info>

      ```cpp theme={null}
      #include <iostream>
      #include <qti/qimsdk.h>
      using namespace qti;
      //  Example pipeline:
      //
      //    source → [videostream] → display
      //
      //  The pipeline reads camera frames, runs ML inference and postprocessing,
      //  and displays the result through Wayland.
      void create_and_execute_pipeline() {
      // Captures frames from the camera source.
      Element source("qtiqmmfsrc", "source");

      // Render video stream on display.
      //
      // async=false enforce state transition to ensure the buffers are returned on time.
      // sync=false disables strict rendering synchronization to the pipeline clock.
      // fullscreen=true renders the output fullscreen on the target display.
      Element display("waylandsink", "display");
      display.set("sync", false);
      display.set("fullscreen", true);

      // Stream filters used in branch links.
      // They define specific stream characteristics from the supported options.
      auto videostream = VideoFilter().format("NV12").resolution(1920, 1080).framerate(30);

      // Creates the pipeline, adds and links elements, and executes it.
      //
      // Linking is implicit and follows the order in which elements are added.
      Pipeline pipeline("cam-pipeline");
      pipeline.add(source)
      .add_stream_filter("videostream", videostream)
      .add(display)
      .execute();
      }
      int main() {
      // Route GStreamer logs through the QIMSDK logger and enable debug output.
      qti::SetImsdkGstLogMode(qti::ImsdkGstLogMode::ImsdkLog);
      qti::SetImsdkLogLevel(qti::ImsdkLogLevel::Debug);
      try {
      create_and_execute_pipeline();
      } catch (const std::exception &ex) {
      std::cerr << "Exception: " << ex.what() << std::endl;
      return 1;
      }
      return 0;
      }
      ```
    </Tab>

    <Tab title="Implicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_camera_impl`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_camera_impl)

        Pre-built application on device: `/usr/bin/qimsdk_ref_camera_impl`
      </Info>

      ```cpp theme={null}
      #include <iostream>
      #include <qti/qimsdk.h>
      using namespace qti;
      void create_and_execute_pipeline() {
      Pipeline pipeline("cam-pipeline");
      pipeline.add("qtiqmmfsrc", "source")
      .add_stream_filter("videostream", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
      .add("waylandsink", "display", "sync", false, "fullscreen", true)
      .execute();
      }
      int main() {
      qti::SetImsdkGstLogMode(qti::ImsdkGstLogMode::ImsdkLog);
      qti::SetImsdkLogLevel(qti::ImsdkLogLevel::Debug);
      try {
      create_and_execute_pipeline();
      } catch (const std::exception &ex) {
      std::cerr << "Exception: " << ex.what() << std::endl;
      return 1;
      }
      return 0;
      }
      ```
    </Tab>
  </Tabs>

  <Steps>
    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_camera
      ```

      The application captures live camera frames and renders them fullscreen on the display.
    </Step>

    <Step title="Expected Output">
      A live preview of the camera feed is rendered fullscreen on the display.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_camera`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_camera), [`qimsdk_ref_camera_impl`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_camera_impl)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Built-in Camera Preview with Still Capture

<Note>
  The application captures a still image a few seconds after starting and then stops automatically — this is expected behavior, not a hang.
</Note>

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_camera_and_capture`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_camera_and_capture)

    Pre-built application on device: `/usr/bin/qimsdk_ref_camera_and_capture`
  </Info>

  ```cpp theme={null}
  #include <iostream>
  #include <thread>
  #include <cstdlib>
  #include <qti/qimsdk.h>
  using namespace qti;
  static const std::string home_path =
  std::getenv("HOME") ? std::getenv("HOME") : "";
  //  Example pipeline:
  //
  //    source → [vf] → display → [if] → imagesink
  //
  //  The pipeline reads camera frames, runs ML inference and postprocessing,
  //  and displays the result through Wayland.
  void create_and_execute_pipeline() {
  // Captures frames from the camera source.
  Element source("qtiqmmfsrc", "source");

  // Render video stream on display.
  //
  // async=false enforce state transition to ensure the buffers are returned on time.
  // sync=false disables strict rendering synchronization to the pipeline clock.
  // fullscreen=true renders the output fullscreen on the target display.
  Element display("waylandsink", "display");
  display.set("async", false);
  display.set("sync", false);
  display.set("fullscreen", true);

  // Writes output buffers to files.
  Element imagesink("multifilesink", "imagesink");
  imagesink.set("enable-last-sample", false);
  imagesink.set("location", home_path + "/Downloads/qimsdk_samples/media/image_%d.jpeg");

  // Stream filters used in branch links.
  // They define specific stream characteristics from the supported options.
  auto vf = VideoFilter().format("NV12").resolution(1920, 1080).framerate(30);
  auto image_filter = ImageFilter().format("jpeg").resolution(3840, 2160);

  // Creates the pipeline, adds and links elements, and executes it.
  //
  // Explicit linking is applied
  Pipeline pipeline("cam-pipeline");
  pipeline.add(source)
  .add_stream_filter("vf", vf)
  .add(display)
  .add_stream_filter("if", image_filter)
  .add(imagesink)
  .link("source", "vf", "display")
  .link("source", "if", "imagesink")
  .start();

  std::this_thread::sleep_for(std::chrono::seconds(2));

  auto cam = pipeline.get<CamSrc>("source");
  cam.image_capture();

  std::this_thread::sleep_for(std::chrono::seconds(2));

  pipeline.stop();
  }
  int main() {
  if (home_path.empty()) {
  std::cerr << "Error: HOME environment variable is not set." << std::endl;
  return 1;
  }
  // Route GStreamer logs through the QIMSDK logger and enable debug output.
  qti::SetImsdkGstLogMode(qti::ImsdkGstLogMode::ImsdkLog);
  qti::SetImsdkLogLevel(qti::ImsdkLogLevel::Debug);
  try {
  create_and_execute_pipeline();
  } catch (const std::exception &ex) {
  std::cerr << "Exception: " << ex.what() << std::endl;
  return 1;
  }
  return 0;
  }
  ```

  <Steps>
    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_camera_and_capture
      ```

      The application displays a live preview and captures a still JPEG image a few seconds after starting, saving it under `~/media/image_%d.jpeg`.
    </Step>

    <Step title="Expected Output">
      A live preview is rendered, followed by a still JPEG image saved to `~/Downloads/qimsdk_samples/media/image_%d.jpeg`. The application then stops automatically.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_camera_and_capture`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_camera_and_capture)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Camera Recording to MP4

This pipeline captures frames from the built-in camera, encodes them into an H.264 stream using the hardware encoder, parses the bitstream, muxes it into an MP4 container, and writes the resulting file to disk.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_camera_encoder`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_camera_encoder)

    Pre-built application on device: `/usr/bin/qimsdk_ref_camera_encoder`
  </Info>

  ```cpp theme={null}
  #include <iostream>
  #include <cstdlib>
  #include <qti/qimsdk.h>
  using namespace qti;
  static const std::string home_path =
  std::getenv("HOME") ? std::getenv("HOME") : "";
  //  Example pipeline:
  //
  //    source -> [vf] -> encoder -> parser -> muxer -> sink
  //
  //  The pipeline reads camera frames, encodes them with H.264, muxes into MP4,
  //  and writes the output to disk.
  void create_and_execute_pipeline() {
  // Captures frames from the camera source.
  Element source("qtiqmmfsrc", "source");
  // Encodes raw video frames into H.264 stream.
  Element encoder("v4l2h264enc", "encoder");
  encoder.set("output-io-mode", "dmabuf-import");
  encoder.set("capture-io-mode", "dmabuf");
  // Parses H.264 bitstream for downstream muxing.
  Element parser("h264parse", "parser");
  // Muxes encoded stream into MP4 container.
  Element muxer("mp4mux", "muxer");
  // Writes output stream to a file.
  Element sink("filesink", "sink");
  sink.set("location", home_path + "/Downloads/qimsdk_samples/media/encoder_output.mp4");
  // Stream filters used in branch links.
  // They define specific stream characteristics from the supported options.
  auto vf = VideoFilter().format("NV12").resolution(1920, 1080).framerate(30);
  // Creates the pipeline, adds and links elements, and executes it.
  //
  // Explicit linking is applied.
  Pipeline pipeline("cam-encoder-pipeline");
  pipeline.add(source)
  .add_stream_filter("vf", vf)
  .add(encoder)
  .add(parser)
  .add(muxer)
  .add(sink)
  .eos(true)
  .execute();
  }
  int main() {
  if (home_path.empty()) {
  std::cerr << "Error: HOME environment variable is not set." << std::endl;
  return 1;
  }
  // Route GStreamer logs through the QIMSDK logger and enable debug output.
  qti::SetImsdkGstLogMode(qti::ImsdkGstLogMode::ImsdkLog);
  qti::SetImsdkLogLevel(qti::ImsdkLogLevel::Debug);
  try {
  create_and_execute_pipeline();
  } catch (const std::exception &ex) {
  std::cerr << "Exception: " << ex.what() << std::endl;
  return 1;
  }
  return 0;
  }
  ```

  <Steps>
    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_camera_encoder
      ```

      The application records camera frames encoded as H.264 into an MP4 file saved to `~/Downloads/qimsdk_samples/media/encoder_output.mp4`.
    </Step>

    <Step title="Expected Output">
      The recorded MP4 file is written to `~/Downloads/qimsdk_samples/media/encoder_output.mp4` and can be played back to verify the camera recording.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_camera_encoder`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_camera_encoder)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### MP4 File Playback

This pipeline ingests an MP4 file, demultiplexes it to extract the video track, decodes the compressed stream (e.g., H.264/H.265) into raw frames, converts (or directly negotiates) the output to NV12 — a hardware-friendly YUV 4:2:0 semi-planar format — and then renders the frames to the display using a sink that supports zero-copy or hardware buffers.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_qtdemux_decode_display`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_qtdemux_decode_display)

    Pre-built application on device: `/usr/bin/qimsdk_ref_qtdemux_decode_display`
  </Info>

  ```cpp theme={null}
  #include <iostream>
  #include <cstdlib>
  #include <qti/qimsdk.h>
  using namespace qti;
  static const std::string home_path =
  std::getenv("HOME") ? std::getenv("HOME") : "";
  //  Example pipeline:
  //
  //    src → demux → parse → decoder → [videofilter] → display
  //
  //  The pipeline reads an MP4/H.264 file, decodes it through the hardware decoder,
  //  overlays detected objects, and displays the result through Wayland.
  void create_and_execute_pipeline() {
  // Reads the input media file as raw bytes.
  Element src("filesrc", "src");
  src.set("location", home_path + "/Downloads/qimsdk_samples/media/ai_demo_sample.mp4");

  // Extracts elementary streams from the MP4 container.
  Element demux("qtdemux", "demux");

  // Prepares the H.264 bitstream for the decoder.
  Element parse("h264parse", "parse");

  // Decodes the compressed H.264 stream into raw video frames.
  //
  // The I/O mode is configured to enforce DMA buffer usage,
  // avoiding unnecessary buffer copies.
  Element decoder("v4l2h264dec", "decoder");
  decoder.set("output-io-mode", 4);
  decoder.set("capture-io-mode", 4);

  // Render video stream on display.
  //
  // async=false enforce state transition to ensure the buffers are returned on time.
  // sync=true keeps rendering synchronized to the pipeline clock.
  // fullscreen=true renders the output fullscreen on the target display.
  Element display("waylandsink", "display");
  display.set("fullscreen", true);

  // Stream filters used in branch links.
  // They define specific stream characteristics from the supported options.
  auto videofilter = VideoFilter().format("NV12");

  // Creates the pipeline, adds and links elements, and executes it.
  //
  // Linking is implicit and follows the order in which elements are added.
  Pipeline pipeline("video-pipeline");
  pipeline.add(src)
  .add(demux)
  .add(parse)
  .add(decoder)
  .add_stream_filter("videofilter", videofilter)
  .add(display);
  pipeline.start().wait().stop();
  }
  int main() {
  if (home_path.empty()) {
  std::cerr << "Error: HOME environment variable is not set." << std::endl;
  return 1;
  }
  // Route GStreamer logs through the QIMSDK logger and enable debug output.
  qti::SetImsdkGstLogMode(qti::ImsdkGstLogMode::ImsdkLog);
  qti::SetImsdkLogLevel(qti::ImsdkLogLevel::Debug);
  try {
  create_and_execute_pipeline();
  } catch (const std::exception &ex) {
  std::cerr << "Exception: " << ex.what() << std::endl;
  return 1;
  }
  return 0;
  }
  ```

  <Steps>
    <Step title="Download Required Files">
      | File | Download | Save as |
      | - | - | - |
      | Sample video | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | `ai_demo_sample.mp4` |

      You can also use your own MP4 (H.264) file, or update the pipeline's `location` property to point to it.
    </Step>

    <Step title="Copy the video file to the device">
      ```bash theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media"
      scp ai_demo_sample.mp4 <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_qtdemux_decode_display
      ```

      The application decodes the video file and renders it fullscreen on the display.
    </Step>

    <Step title="Expected Output">
      The MP4 video plays back fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_qtdemux_decode_display`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_qtdemux_decode_display)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Transport Stream (.ts) File Playback

This pipeline is similar to the Offline Video Source pipeline, but ingests an MPEG transport stream (`.ts`) file instead of an MP4 container. It demultiplexes the transport stream using `tsdemux` to extract the video track, decodes the compressed H.264 stream into raw frames, converts (or directly negotiates) the output to NV12, and renders the frames to the display.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_tsdemux_decode_display`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_tsdemux_decode_display)

    Pre-built application on device: `/usr/bin/qimsdk_ref_tsdemux_decode_display`
  </Info>

  ```cpp theme={null}
  #include <iostream>
  #include <cstdlib>
  #include <qti/qimsdk.h>
  using namespace qti;
  static const std::string home_path =
  std::getenv("HOME") ? std::getenv("HOME") : "";
  //  Example pipeline:
  //
  //    src → demux → [vf1] → parse → decoder → [vf2] → display
  //
  //  The pipeline reads an MPEG transport stream (.ts) file, demultiplexes it,
  //  decodes the H.264 stream through the hardware decoder, and displays the
  //  result through Wayland.
  void create_and_execute_pipeline() {
  // Reads a sequence of input files as stream data.
  Element src("multifilesrc", "src");
  src.set("location", home_path + "/Downloads/qimsdk_samples/media/ai_demo_sample.ts");
  src.set("stop-index", 0);
  // Extracts elementary streams from the transport stream.
  Element demux("tsdemux", "demux");
  // Prepares the H.264 bitstream for the decoder.
  Element parse("h264parse", "parse");
  // Decodes the compressed H.264 stream into raw video frames.
  //
  // The I/O mode is configured to enforce DMA buffer usage,
  // avoiding unnecessary buffer copies.
  Element decoder("v4l2h264dec", "decoder");
  decoder.set("output-io-mode", 4);
  decoder.set("capture-io-mode", 4);
  // Render video stream on display.
  //
  // async=false enforce state transition to ensure the buffers are returned on time.
  // sync=true keeps rendering synchronized to the pipeline clock.
  // fullscreen=true renders the output fullscreen on the target display.
  Element display("waylandsink", "display");
  display.set("fullscreen", true);
  auto vf1 = H264Filter().framerate(30);
  // Stream filters used in branch links.
  // They define specific stream characteristics from the supported options.
  auto vf2 = VideoFilter().format("NV12");
  // Creates the pipeline, adds and links elements, and executes it.
  //
  // Linking is implicit and follows the order in which elements are added.
  Pipeline pipeline("video-pipeline");
  pipeline.add(src)
  .add(demux)
  .add_stream_filter("vf1", vf1)
  .add(parse)
  .add(decoder)
  .add_stream_filter("vf2", vf2)
  .add(display);
  pipeline.start().wait().stop();
  }
  int main() {
  if (home_path.empty()) {
  std::cerr << "Error: HOME environment variable is not set." << std::endl;
  return 1;
  }
  // Route GStreamer logs through the QIMSDK logger and enable debug output.
  qti::SetImsdkGstLogMode(qti::ImsdkGstLogMode::ImsdkLog);
  qti::SetImsdkLogLevel(qti::ImsdkLogLevel::Debug);
  try {
  create_and_execute_pipeline();
  } catch (const std::exception &ex) {
  std::cerr << "Exception: " << ex.what() << std::endl;
  return 1;
  }
  return 0;
  }
  ```

  <Steps>
    <Step title="Download Required Files">
      | File | Download | Save as |
      | - | - | - |
      | Sample transport stream | [ai\_demo\_sample.ts](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.ts) | `ai_demo_sample.ts` |
    </Step>

    <Step title="Copy the transport stream file to the device">
      ```bash theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media"
      scp ai_demo_sample.ts <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_tsdemux_decode_display
      ```

      The application decodes the transport stream file and renders it fullscreen on the display.
    </Step>

    <Step title="Expected Output">
      The transport stream video plays back fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_tsdemux_decode_display`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_tsdemux_decode_display)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Synthetic Test Pattern (VideoFilter API)

This pipeline generates synthetic test video frames using `videotestsrc` (a bouncing ball pattern), instead of reading from a camera or file. The frames are restricted to NV12/1080p/30fps, rotated 180 degrees by [`qtivtransform`](/plugin-reference/qtivtransform), downscaled to 1280x720 by a second stream filter, and rendered to the display. This is a useful pipeline for testing display and transform stages without any camera or media file dependency.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_videotestsrc_videofilter_display`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_videotestsrc_videofilter_display)

    Pre-built application on device: `/usr/bin/qimsdk_ref_videotestsrc_videofilter_display`
  </Info>

  ```cpp theme={null}
  #include <iostream>
  #include <qti/qimsdk.h>
  using namespace qti;
  //  Example pipeline:
  //
  //    src → [vf1] → transform → [vf2] → sink
  //
  //  The pipeline generates synthetic test video frames, rotates them, restricts
  //  the stream to NV12/1080p/30fps, downscales to 720p, and displays the result
  //  through Wayland.
  void create_and_execute_pipeline() {
  // Generates synthetic test video frames.
  Element src("videotestsrc", "src");
  src.set("pattern", "ball");
  // Applies geometric transforms to video frames.
  Element transform("qtivtransform", "transform");
  transform.set("rotate", "180");
  // Render video stream on display.
  Element sink("waylandsink", "sink");
  sink.set("fullscreen", true);
  // Stream filters used in branch links.
  // They define specific stream characteristics from the supported options.
  auto vf1 = qti::VideoFilter().format("NV12").resolution(1920, 1080).framerate(30);
  auto vf2 = qti::VideoFilter().resolution(1280, 720);
  // Creates the pipeline, adds and links elements, and executes it.
  //
  // Linking is implicit and follows the order in which elements are added.
  Pipeline pipeline("video-pipeline");
  pipeline.add(src)
  .add_stream_filter("vf1", vf1)
  .add(transform)
  .add_stream_filter("vf2", vf2)
  .add(sink)
  .execute();
  }
  int main() {
  // Route GStreamer logs through the QIMSDK logger and enable debug output.
  qti::SetImsdkGstLogMode(qti::ImsdkGstLogMode::ImsdkLog);
  qti::SetImsdkLogLevel(qti::ImsdkLogLevel::Debug);
  try {
  create_and_execute_pipeline();
  } catch (const std::exception &ex) {
  std::cerr << "Exception: " << ex.what() << std::endl;
  return 1;
  }
  return 0;
  }
  ```

  <Steps>
    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_videotestsrc_videofilter_display
      ```

      The application generates a synthetic bouncing-ball test pattern, rotates and rescales it, and renders it fullscreen on the display.
    </Step>

    <Step title="Expected Output">
      A rotated, rescaled bouncing-ball test pattern is rendered fullscreen on the display.

      <video />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_videotestsrc_videofilter_display`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_videotestsrc_videofilter_display)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Synthetic Test Pattern (Raw Caps StreamFilter)

This pipeline is functionally identical to the previous example, but it defines its stream constraints using raw GStreamer caps strings via `StreamFilter` (e.g. `"video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1"`), instead of the structured `VideoFilter` builder API. This is useful when you need to express filter capabilities that aren't covered by the structured builder, or when porting caps strings directly from existing GStreamer pipelines.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_videotestsrc_streamfilter_display`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_videotestsrc_streamfilter_display)

    Pre-built application on device: `/usr/bin/qimsdk_ref_videotestsrc_streamfilter_display`
  </Info>

  ```cpp theme={null}
  #include <iostream>
  #include <qti/qimsdk.h>
  using namespace qti;
  //  Example pipeline:
  //
  //    src → [vf1] → transform → [vf2] → sink
  //
  //  The pipeline generates synthetic test video frames, rotates them, restricts
  //  the stream using raw caps strings, and displays the result through Wayland.
  void create_and_execute_pipeline() {
  // Generates synthetic test video frames.
  Element src("videotestsrc", "src");
  src.set("pattern", "ball");
  // Applies geometric transforms to video frames.
  Element transform("qtivtransform", "transform");
  transform.set("rotate", "180");
  // Render video stream on display.
  Element sink("waylandsink", "sink");
  sink.set("fullscreen", true);
  // Stream filters expressed as raw GStreamer caps strings, used in branch links.
  qti::StreamFilter vf1(
  "video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1");
  qti::StreamFilter vf2("video/x-raw,width=1280,height=720");
  // Creates the pipeline, adds and links elements, and executes it.
  //
  // Linking is implicit and follows the order in which elements are added.
  Pipeline pipeline("video-pipeline-streamfilter-string");
  pipeline.add(src)
  .add_stream_filter("vf1", vf1)
  .add(transform)
  .add_stream_filter("vf2", vf2)
  .add(sink)
  .execute();
  }
  int main() {
  // Route GStreamer logs through the QIMSDK logger and enable debug output.
  qti::SetImsdkGstLogMode(qti::ImsdkGstLogMode::ImsdkLog);
  qti::SetImsdkLogLevel(qti::ImsdkLogLevel::Debug);
  try {
  create_and_execute_pipeline();
  } catch (const std::exception &ex) {
  std::cerr << "Exception: " << ex.what() << std::endl;
  return 1;
  }
  return 0;
  }
  ```

  <Steps>
    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_videotestsrc_streamfilter_display
      ```

      The application generates a synthetic bouncing-ball test pattern, rotates and rescales it, and renders it fullscreen on the display.
    </Step>

    <Step title="Expected Output">
      A rotated, rescaled bouncing-ball test pattern is rendered fullscreen on the display.

      <video />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_videotestsrc_streamfilter_display`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_videotestsrc_streamfilter_display)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### RTSP Camera Stream Playback

This pipeline connects to a network RTSP camera stream instead of a local (USB/built-in) camera or file. `rtspsrc` establishes the RTSP session and negotiates the H.264 RTP stream, a caps filter restricts it to H.264 video, `rtph264depay` extracts the H.264 payload from the RTP packets, `h264parse` prepares the bitstream for the hardware decoder, and the decoded frames are converted to NV12 and rendered to the display.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    See the full YAML configuration here: [qimsdk\_ref\_rtsp\_camera.yaml](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_rtsp_camera.yaml)

    Check application source code on GitHub: [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)

    Pre-built application on device: `/usr/bin/qimsdk_ref_yml`

    YAML configuration saved on device: `/etc/qimsdk/qimsdk_ref_rtsp_camera.yaml`
  </Info>

  ```yaml theme={null}
  pipeline:
    elements:
      - type: rtspsrc
        name: source
        location: rtsp://username:password@192.168.1.188:554/Streaming/Channels/101
        latency: 0

      - type: capsfilter
        name: rtp_caps
        caps: application/x-rtp,media=video,encoding-name=H264

      - type: rtph264depay
        name: depay

      - type: h264parse
        name: parse

      - type: v4l2h264dec
        name: decoder
        capture-io-mode: 4
        output-io-mode: 4

      - type: filter
        name: videostream
        video:
          format: NV12

      - type: waylandsink
        name: display
        sync: false
        fullscreen: true
  ```

  <Steps>
    <Step title="Update the RTSP camera credentials">
      On the device, edit `/etc/qimsdk/qimsdk_ref_rtsp_camera.yaml` and update the `location` property with your RTSP camera's username, password, IP address, and stream path.
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      qimsdk_ref_yml /etc/qimsdk/qimsdk_ref_rtsp_camera.yaml
      ```

      The application connects to the RTSP camera, decodes the H.264 stream, and renders it fullscreen on the display.
    </Step>

    <Step title="Expected Output">
      The live RTSP camera stream plays back fullscreen on the display.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

## AI Pipeline

### Video-to-Tensor Dump (Offline Preprocessing)

This pipeline ingests an MP4 file, demultiplexes it to extract the video track, decodes the compressed stream (e.g., H.264/H.265) into raw frames, converts (or directly negotiates) the output to NV12 — a hardware-friendly YUV 4:2:0 semi-planar format — and then goes through [`qtimlvconverter`](/plugin-reference/qtimlvconverter) (ML preprocessing). The `TensorFilter` stream filter describes the tensor format and shape as UINT8 with dimensions \[1, 520, 520, 3] (batch, height, width, channels). Finally, `multifilesink` writes the preprocessed tensors out as sequential `.rgb` files to `~/Downloads/qimsdk_samples/media/tensor_520_520_%d.rgb`, rolling after every 10 files.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_tensor_dump`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_tensor_dump)

    Pre-built application on device: `/usr/bin/qimsdk_ref_tensor_dump`
  </Info>

  <Steps>
    <Step title="Download Required Files">
      | File | Download | Save as |
      | - | - | - |
      | Sample video | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | `ai_demo_sample.mp4` |

      You can also use your own MP4 (H.264) file, or update the pipeline's `location` property to point to it.
    </Step>

    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media"
        scp ai_demo_sample.mp4 <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_tensor_dump
      ```

      The application writes preprocessed tensors to `~/Downloads/qimsdk_samples/media/tensor_520_520_%d.rgb`.
    </Step>

    <Step title="Expected Output">
      Sequential `.rgb` tensor files accumulate under `~/Downloads/qimsdk_samples/media/`, rolling after every 10 files.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_tensor_dump`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_tensor_dump)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Live Camera YOLOv8 Detection (Manual Chain)

This pipeline captures live video from the on-device camera, then drives a full detection-and-visualization loop in real time: the raw NV12 stream is normalized and preprocessed into one or more tensors, fed to a TFLite inference stage (running on the external QNN delegate for efficient hardware execution), and the resulting tensors are postprocessed into semantic detections (classes, boxes, confidences). Those results are converted into QIM SDK-friendly metadata and attached back to the original video timeline, so the visual stream and its ML annotations stay synchronized. A parallel branch from the early tee ensures the original frames are available to a metadata muxer, which merges the detections with the video. Finally, the overlay stage renders the boxes/labels directly on the live frames, and the composed output is presented fullscreen to the display. In short, the graph turns a live camera feed into actionable on-screen insights by (1) creating a tensor view of the stream, (2) running detection, (3) translating inference outputs into QIM SDK metadata, (4) fusing metadata with the original frames, and (5) drawing the results on top of the video for low-latency, on-screen visualization.

<img alt="Introduction" />

<Accordion title="Try me">
  <Tabs>
    <Tab title="Explicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_camera_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_camera_yolov8)

        Pre-built application on device: `/usr/bin/qimsdk_ref_camera_yolov8`
      </Info>

      #### Download Required Files

      | File | Save as |
      | - | - |
      | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |

      <Note>
        If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

        Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
      </Note>

      <Steps>
        <Step title="Copy Files to Device">
          <CodeGroup>
            ```bash SCP (SSH) theme={null}
            # Replace <user> and <device-ip> with your device credentials.
            # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

            ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
            scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            ```
          </CodeGroup>
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          /usr/bin/qimsdk_ref_camera_yolov8
          ```

          The application overlays detected bounding boxes and labels on the live camera feed and renders it fullscreen on the display.
        </Step>

        <Step title="Expected Output">
          The live camera feed is rendered fullscreen with detected objects outlined by bounding boxes and labeled with class names.

          <img alt="Expected Output" />

          To stop the application, press **CTRL + C**.
        </Step>

        <Step title="Customize application">
          * **Source code:** [`qimsdk_ref_camera_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_camera_yolov8)
          * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
        </Step>
      </Steps>
    </Tab>

    <Tab title="YAML config style">
      <Info>
        See the full YAML configuration here: [qimsdk\_ref\_camera\_yolov8.yaml](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_camera_yolov8.yaml)

        Check application source code on GitHub: [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)

        Pre-built application on device: `/usr/bin/qimsdk_ref_yml`

        YAML configuration saved on device: `/etc/qimsdk/qimsdk_ref_camera_yolov8.yaml`
      </Info>

      ```yaml theme={null}
      pipeline:
        elements:
          - type: qtiqmmfsrc
            name: source
            camera: 0

          - type: filter
            name: videostream
            video:
              format: NV12
              width: 1920
              height: 1080
              framerate: 30

          - type: tee
            name: split

          - type: queue
            name: q1

          - type: qtimlvconverter
            name: preprocessing

          - type: queue
            name: q3

          - type: qtimltflite
            name: inferencing
            delegate: external
            external-delegate-path: libQnnTFLiteDelegate.so
            external-delegate-options: "QNNExternalDelegate,backend_type=htp;"
            model: ~/Downloads/qimsdk_samples/models/yolov8_det_quantized.tflite

          - type: queue
            name: q4

          - type: qtimlpostprocess
            name: postprocessing
            results: 5
            module: yolov8
            labels: ~/Downloads/qimsdk_samples/labels/yolov8.json
            settings: '{"confidence": 70.0}'

          - type: filter
            name: mlf
            text: {}

          - type: qtimetamux
            name: mlmuxer

          - type: queue
            name: q5

          - type: qtivoverlay
            name: overlay

          - type: waylandsink
            name: display
            sync: false
            fullscreen: true

        links:
          - [split, mlmuxer]
          - [source, videostream, split, q1, preprocessing, q3, inferencing, q4, postprocessing, mlf, mlmuxer, q5, overlay, display]
      ```

      #### Download Required Files

      | File | Save as |
      | - | - |
      | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |

      <Note>
        If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

        Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
      </Note>

      <Steps>
        <Step title="Copy Files to Device">
          <CodeGroup>
            ```bash SCP (SSH) theme={null}
            # Replace <user> and <device-ip> with your device credentials.
            # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

            ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
            scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            ```
          </CodeGroup>
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          qimsdk_ref_yml /etc/qimsdk/qimsdk_ref_camera_yolov8.yaml
          ```

          The application overlays detected bounding boxes and labels on the live camera feed and renders it fullscreen on the display.
        </Step>

        <Step title="Expected Output">
          The live camera feed is rendered fullscreen with detected objects outlined by bounding boxes and labeled with class names.

          <img alt="Expected Output" />

          To stop the application, press **CTRL + C**.
        </Step>

        <Step title="Customize application">
          * **Source code:** [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)
          * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
        </Step>
      </Steps>
    </Tab>
  </Tabs>
</Accordion>

### Live Camera YOLOv8 Detection (ML Bin)

This solution is identical to the previous one. The only difference is that here we use `qtimlvideotflitebin` instead of manually linking the preprocessing, inference, postprocessing, and ML muxer components, which simplifies the pipeline.

<img alt="Introduction" />

<Accordion title="Try me">
  <Tabs>
    <Tab title="Explicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_camera_yolov8_mlbin`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_camera_yolov8_mlbin)

        Pre-built application on device: `/usr/bin/qimsdk_ref_camera_yolov8_mlbin`
      </Info>

      #### Download Required Files

      | File | Save as |
      | - | - |
      | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |

      <Note>
        If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

        Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
      </Note>

      <Steps>
        <Step title="Copy Files to Device">
          <CodeGroup>
            ```bash SCP (SSH) theme={null}
            # Replace <user> and <device-ip> with your device credentials.
            # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

            ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
            scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            ```
          </CodeGroup>
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          /usr/bin/qimsdk_ref_camera_yolov8_mlbin
          ```

          The application overlays detected bounding boxes and labels on the live camera feed and renders it fullscreen on the display.
        </Step>

        <Step title="Expected Output">
          The live camera feed is rendered fullscreen with detected objects outlined by bounding boxes and labeled with class names.

          <img alt="Expected Output" />

          To stop the application, press **CTRL + C**.
        </Step>

        <Step title="Customize application">
          * **Source code:** [`qimsdk_ref_camera_yolov8_mlbin`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_camera_yolov8_mlbin)
          * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
        </Step>
      </Steps>
    </Tab>

    <Tab title="YAML config style">
      <Info>
        See the full YAML configuration here: [qimsdk\_ref\_mlbin\_yolov8.yaml](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_mlbin_yolov8.yaml)

        Check application source code on GitHub: [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)

        Pre-built application on device: `/usr/bin/qimsdk_ref_yml`

        YAML configuration saved on device: `/etc/qimsdk/qimsdk_ref_mlbin_yolov8.yaml`
      </Info>

      ```yaml theme={null}
      pipeline:
        elements:
          - type: filesrc
            name: src
            location: ~/Downloads/qimsdk_samples/media/ai_demo_sample.mp4

          - type: qtdemux
            name: demux

          - type: h264parse
            name: parse

          - type: v4l2h264dec
            name: decoder
            output-io-mode: 4
            capture-io-mode: 4

          - type: filter
            name: videofilter
            video:
              format: NV12

          - type: qtimlvideotflitebin
            name: mlbin
            inference-delegate: "external"
            inference-external-delegate-path: "libQnnTFLiteDelegate.so"
            inference-external-delegate-options: "QNNExternalDelegate,backend_type=htp;"
            inference-model: "~/Downloads/qimsdk_samples/models/yolov8_det_quantized.tflite"
            postprocess-module: "yolov8"
            postprocess-labels: "~/Downloads/qimsdk_samples/labels/yolov8.json"

          - type: qtivoverlay
            name: overlay

          - type: waylandsink
            name: display
            fullscreen: true
      ```

      #### Download Required Files

      | File | Save as |
      | - | - |
      | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |

      <Note>
        If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

        Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
      </Note>

      <Steps>
        <Step title="Copy Files to Device">
          <CodeGroup>
            ```bash SCP (SSH) theme={null}
            # Replace <user> and <device-ip> with your device credentials.
            # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

            ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
            scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            ```
          </CodeGroup>
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          qimsdk_ref_yml /etc/qimsdk/qimsdk_ref_mlbin_yolov8.yaml
          ```

          The application overlays detected bounding boxes and labels on the live camera feed and renders it fullscreen on the display.
        </Step>

        <Step title="Expected Output">
          The live camera feed is rendered fullscreen with detected objects outlined by bounding boxes and labeled with class names.

          <img alt="Expected Output" />

          To stop the application, press **CTRL + C**.
        </Step>

        <Step title="Customize application">
          * **Source code:** [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)
          * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
        </Step>
      </Steps>
    </Tab>
  </Tabs>
</Accordion>

### Offline Video YOLOv8 Detection (Manual Chain, Metamux Overlay)

This pipeline is the offline-video counterpart to Live Camera YOLOv8 Detection (Manual Chain): instead of a live camera feed, an MP4 file is demuxed and decoded, then split by a `tee` into a display branch and an ML branch. The ML branch runs preprocessing, inference, and postprocessing manually, and the resulting detections are attached back onto the original frame via [`qtimetamux`](/plugin-reference/qtimetamux) before being overlaid and rendered. This Metamux-based approach is more flexible than the video composer approach (it allows chaining additional ML models or streaming metadata separately), whereas the composer overlay approach used in the next example offers better performance.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_yolov8_metamux_overlay`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yolov8_metamux_overlay)

    Pre-built application on device: `/usr/bin/qimsdk_ref_yolov8_metamux_overlay`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
  | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4          <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_yolov8_metamux_overlay
      ```

      The application decodes the video file, runs YOLOv8 detection, attaches the detections to the original frame via [`qtimetamux`](/plugin-reference/qtimetamux), and overlays the results, rendering them fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with YOLOv8 bounding boxes and class labels overlaid on each detected object, rendered fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_yolov8_metamux_overlay`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yolov8_metamux_overlay)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Offline Video YOLOv8 Detection (ML Bin, Composer Overlay)

The AI output can be overlaid directly on top of the frame instead of attaching AI metadata to the main frame. This approach provides better performance, whereas the Metamux solution is more flexible and can be used for chaining AI models or streaming AI metadata separately.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_yolov8_composer_overlay`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yolov8_composer_overlay)

    Pre-built application on device: `/usr/bin/qimsdk_ref_yolov8_composer_overlay`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
  | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4          <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_yolov8_composer_overlay
      ```

      The application composes the AI detection overlay directly on top of the decoded video frames and displays it fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with YOLOv8 bounding boxes and class labels composited directly onto each frame, rendered fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_yolov8_composer_overlay`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yolov8_composer_overlay)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Gesture Recognition

This pipeline demonstrates a daisy-chained, multi-model gesture recognition graph built entirely from standard SDK elements ([`qtimlvconverter`](/plugin-reference/qtimlvconverter), [`qtimltflite`](/plugin-reference/qtimltflite), [`qtimlpostprocess`](/plugin-reference/qtimlpostprocess), [`qtimetamux`](/plugin-reference/qtimetamux), `qtimetatransform`) rather than a custom postprocessing callback. Palm detection locates the hand region in the live camera frame; that region is transformed and fed into hand landmark detection; the resulting landmarks are passed through a gesture embedder followed by a canned gesture classifier to decode the recognized gesture. Two parallel [`qtimetamux`](/plugin-reference/qtimetamux) merge points combine metadata from each stage back onto the original frame before the final overlay is rendered to the display.

<img alt="Introduction" />

<Accordion title="Try me">
  <Tabs>
    <Tab title="Explicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_gesture_recognition`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_gesture_recognition)

        Pre-built application on device: `/usr/bin/qimsdk_ref_gesture_recognition`
      </Info>
    </Tab>

    <Tab title="YAML config style">
      <Info>
        See the full YAML configuration here: [qimsdk\_ref\_gesture\_recognition.yaml](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_gesture_recognition.yaml)

        Check application source code on GitHub: [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)

        Pre-built application on device: `/usr/bin/qimsdk_ref_yml`

        YAML configuration saved on device: `/etc/qimsdk/qimsdk_ref_gesture_recognition.yaml`
      </Info>
    </Tab>
  </Tabs>

  #### Download Required Files

  Download the gesture recognizer task bundle from Google MediaPipe to obtain the palm detection, hand landmark, gesture embedder, and canned gesture classifier models:

  ```bash theme={null}
  # Download the gesture recognizer task bundle
  wget https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/latest/gesture_recognizer.task

  # Extract the top-level task
  unzip gesture_recognizer.task

  # Extract hand landmarker models
  unzip hand_landmarker.task
  # → hand_detector.tflite, hand_landmarks_detector.tflite

  # Extract gesture recognizer models
  unzip hand_gesture_recognizer.task
  # → gesture_embedder.tflite, canned_gesture_classifier.tflite
  ```

  | File | Save as |
  | - | - |
  | hand\_detector.tflite (see steps above) | palm\_detection\_full.tflite |
  | hand\_landmarks\_detector.tflite (see steps above) | hand\_landmark\_full.tflite |
  | gesture\_embedder.tflite (see steps above) | gesture\_embedder.tflite |
  | canned\_gesture\_classifier.tflite (see steps above) | canned\_gesture\_classifier.tflite |
  | <a href="../labels/palmd_labels.json">palmd\_labels.json</a> | palmd\_labels.json |
  | <a href="../labels/palmd_settings.json">palmd\_settings.json</a> | palmd\_settings.json |
  | <a href="../labels/hlandmarks.json">hlandmarks.json</a> | hlandmarks.json |
  | <a href="../labels/hlandmark_settings.json">hlandmark\_settings.json</a> | hlandmark\_settings.json |
  | <a href="../labels/gesture_rec.json">gesture\_rec.json</a> | gesture\_rec.json |

  <Note>
    If a downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      This pipeline's source is the on-device camera, so no video file needs to be copied.

      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
        scp palm_detection_full.tflite       <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp hand_landmark_full.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp gesture_embedder.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp canned_gesture_classifier.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp palmd_labels.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp palmd_settings.json              <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp hlandmarks.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp hlandmark_settings.json          <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp gesture_rec.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_gesture_recognition
      ```

      The application captures live video from the on-device camera, runs the daisy-chained palm detection → hand landmark → gesture embedder → canned gesture classifier models, and overlays the recognized gesture on the display fullscreen.
    </Step>

    <Step title="Expected Output">
      The live camera feed is rendered fullscreen with hand landmarks and the recognized gesture label overlaid.

      <video />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_gesture_recognition`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_gesture_recognition), [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### PPE & Person/Foot Detection (ML Bin)

This pipeline demonstrates a two-model ML bin pipeline for personal protective equipment (PPE) and person/foot detection: `foot_track_net` detects persons and their feet in the frame, while `gear_guard_net` detects PPE gear. Both models run through `qtimlvideotflitebin`, which internally handles preprocessing, inference, and postprocessing, simplifying the pipeline compared to manually chaining each stage.

<img alt="Introduction" />

<Accordion title="Try me">
  <Tabs>
    <Tab title="Explicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_mlbin_ppe`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_mlbin_ppe)

        Pre-built application on device: `/usr/bin/qimsdk_ref_mlbin_ppe`
      </Info>
    </Tab>

    <Tab title="YAML config style">
      <Info>
        See the full YAML configuration here: [qimsdk\_ref\_mlbin\_ppe.yaml](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_mlbin_ppe.yaml)

        Check application source code on GitHub: [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)

        Pre-built application on device: `/usr/bin/qimsdk_ref_yml`

        YAML configuration saved on device: `/etc/qimsdk/qimsdk_ref_mlbin_ppe.yaml`
      </Info>
    </Tab>
  </Tabs>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/models/gear_guard_net?searchTerm=gear) | gear\_guard\_net-ppe-detection-w8a8.tflite |
  | [Model](https://aihub.qualcomm.com/models/foot_track_net?searchTerm=foot) | foot\_track\_net-person-foot-detection-w8a8.tflite |
  | <a href="../labels/gear_guard_net.json">gear\_guard\_net.json</a> | gear\_guard\_net.json |
  | <a href="../labels/foot_track_net.json">foot\_track\_net.json</a> | foot\_track\_net.json |
  | <a href="../labels/foot_track_net_settings.json">foot\_track\_net\_settings.json</a> | foot\_track\_net\_settings.json |
  | [ppe\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/ppe_sample.mp4) | ppe\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp gear_guard_net-ppe-detection-w8a8.tflite      <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp foot_track_net-person-foot-detection-w8a8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp gear_guard_net.json                           <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp foot_track_net.json                           <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp foot_track_net_settings.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ppe_sample.mp4                                <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_mlbin_ppe
      ```

      The application runs inference on the video file, detecting persons/feet with `foot_track_net` and PPE gear with `gear_guard_net`, overlaying the resulting detections on the video and rendering it fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back fullscreen with detected persons, feet, and PPE gear outlined by bounding boxes and labeled with class names.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_mlbin_ppe`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_mlbin_ppe), [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### YOLOv8 Detection (ML Bin)

This pipeline reads an MP4/H.264 file, decodes it through the hardware decoder, and runs YOLOv8 object detection using `qtimlvideotflitebin`, which internally handles preprocessing, inference, and postprocessing. The detected objects are then overlaid on the video and displayed.

<img alt="Introduction" />

<Accordion title="Try me">
  <Tabs>
    <Tab title="Explicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_mlbin_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_mlbin_yolov8)

        Pre-built application on device: `/usr/bin/qimsdk_ref_mlbin_yolov8`
      </Info>
    </Tab>

    <Tab title="YAML config style">
      <Info>
        See the full YAML configuration here: [qimsdk\_ref\_mlbin\_yolov8.yaml](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_mlbin_yolov8.yaml)

        Check application source code on GitHub: [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)

        Pre-built application on device: `/usr/bin/qimsdk_ref_yml`

        YAML configuration saved on device: `/etc/qimsdk/qimsdk_ref_mlbin_yolov8.yaml`
      </Info>
    </Tab>
  </Tabs>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
  | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4          <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_mlbin_yolov8
      ```

      The application decodes the video file, runs YOLOv8 detection through `qtimlvideotflitebin`, and overlays the detected objects on the video, rendering it fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with YOLOv8 bounding boxes and class labels overlaid on each detected object, rendered fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_mlbin_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_mlbin_yolov8), [qimsdk\_ref\_yml](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_yml)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

## AppSink and AppSrc

<img alt="Introduction" />

AppSink and AppSrc are special elements because they expose buffers directly to and from the application. Because of this, we create and configure these elements separately, using their dedicated wrapper classes. Once configured, we insert them into the pipeline as fully constructed element objects.

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_appsrc_and_appsink`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_appsrc_and_appsink)

    Pre-built application on device: `/usr/bin/qimsdk_ref_appsrc_and_appsink`
  </Info>

  <Steps>
    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_appsrc_and_appsink
      ```

      The application generates synthetic test frames, pushes them through an `AppSink`/`AppSrc` bridge, and renders the resulting stream fullscreen on the display.
    </Step>

    <Step title="Expected Output">
      The synthetic test frames generated in application code are rendered fullscreen on the display after passing through the `AppSink`/`AppSrc` bridge.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_appsrc_and_appsink`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_appsrc_and_appsink)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

## Custom Pre and Post Processing

<img alt="Introduction" />

### Custom Preprocessing for YOLOv8 (Detection)

QIM SDK plugin support different backend: GLES, OpenCV, etc. This level of interface could be exposed to application. This API takes all input and output at once in case of batching or daisy for example. It can be simplified by calling application implementation for every input. Advantage of this approach is that all complexity of batching, depth, daisy chain support remains hidden in the plugin and application will do only pre-processing on a single frame/tensor at once.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_preprocess_detection_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_preprocess_detection_yolov8)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_preprocess_detection_yolov8`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
  | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4          <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_external_preprocess_detection_yolov8
      ```
    </Step>

    <Step title="Expected Output">
      The video plays back with YOLOv8 bounding boxes and class labels overlaid on each detected object, rendered fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_preprocess_detection_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_preprocess_detection_yolov8)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Custom Preprocessing for YOLOv8 (ML Bin variant)

This example demonstrates external preprocessing support inside `qtimlvideotflitebin`. Setting `preprocess-engine` to `none` disables the bin's internal preprocessing path and hands frame-to-tensor conversion off to an application-defined callback registered via `MLVideoTFLiteBin::set_preprocess_handler(...)`. The callback receives the decoded NV12 blit(s) and writes directly into the bin's tensor buffer, converting each pixel to the quantized int8 NHWC layout expected by the YOLOv8 model, with letterbox padding for any area the blit doesn't cover. Inference and postprocessing then run inside the same bin, and the detected objects are overlaid on the video and displayed.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_preprocess_detection_mlbin_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_preprocess_detection_mlbin_yolov8)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_preprocess_detection_mlbin_yolov8`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
  | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4          <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_external_preprocess_detection_mlbin_yolov8
      ```

      The application decodes the video file, runs the custom, application-defined preprocessing callback inside `mlbin` to convert NV12 frames into the quantized int8 tensor expected by YOLOv8, then runs inference and postprocessing, overlaying the detected objects and rendering the result fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with YOLOv8 bounding boxes and class labels overlaid on each detected object, rendered fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_preprocess_detection_mlbin_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_preprocess_detection_mlbin_yolov8)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Custom Preprocessing with Lightweight Face Detection

This example demonstrates external preprocessing with [`qtimlvconverter`](/plugin-reference/qtimlvconverter) (rather than `qtimlvideotflitebin`), so the pipeline links preprocessing, inference, and postprocessing manually through explicit queues, and merges the detections back onto the original frame via [`qtimetamux`](/plugin-reference/qtimetamux) before overlaying. Setting `engine` to `none` on the [`MLVConverter`](/plugin-reference/qtimlvconverter) element disables its internal preprocessing path and hands frame-to-tensor conversion off to an application-defined callback registered via [`MLVConverter::set_handler(...)`](/plugin-reference/qtimlvconverter). Unlike the YOLOv8 example, the `face_det_lite_w8a8.tflite` model consumes raw uint8 grayscale, so the callback resizes only the NV12 luma (Y) plane into a single-channel `[1, H, W, 1]` tensor, skipping color conversion and normalization entirely and leaving the chroma plane unread. A `tee` right after decoding splits the stream into a display branch and an ML branch, and a queue on each branch (including the display one) keeps the two branches from stalling each other on a shared thread.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_preprocess_lightweight_face_detect`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_preprocess_lightweight_face_detect)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_preprocess_lightweight_face_detect`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/face_det_lite) | face\_det\_lite\_w8a8.tflite |
  | <a href="../labels/qfd-labels.json">qfd-labels.json</a> | qfd-labels.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | face\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp face_det_lite_w8a8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp qfd-labels.json            <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp face_sample.mp4            <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_external_preprocess_lightweight_face_detect
      ```

      The application decodes the video file, runs the custom, application-defined preprocessing callback to convert the NV12 luma plane into the grayscale tensor expected by the lightweight face detector, then runs inference and postprocessing, merging the detected faces onto the original frame via [`qtimetamux`](/plugin-reference/qtimetamux), overlaying them, and rendering the result fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back fullscreen with detected faces outlined by bounding boxes.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_preprocess_lightweight_face_detect`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_preprocess_lightweight_face_detect)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Custom Preprocessing with Palm Detection

This example is the live-camera counterpart to the lightweight face detection pipeline: [`qtiqmmfsrc`](/plugin-reference/qticamsrc) captures frames directly instead of decoding a file, but the branching structure is otherwise identical — a `tee` right after the stream filter splits the pipeline into a display branch and an ML branch, each guarded by its own queue, and [`qtimetamux`](/plugin-reference/qtimetamux) merges the ML branch's detections back onto the original frame before overlay. Setting `engine` to `none` on the [`MLVConverter`](/plugin-reference/qtimlvconverter) element disables its internal preprocessing path and hands frame-to-tensor conversion off to an application-defined callback registered via [`MLVConverter::set_handler(...)`](/plugin-reference/qtimlvconverter). Unlike the grayscale face-detection example, `palm_detection_full.tflite` expects a `[1, H, W, 3]` float32 RGB tensor normalized to `[0, 1]`, so the callback converts the full NV12 image (both luma and chroma planes) to RGB and scales each channel into that range, with letterbox padding for any area the blit doesn't cover.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_preprocess_palm_detect`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_preprocess_palm_detect)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_preprocess_palm_detect`
  </Info>

  #### Download Required Files

  Download the gesture recognizer task bundle from Google MediaPipe to obtain the palm detection model:

  ```bash theme={null}
  # Download the gesture recognizer task bundle
  wget https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/latest/gesture_recognizer.task

  # Extract the top-level task
  unzip gesture_recognizer.task

  # Extract hand landmarker models
  unzip hand_landmarker.task
  # → hand_detector.tflite, hand_landmarks_detector.tflite
  ```

  | File | Save as |
  | - | - |
  | hand\_detector.tflite (see steps above) | palm\_detection\_full.tflite |
  | <a href="../labels/palmd_labels.json">palmd\_labels.json</a> | palmd\_labels.json |
  | <a href="../labels/palmd_settings.json">palmd\_settings.json</a> | palmd\_settings.json |

  <Note>
    If a downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      This pipeline's source is the on-device camera, so no video file needs to be copied.

      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
        scp palm_detection_full.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp palmd_labels.json          <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp palmd_settings.json        <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_external_preprocess_palm_detect
      ```

      The application captures live video from the on-device camera, runs the custom, application-defined preprocessing callback to convert NV12 frames into the normalized float32 RGB tensor expected by the palm detector, then runs inference and postprocessing, merging the detected palms onto the original frame via [`qtimetamux`](/plugin-reference/qtimetamux), overlaying them, and rendering the result fullscreen.
    </Step>

    <Step title="Expected Output">
      The live camera feed is rendered fullscreen with detected palms outlined by bounding boxes.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_preprocess_palm_detect`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_preprocess_palm_detect)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Custom Postprocessing with YOLOv5 (Detection)

This example demonstrates the object detection postprocessing callback (`ObjectDetectionPostprocessCallback`) using a YOLOv5 model. Preprocessing and inference are handled by the SDK, and the raw output tensors are decoded into bounding boxes, class names, and confidence scores by an application-defined callback, which are then overlaid on the video and displayed.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_detection_yolov5`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_detection_yolov5)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_detection_yolov5`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/models/yolov5?searchTerm=yolov5) | yolov5m-320x320-int8.tflite |
  | <a href="../labels/yolov5m.json">yolov5m.json</a> | yolov5m.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov5m-320x320-int8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov5m.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4           <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_external_postprocess_detection_yolov5
      ```

      The application runs inference on the video file and uses the custom, application-defined postprocessing callback to decode detections, which are then overlaid and displayed fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with YOLOv5 bounding boxes and class labels decoded by the custom postprocessing callback, overlaid and rendered fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_postprocess_detection_yolov5`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_detection_yolov5)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Custom Post processing with ML Bin

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_mlbin_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_mlbin_yolov8)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_mlbin_yolov8`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
  | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4          <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_external_postprocess_mlbin_yolov8
      ```

      The application runs preprocessing, inference, and the custom, application-defined postprocessing callback all inside `mlbin`, overlaying the resulting detections on the video and rendering it fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with YOLOv8 bounding boxes and class labels decoded by the custom postprocessing callback, overlaid and rendered fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_postprocess_mlbin_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_mlbin_yolov8)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Custom Postprocessing with ResNet101 (Classification)

This example demonstrates the classification postprocessing callback (`ClassificationPostprocessCallback`). Preprocessing and inference are handled by the SDK, and the raw output tensors from a ResNet101 model are decoded into labeled classification results (name and confidence) by an application-defined callback.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_classification_resnet101`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_classification_resnet101)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_classification_resnet101`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/resnet101) | Resnet101\_Quantized.tflite |
  | <a href="../labels/resnet101.json">resnet101.json</a> | resnet101.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | classification\_sample.mp4 |

  <Note>
    No ideal sample video ships with the SDK for this use case. Supply your own content for best results.
  </Note>

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp Resnet101_Quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp resnet101.json             <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp classification_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_external_postprocess_classification_resnet101
      ```

      The application runs inference on the ResNet101 model and uses the custom, application-defined postprocessing callback to decode classification results, which are then displayed.
    </Step>

    <Step title="Expected Output">
      The video plays back with the predicted classification label and confidence score displayed on screen for each frame.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_postprocess_classification_resnet101`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_classification_resnet101)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Custom Postprocessing with MiDaS v2 (Depth Estimation)

This example demonstrates the depth estimation postprocessing callback (`DepthEstimationPostprocessCallback`). Preprocessing and inference are handled by the SDK, and the raw output tensors from a MiDaS v2 model are decoded by an application-defined callback into a per-pixel depth map (values and pseudo-color) for downstream visualization.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_depth_estimation_midasv2`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_depth_estimation_midasv2)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_depth_estimation_midasv2`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/models/midas) | midas-tflite-w8a8.tflite |
  | <a href="../labels/midas-v2-labels.json">midas-v2-labels.json</a> | midas-v2-labels.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp midas-tflite-w8a8.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp midas-v2-labels.json      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4        <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_external_postprocess_depth_estimation_midasv2
      ```

      The application runs inference on the MiDaS v2 model and uses the custom, application-defined postprocessing callback to decode the depth map, which is then rendered.
    </Step>

    <Step title="Expected Output">
      The video plays back with a pseudo-colored depth map rendered fullscreen, where color represents estimated per-pixel depth.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_postprocess_depth_estimation_midasv2`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_depth_estimation_midasv2)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Custom Postprocessing with Gesture Recognition

This example demonstrates gesture recognition using a daisy-chained, multi-model pipeline: palm detection locates the hand region, hand landmark detection extracts keypoints from that region, and a gesture embedder followed by a canned gesture classifier decode the landmarks into a recognized gesture. Preprocessing and inference are handled by the SDK, and the raw output tensors are decoded by an application-defined postprocessing callback. The pipeline uses the on-device camera as its source.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_gesture_recognition`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_gesture_recognition)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_gesture_recognition`
  </Info>

  #### Download Required Files

  Download the gesture recognizer task bundle from Google MediaPipe to obtain the palm detection, hand landmark, gesture embedder, and canned gesture classifier models:

  ```bash theme={null}
  # Download the gesture recognizer task bundle
  wget https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/latest/gesture_recognizer.task

  # Extract the top-level task
  unzip gesture_recognizer.task

  # Extract hand landmarker models
  unzip hand_landmarker.task
  # → hand_detector.tflite, hand_landmarks_detector.tflite

  # Extract gesture recognizer models
  unzip hand_gesture_recognizer.task
  # → gesture_embedder.tflite, canned_gesture_classifier.tflite
  ```

  | File | Save as |
  | - | - |
  | hand\_detector.tflite (see steps above) | palm\_detection\_full.tflite |
  | hand\_landmarks\_detector.tflite (see steps above) | hand\_landmark\_full.tflite |
  | gesture\_embedder.tflite (see steps above) | gesture\_embedder.tflite |
  | canned\_gesture\_classifier.tflite (see steps above) | canned\_gesture\_classifier.tflite |
  | <a href="../labels/palmd_labels.json">palmd\_labels.json</a> | palmd\_labels.json |
  | <a href="../labels/hlandmarks.json">hlandmarks.json</a> | hlandmarks.json |
  | <a href="../labels/gesture_rec.json">gesture\_rec.json</a> | gesture\_rec.json |

  <Note>
    If a downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
        scp palm_detection_full.tflite       <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp hand_landmark_full.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp gesture_embedder.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp canned_gesture_classifier.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp palmd_labels.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp hlandmarks.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp gesture_rec.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_external_postprocess_gesture_recognition
      ```

      The application captures live video from the on-device camera, runs the daisy-chained palm detection → hand landmark → gesture embedder → canned gesture classifier models, and uses the custom, application-defined postprocessing callback to decode the recognized gesture, which is then overlaid and displayed fullscreen.
    </Step>

    <Step title="Expected Output">
      The live camera feed is rendered fullscreen with the recognized gesture label overlaid.

      <video />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_postprocess_gesture_recognition`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_gesture_recognition)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Custom Postprocessing with HRNet (Pose Estimation)

This example demonstrates the pose estimation postprocessing callback (`PoseEstimationPostprocessCallback`) using a two-stage pipeline: a YOLOv5 model detects persons in the frame, and an HRNet model estimates keypoints for each detected person. Preprocessing and inference are handled by the SDK, and the raw output tensors are decoded into keypoints and skeleton links by an application-defined callback, which are then overlaid on the video and displayed.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_pose_estimation_hrnet`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_pose_estimation_hrnet)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_pose_estimation_hrnet`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/models/yolov5?searchTerm=yolov5) | yolov5m-320x320-int8.tflite |
  | [Model](https://aihub.qualcomm.com/models/hrnet_pose?searchTerm=hr) | hrnet\_pose\_w8a8.tflite |
  | <a href="../labels/yolov5m.json">yolov5m.json</a> | yolov5m.json |
  | <a href="../labels/hrnet.json">hrnet.json</a> | hrnet.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | pose\_sample.mp4 |

  <Note>
    No ideal sample video ships with the SDK for this use case. Supply your own content for best results.
  </Note>

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov5m-320x320-int8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp hrnet_pose_w8a8.tflite      <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov5m.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp hrnet.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp pose_sample.mp4             <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_external_postprocess_pose_estimation_hrnet
      ```

      The application runs inference on the video file, first detecting persons with YOLOv5 and then estimating pose keypoints with HRNet, using the custom, application-defined postprocessing callback to decode the results, which are then overlaid and displayed fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back fullscreen with detected persons' pose keypoints and skeleton links overlaid.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_postprocess_pose_estimation_hrnet`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_pose_estimation_hrnet)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>

### Custom Postprocessing with DeepLabv3 (Segmentation)

This example demonstrates the segmentation postprocessing callback (`SegmentationPostprocessCallback`) using a DeepLabv3 model. Preprocessing and inference are handled by the SDK, and the raw output tensors are decoded into per-pixel semantic labels and colors by an application-defined callback, which are then overlaid on the video and displayed.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_segmentation_deeplab`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_segmentation_deeplab)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_segmentation_deeplab`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/deeplabv3_plus_mobilenet) | dv3\_argmax\_int32.tflite |
  | <a href="../labels/dv3-argmax-labels.json">dv3-argmax-labels.json</a> | dv3-argmax-labels.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp dv3_argmax_int32.tflite    <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp dv3-argmax-labels.json     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4         <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      /usr/bin/qimsdk_ref_external_postprocess_segmentation_deeplab
      ```

      The application runs inference on the video file, segmenting the frame with DeepLabv3, and uses the custom, application-defined postprocessing callback to decode the per-pixel segmentation results, which are then overlaid and displayed fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back fullscreen with a color-coded segmentation mask overlaid, where each color represents a detected semantic class.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_postprocess_segmentation_deeplab`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_segmentation_deeplab)
      * **Build instructions:** [Steps to build custom application](/advanced/yocto-build#steps-to-build-custom-application)
    </Step>
  </Steps>
</Accordion>


# Coding Agent
Source: https://imsdkdocs.qualcomm.com/app-builder/gstreamer-coding-agent

Use the QIM SDK GStreamer App Builder skill to generate native C pipeline apps and gst-launch scripts with an AI coding agent.

## What is the GStreamer App Builder Skill?

The **QIM SDK GStreamer App Builder** is an AI coding skill that generates GStreamer pipelines/applications using the `QIM SDK GStreamer API`. You describe the pipeline behavior and configuration in natural language, and the agent produces either a **native GStreamer C application** (`main.c` + `CMakeLists.txt`) or a **gst-launch shell script** — along with a `README.md` that documents the generated app.

<Note>
  The skill can generate both **native C applications** and **gst-launch shell scripts**. The example on this page shows a native C application; use the prompt to request a `gst-launch` script instead when you want a quick, build-free pipeline.
</Note>

**Flow**

<img alt="QIM SDK coding agent workflow" />

**What it generates**

For a native C application request, the agent produces an artifact folder containing:

* **`main.c`** — a complete, buildable native GStreamer C application, including element creation, property configuration, dynamic pad handling, linking, and bus/event-loop management
* **`CMakeLists.txt`** — the build script that compiles `main.c` against GStreamer and links `gstappsutils`
* **`README.md`** — a detailed document covering:
  * Purpose and pipeline behavior summary
  * Configuration placeholders (input, model, labels, output paths)
  * A step-by-step pipeline flow (text summary + Mermaid diagram)
  * Steps to build and run the app on device

For a `gst-launch` request, the agent instead produces a shell script containing the full `gst-launch-1.0` pipeline plus a `README.md`.

**Supported use cases:**

| Category | Details |
| - | - |
| **AI pipelines** | Object detection, image classification, semantic segmentation, face detection, pose estimation, super resolution, depth estimation, audio classification, object tracking, gesture recognition |
| **Inputs** | ISP camera, USB camera, file source (MP4/H.264), RTSP stream |
| **Outputs** | HDMI display, encoded file (MP4), RTSP stream, Redis/MQTT metadata |
| **ML backends** | TFLite (HTP/NPU), QNN, SNPE |
| **Pipeline patterns** | Single-stream, multi-stream, daisy-chained models, ML-bin fused inference, batching, zero-copy cross-process pipelines, gst-launch scripts |

## Prerequisites

### For generating code

<Note>
  Qualcomm dev kits are not needed for code generation.
</Note>

**Host Machine**: Any PC with internet access

**Coding Agent**: Any AI coding agent installed (Claude Code, Cursor, Codex, etc.) which supports skills — the skills provided are agent-agnostic.

<Note>
  This skill is designed to work well even with low-reasoning / smaller models.

  Eg: A Sonnet 4.5 model is sufficient.
</Note>

### For deploying and running apps

The target devices where QIM SDK is supported (see [QIM SDK Installation Guide](/installation))

## How to Use

### Step 1: Get the skill

Choose one installation method:

<Tabs>
  <Tab title="Install Skill">
    Clone the skill repository and copy the **QIM SDK GStreamer App Builder** and **qimsdk-deploy** skills into your coding agent's skills directory:

    ```bash theme={null}
    git clone https://github.com/qualcomm/qimsdk-agentic-skills

    # The following commands are for Claude Code.
    # Set this to your coding agent's skills directory.
    cp -r qimsdk-agentic-skills/skills/qimsdk-gstreamer-app-builder ~/.claude/skills
    cp -r qimsdk-agentic-skills/skills/qimsdk-deploy ~/.claude/skills
    ```

    The skills directory is agent-specific. Check your coding agent's documentation for its skills location.

    Restart or reload your coding agent after installation. Verify that `qimsdk-gstreamer-app-builder` and `qimsdk-deploy` are available with `/skills`.
  </Tab>

  <Tab title="Install Plugin">
    **Claude Code** — run inside a Claude Code session:

    ```text theme={null}
    /plugin marketplace add https://github.com/qualcomm/qimsdk-agentic-skills.git
    /plugin install qimsdk-agentic-skills@qimsdk-marketplace
    /reload-plugins
    ```

    **Codex** — run in your terminal:

    ```bash theme={null}
    codex plugin marketplace add https://github.com/qualcomm/qimsdk-agentic-skills.git
    codex plugin add qimsdk-agentic-skills@qimsdk-marketplace
    ```
  </Tab>
</Tabs>

***

### Step 2: Generate the application

Describe your pipeline to the agent. It will automatically activate the skill and generate a complete `main.c`, `CMakeLists.txt`, and `README.md` (or a `gst-launch` script plus `README.md`).

#### Sample Prompt

**Prompt format — describe pipeline behavior and configuration:**

```
Create a QIM SDK Native C app for single-stream YOLOX object detection from an
MP4 file, encoding the result to a file.

## Pipeline Behavior
- Decode an MP4 file using Qualcomm hardware decoder
- Run YOLOX object detection on full frames using TFLite external delegate on HTP/NPU
- Merge metadata with the video stream before overlay
- Overlay bounding boxes and labels
- Encode the overlaid video and save to an output MP4 file

## Configuration
- Input:   ~/Downloads/qimsdk_samples/media/ai_demo_sample.mp4
- Model:   ~/Downloads/qimsdk_samples/models/yolox_w8a8.tflite
- Labels:  ~/Downloads/qimsdk_samples/labels/yolov8.json
- Backend: TFLite external delegate, HTP/NPU
- Confidence: 51.0
- Output:  ~/Downloads/qimsdk_samples/media/obj_detect_out.mp4
```

<Note>
  To generate a `gst-launch` shell script instead of a C app, ask for it explicitly — e.g. "Create a `gst-launch` script for single-stream YOLOX object detection ...".
</Note>

More sample prompts can be found in the [Available Sample Prompts](#available-sample-prompts) section below.

#### Generated `README.md`

The README documents the generated app in full:

* **Purpose** — a plain-English summary of what the pipeline does
* **Files** — lists `main.c`, `CMakeLists.txt`, and `README.md` with descriptions
* **Assumptions** — codec format, quantization requirements, camera defaults, output directory pre-conditions
* **Configuration** — all user-supplied paths (input, model, labels, output) with instructions on where to change them
* **Placeholders to Fill** — any values you still need to supply, or a note that none remain when the request was fully specified
* **Pipeline Flow** — a `Text Summary` walkthrough of every element plus a `Mermaid Diagram` of the full pipeline
* **Steps to Compile** — how to build the app for Yocto
* **Steps to Run** — exact commands to run on device

<Accordion title="README.md">
  # QIM SDK GStreamer C App — Single-Stream YOLOX Object Detection (MP4 → MP4)

  ## Purpose

  Decode an MP4 file with the Qualcomm hardware decoder, run **YOLOX** object
  detection on full frames using the **TFLite external delegate on the HTP/NPU**,
  merge the detection metadata with the video stream, overlay bounding boxes and
  class labels, then hardware-encode the annotated video to an output MP4 file.
  Headless (no display).

  ## Files

  * `main.c` — GStreamer C sample app (uses `gst_parse_launch`)
  * `CMakeLists.txt` — build target `gst-qimsdk-yolox-obj-detect-encode`
  * `README.md` — this file

  ## Assumptions

  * Input is H.264-in-MP4, decoded with `v4l2h264dec` (`capture-io-mode=4`,
    `output-io-mode=4`).
  * YOLOX (`yolox_w8a8.tflite`) uses the `yolov8` postprocess module with
    `yolov8.json` labels, per the model catalog.
  * Confidence threshold: `{"confidence": 51.0}`.
  * No `bbox-stabilization` — this is a file source, not a live camera feed.
  * Encoder `v4l2h264enc` io-modes `4/4` — correct for decoder-produced (DMA)
    NV12 buffers from a file source.
  * All file paths are resolved at runtime via `g_getenv("HOME")` — C string
    literals do not expand `$HOME`.

  ## Configuration (resolved at runtime from `$HOME`)

  | Value | Path |
  | - | - |
  | Input | `$HOME/Downloads/qimsdk_samples/media/ai_demo_sample.mp4` |
  | Model | `$HOME/Downloads/qimsdk_samples/models/yolox_w8a8.tflite` |
  | Labels | `$HOME/Downloads/qimsdk_samples/labels/yolov8.json` |
  | Output | `$HOME/Downloads/qimsdk_samples/media/obj_detect_out.mp4` |

  ## Placeholders to Fill

  None — all paths and values are concrete. If you need different paths, edit
  the `g_strdup_printf` calls in `main.c`.

  ## Pipeline Flow

  ### Text Summary

  `filesrc` reads the MP4; `qtdemux` demuxes the H.264 stream, `h264parse`
  prepares it, and `v4l2h264dec` hardware-decodes to NV12. A `tee` (`t`) splits:

  * **AI branch**: `qtimlvconverter` → `qtimltflite` (YOLOX, HTP/NPU external
    delegate) → `qtimlpostprocess` (`module=yolov8`, confidence 51.0) →
    `text/x-raw` caps-filter → `qtimetamux.`
  * **Passthrough / video branch**: → `qtimetamux` (`metamux`).

  `qtimetamux` merges metadata onto the video frames, `qtivoverlay` draws
  bounding boxes and labels, and `v4l2h264enc` encodes back to H.264.
  `h264parse` + `mp4mux` + `filesink` write the output MP4.

  ### Mermaid Diagram

  ```mermaid theme={null}
  flowchart TD
    SRC[filesrc] --> DEMUX[qtdemux]
    DEMUX --> PARSE[h264parse]
    PARSE --> DEC[v4l2h264dec]
    DEC --> NV12[video/x-raw NV12]
    NV12 --> TEE[tee t]
    TEE -->|AI| PRE[qtimlvconverter]
    PRE --> INFER[qtimltflite YOLOX HTP]
    INFER --> POST[qtimlpostprocess yolov8]
    POST --> TEXT[text/x-raw]
    TEXT --> MUX[qtimetamux metamux]
    TEE -->|passthrough| MUX
    MUX --> OVL[qtivoverlay]
    OVL --> ENC[v4l2h264enc]
    ENC --> ENCQ[queue]
    ENCQ --> PARSEOUT[h264parse]
    PARSEOUT --> MP4[mp4mux]
    MP4 --> SINK[filesink]
  ```

  ## Steps to Compile

  Yocto: [https://imsdkdocs.qualcomm.com/advanced/yocto-build#steps-to-build-custom-application](https://imsdkdocs.qualcomm.com/advanced/yocto-build#steps-to-build-custom-application)

  ## Steps to Run

  All referenced files — the input MP4, the `yolox_w8a8.tflite` model, and the
  `yolov8.json` labels — must already be present on the device at the paths
  listed in Configuration above.

  ```bash theme={null}
  # Copy the binary to device
  scp gst-qimsdk-yolox-obj-detect-encode <user>@<device-ip>:~/

  # SSH into device and run
  ssh <user>@<device-ip>
  chmod +x ~/gst-qimsdk-yolox-obj-detect-encode
  ./gst-qimsdk-yolox-obj-detect-encode
  ```

  The app runs to end-of-file, finalizes the MP4 container, and exits. The
  output file is written to
  `$HOME/Downloads/qimsdk_samples/media/obj_detect_out.mp4` on the device.
</Accordion>

#### Generated `main.c`

* The generated application uses `gst_parse_launch` to build the full pipeline from a single pipeline string — no manual element creation or pad linking.
* All file paths are resolved at runtime via `g_getenv("HOME")` and formatted with `g_strdup_printf` — C string literals do not expand `$HOME`.
* A GLib main loop and bus watch handle EOS and errors; a `SIGINT` handler (`g_unix_signal_add`) allows clean shutdown.
* `qtivoverlay` draws bounding boxes, and the result is hardware-encoded and written to an MP4 file via `filesink`.

<Accordion title="main.c">
  ```c theme={null}
  #include <glib-unix.h>
  #include <stdio.h>

  #include <gst/sampleapps/gst_sample_apps_utils.h>

  /*
   * Single-stream YOLOX object detection from an MP4 file, encoding the
   * annotated video to an output MP4 file.
   *
   * Pipeline:
   *   filesrc -> qtdemux -> h264parse -> v4l2h264dec -> video/x-raw,format=NV12
   *     -> tee name=t
   *   t. -> qtimlvconverter -> qtimltflite (YOLOX, HTP/NPU external delegate)
   *       -> qtimlpostprocess (module=yolov8, confidence=51.0) -> text/x-raw
   *       -> qtimetamux.
   *   t. -> qtimetamux name=metamux -> qtivoverlay
   *       -> v4l2h264enc -> h264parse -> mp4mux -> filesink
   */

  int main(int argc, char *argv[])
  {
    GstAppContext appctx = { 0 };
    guint interrupt_watch_id = 0;
    GstBus *bus = NULL;
    GError *error = NULL;

    /* Resolve HOME at runtime; C string literals do not expand $HOME. */
    const gchar *home = g_getenv ("HOME");
    if (!home || *home == '\0') {
      g_printerr ("ERROR: HOME is not set; cannot resolve file paths.\n");
      return -1;
    }

    gchar *input_file  = g_strdup_printf (
        "%s/Downloads/qimsdk_samples/media/ai_demo_sample.mp4", home);
    gchar *model_path  = g_strdup_printf (
        "%s/Downloads/qimsdk_samples/models/yolox_w8a8.tflite", home);
    gchar *labels_path = g_strdup_printf (
        "%s/Downloads/qimsdk_samples/labels/yolov8.json", home);
    gchar *output_file = g_strdup_printf (
        "%s/Downloads/qimsdk_samples/media/obj_detect_out.mp4", home);

    gst_init (&argc, &argv);

    /*
     * Build the full pipeline string from source, AI overlay, and encode sink.
     *
     * Notes:
     *  - video/x-raw,format=NV12 caps-filter after hardware decode normalizes
     *    the decoded buffers before branching.
     *  - No bbox-stabilization: this is a file source (not a live camera feed).
     *  - text/x-raw caps-filter routes the postprocess metadata to qtimetamux.
     *  - Encoder io-modes 4/4: file-decoded NV12 (DMA, driver-managed buffers).
     */
    gchar *pipeline_str = g_strdup_printf (
      /* Source: hardware-decode the input MP4 */
      "filesrc location=\"%s\" ! qtdemux ! h264parse ! "
      "v4l2h264dec capture-io-mode=4 output-io-mode=4 ! "
      "video/x-raw,format=NV12 ! "

      /* Split into passthrough (video) branch and AI inference branch */
      "tee name=t "

      /* AI branch: preprocess -> inference -> postprocess -> metadata */
      "t. ! qtimlvconverter name=preprocess ! queue ! "
      "qtimltflite name=inference delegate=external "
      "external-delegate-path=libQnnTFLiteDelegate.so "
      "external-delegate-options=\"QNNExternalDelegate,backend_type=htp,log_level=(string)1;\" "
      "model=\"%s\" ! queue ! "
      "qtimlpostprocess name=postprocess module=yolov8 "
      "labels=\"%s\" "
      "settings=\"{\\\"confidence\\\": 51.0}\" ! text/x-raw ! metamux. "

      /* Passthrough branch: merge metadata, overlay, encode, write */
      "t. ! qtimetamux name=metamux ! "
      "qtivoverlay ! "
      "v4l2h264enc capture-io-mode=4 output-io-mode=4 ! queue ! "
      "h264parse ! mp4mux ! "
      "filesink location=\"%s\"",
      input_file,
      model_path,
      labels_path,
      output_file
    );

    g_print ("Pipeline:\n%s\n\n", pipeline_str);

    appctx.pipeline = gst_parse_launch (pipeline_str, &error);
    g_free (pipeline_str);
    if (!appctx.pipeline) {
      g_printerr ("ERROR: Pipeline creation failed: %s\n",
                  error ? error->message : "unknown error");
      if (error) g_error_free (error);
      goto cleanup;
    }

    appctx.mloop = g_main_loop_new (NULL, FALSE);

    bus = gst_pipeline_get_bus (GST_PIPELINE (appctx.pipeline));
    if (!bus) {
      g_printerr ("ERROR: Failed to get pipeline bus.\n");
      goto cleanup;
    }
    gst_bus_add_signal_watch (bus);
    g_signal_connect (bus, "message::eos",   G_CALLBACK (eos_cb),   appctx.mloop);
    g_signal_connect (bus, "message::error", G_CALLBACK (error_cb), appctx.mloop);
    gst_object_unref (bus);

    interrupt_watch_id = g_unix_signal_add (SIGINT,
        handle_interrupt_signal, &appctx);

    g_print ("Starting pipeline...\n");
    gst_element_set_state (appctx.pipeline, GST_STATE_PAUSED);

    g_main_loop_run (appctx.mloop);

    g_print ("Pipeline finished.\n");
    g_source_remove (interrupt_watch_id);

  cleanup:
    if (appctx.pipeline) {
      gst_element_set_state (appctx.pipeline, GST_STATE_NULL);
      gst_object_unref (appctx.pipeline);
    }
    if (appctx.mloop)
      g_main_loop_unref (appctx.mloop);

    g_free (input_file);
    g_free (model_path);
    g_free (labels_path);
    g_free (output_file);

    gst_deinit ();
    return 0;
  }
  ```
</Accordion>

#### Generated `CMakeLists.txt`

* Uses `pkg-config` to locate the required `gstreamer-1.0` development package.
* Builds a single executable (`gst-qimsdk-yolox-obj-detect-encode`) from `main.c` and links GStreamer libraries plus `gstappsutils`.
* Installs the binary to `${GST_PLUGINS_QTI_OSS_INSTALL_BINDIR}` with standard executable permissions.

<Accordion title="CMakeLists.txt">
  ```cmake theme={null}
  cmake_minimum_required(VERSION 3.16)
  project(GST-QIMSDK-YOLOX-OBJ-DETECT-ENCODE LANGUAGES C CXX)

  set(CMAKE_INCLUDE_CURRENT_DIR ON)

  find_package(PkgConfig)

  pkg_check_modules(GST
    REQUIRED gstreamer-1.0>=${GST_VERSION_REQUIRED})

  set(GST_EXAMPLE_BIN gst-qimsdk-yolox-obj-detect-encode)

  set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wall -Wextra -Wno-unused-parameter")

  add_executable(${GST_EXAMPLE_BIN}
    main.c
  )

  target_include_directories(${GST_EXAMPLE_BIN} PRIVATE
    ${GST_INCLUDE_DIRS}
  )

  target_link_libraries(${GST_EXAMPLE_BIN} PRIVATE
    ${GST_LIBRARIES}
    gstappsutils
  )

  install(
    TARGETS ${GST_EXAMPLE_BIN}
    RUNTIME DESTINATION ${GST_PLUGINS_QTI_OSS_INSTALL_BINDIR}
    PERMISSIONS OWNER_EXECUTE OWNER_WRITE OWNER_READ
                GROUP_EXECUTE GROUP_READ
                WORLD_EXECUTE WORLD_READ
  )
  ```
</Accordion>

***

### Step 3: Running the applications

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy the assets to the device">
    The input video, model, and labels must be present on the device before running — regardless of how you build and run the app. Copy them to the paths `main.c` expects:

    ```bash theme={null}
    # Replace <user> and <device-ip> with your device credentials.
    ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"

    # Copy the assets to device (paths must match main.c)
    scp ai_demo_sample.mp4 <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
    scp yolox_w8a8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
    scp yolov8.json <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
    ```
  </Step>

  <Step title="Build and run the application">
    Unlike a `gst-launch` script, a native C app must be compiled before it can run. With the assets in place, you have two options to build and run the agent-generated app:

    **Option A — Use the QIM SDK Deploy skill:**

    The `qimsdk-deploy` skill builds the app on your host against the SDK, then pushes the compiled binary to the device and runs it over SSH. It handles only the app build and binary transfer — the assets from the previous step must already be on the device.

    **Option B — Follow the generated README:**

    Follow the steps in the [generated `README.md`](#generated-readme-md) to compile the app, then push the binary to the device and run it:

    1. **Compile the app** by following the build steps for your platform:
       * **Yocto:** [Yocto Build](/advanced/yocto-build)

    2. **Push the compiled binary to the device and run it:**

       ```bash theme={null}
       # Copy the built binary to device
       scp gst-qimsdk-yolox-obj-detect-encode <user>@<device-ip>:~/

       # SSH into device and run
       ssh <user>@<device-ip>
       chmod +x ~/gst-qimsdk-yolox-obj-detect-encode
       ./gst-qimsdk-yolox-obj-detect-encode
       ```

    On completion (EOS), the annotated video is written to `~/Downloads/qimsdk_samples/media/obj_detect_out.mp4`.
  </Step>
</Steps>

***

## Available Sample Prompts

<Tabs>
  <Tab title="C App">
    | # | Prompt |
    | - | - |
    | 1 | [Single-stream YOLOX object detection from MP4, encoded to file](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/c-app/AI_01_single_stream_object_detection.md) |
    | 2 | [AI wall — four parallel AI inference streams composed into a 2×2 grid display](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/c-app/AI_02_ai_wall.md) |
    | 3 | [Two-stage detection + classification daisy-chain pipeline](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/c-app/AI_03_detection_classification_daisy_chain.md) |
    | 4 | [Event-triggered recording — record to file only when a person is detected](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/c-app/AI_04_event_triggered_recording.md) |
    | 5 | [Four-stage gesture recognition daisy-chain pipeline](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/c-app/AI_05_gesture_recognition.md) |
    | 6 | [Audio classification — classify audio events from video and overlay results](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/c-app/AI_06_audio_classification.md) |
    | 7 | [Smart codec detection — YOLOv8 detection driving adaptive bitrate encoding](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/c-app/AI_07_smartcodec_detection.md) |
    | 8 | [Event encoder conditional recording from MP4 file, result on Wayland display](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/c-app/AI_08_event_encoder_conditional_recording.md) |
    | 9 | [2-stream face detection from MP4 file, result on Wayland display](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/c-app/AI_09_multistream_face_detection.md) |
    | 10 | [AV playback — H.264 video + MP3 audio from MP4, video on display, audio on speaker](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/c-app/MM_01_av_playback_h264_mp3.md) |
  </Tab>

  <Tab title="gst-launch">
    | # | Prompt |
    | - | - |
    | 1 | [Single-stream YOLOX object detection from MP4, encoded to file](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/gst-launch/AI_01_single_stream_object_detection.md) |
    | 2 | [AI wall — four parallel AI inference streams composed into a 2×2 grid display](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/gst-launch/AI_02_ai_wall.md) |
    | 3 | [Two-stage detection + classification daisy-chain pipeline](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/gst-launch/AI_03_detection_classification_daisy_chain.md) |
    | 4 | [Four-stage gesture recognition daisy-chain pipeline](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/gst-launch/AI_04_gesture_recognition.md) |
    | 5 | [Audio classification — classify audio events from video and overlay results](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/gst-launch/AI_05_audio_classification.md) |
    | 6 | [Single-stream YOLOv8 object detection using SNPE DSP delegate, MP4 to file](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/gst-launch/AI_06_snpe_dsp_object_detection.md) |
    | 7 | [RTSP input YOLOv8 detection — video + metadata streamed over RTSP](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/gst-launch/AI_07_rtsp_input_yolov8_meta_over_rtsp.md) |
    | 8 | [2-stream face detection pipeline](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/gst-launch/AI_08_multistream_face_detection.md) |
    | 9 | [AV record — ISP camera 1080p video + microphone audio muxed to MP4](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/gst-launch/MM_01_av_record_h264_mp3.md) |
    | 10 | [AV playback — H.264 video + MP3 audio from MP4, video on display, audio on speaker](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-gstreamer-app-builder/gst-launch/MM_02_av_playback_h264_mp3.md) |
  </Tab>
</Tabs>


# SDK Architecture and API Reference
Source: https://imsdkdocs.qualcomm.com/app-builder/python-architecture-api-reference

Design and API details of the QIM SDK Pipeline SDK (C++ and Python)

## Pipeline SDK Design

**Overview** This SDK delivers a streamlined, modern interfaces, engineered to facilitate the rapid development, execution, and management of sophisticated multimedia pipelines. By implementing a high-level abstraction layer, the SDK encapsulates complex underlying GStreamer based mechanisms — including element negotiation, signal management, event loops, and state transitions — allowing engineers to prioritize core application logic over boilerplate framework intricacies.

**Key Capabilities** The wrapper empowers developers to architect advanced multimedia and AI-enhanced workflows with precision and efficiency. It supports the seamless integration of diverse components, such as video capture sources, signal processing nodes, machine learning inference engines, graphical overlays, and rendering outputs. Uniquely designed to decouple implementation from the underlying framework, the SDK eliminates the need for deep GStreamer expertise or compile-time dependencies on GStreamer headers, ensuring a clean, robust, and accessible development experience.

<img alt="Introduction" />

## APIs

**Public APIs:**

* `Pipeline` — build & control pipelines fluently.
* `Element` — generic element wrapper with variadic property setting, and state control. It can be instantiated at create time as well as at runtime to update the pipeline. Some elements have special functions which can be controlled by dedicated element types. Those derived types can be used as a plain `Element` if the special functions are not needed.
  * `AppSrc` — app-facing source for zero-copy buffer pull. Feeds application-produced buffers into the pipeline through a callback or a direct push API, and can signal EOS.
  * `AppSink` — app-facing sink for zero-copy buffer push. Delivers pipeline output back to application code with zero copy.
  * `CamSrc` — built-in camera element that exposes an API for capturing images at runtime.
  * `MLVConverter` — ML video converter (`qtimlvconverter`). Exposes an API for custom pre-processing implementation in application code through a callback.
  * `MLPostprocess` — ML postprocess element (`qtimlpostprocess`). Exposes an API for custom post-processing implementation in application code through a callback. The callback's output type annotation selects the ML task (classification, object detection, pose estimation, depth estimation, segmentation, raw tensors).
  * `MLVideoONNXBin` — ONNX inference bin (`qtimlvideoonnxbin`). Exposes an API for custom pre-processing and post-processing implementation through callbacks.
  * `MLVideoQNNBin` — QNN inference bin (`qtimlvideoqnnbin`). Exposes an API for custom pre-processing and post-processing implementation through callbacks.
  * `MLVideoSNPEBin` — SNPE inference bin (`qtimlvideosnpebin`). Exposes an API for custom pre-processing and post-processing implementation through callbacks.
  * `MLVideoTFLiteBin` — TFLite inference bin (`qtimlvideotflitebin`). Exposes an API for custom pre-processing and post-processing implementation through callbacks.
* `Buffer` — opaque buffer that bridges native GStreamer memory with timestamp helpers and zero-copy paths.
* `StreamFilter` and derived concrete stream helpers: `VideoFilter` (all RAW formats - YUV, RGB, bayer), `H264Filter`, `TensorFilter`, `TextFilter`, `AudioFilter`, `ImageFilter`. These filters are used to set characteristics of the streams.
* Logging controls — `SetImsdkLogLevel()` and `SetImsdkGstLogMode()` select the SDK's log verbosity and whether GStreamer's output is simplified or passed through raw. See [Logging and Diagnostics](/app-builder/python-logging).

**Reference flows**

### Pipeline

**Purpose**

A convenient wrapper around a `Gst.Pipeline` that:

* Builds pipelines using element factory names or pre-constructed wrapper elements.
* Manages pipeline lifecycle and execution state.
* Performs delayed linking for dynamic/request-pad elements automatically via `pad-added`.

**Key Responsibilities**

* Create and own the underlying `Gst.Pipeline` instance.
* Add elements, assign properties, and insert caps filters.
* Link elements (immediately when possible, deferred otherwise).
* Manage the full pipeline lifecycle (`prepare`, `start`, `wait`, `stop`).
* Install bus watches to handle `ERROR`, `EOS`, and state transitions.
* Integrate with runtime shutdown notifications.

**A minimal pipeline**

Add the elements in the order data flows through them, then run:

```python theme={null}
from qimsdk import Pipeline, VideoFilter

pipeline = (
    Pipeline("my-pipeline")
    .add("filesrc", "src", "location", "/home/user/media/video.mp4")
    .add("qtdemux", "demux")
    .add("h264parse", "parse")
    .add("v4l2h264dec", "decoder", "output-io-mode", 4, "capture-io-mode", 4)
    .add_stream_filter("vf", VideoFilter().format("NV12"))
    .add("waylandsink", "display", "fullscreen", True)
)

pipeline.execute()
```

`execute()` starts the pipeline, blocks until EOS/error/termination, and tears it down.

<Tip>
  **Adjacent elements are linked in add order.** In the example above `src` → `demux` → `parse` → `decoder` → `vf` → `display` are linked simply because that is the order of the `add()` calls, so a linear graph needs no explicit linking. Use [`link()`](#branching-with-explicit-links) when the graph is not linear — a `tee` splitting into a display branch and an inference branch, or a `qtimetamux` rejoining them.
</Tip>

<Note>
  `qtdemux` is an example of the delayed linking described above: its src pads appear only once the container has been parsed, so the `demux` → `parse` link is completed automatically when the pad shows up. No `pad-added` handling is needed in application code.
</Note>

**Building the pipeline**

| Method | Description |
| - | - |
| `Pipeline(name: str)` | Creates an empty pipeline. `name` is the `Gst.Pipeline` name, used in log messages. |
| `classmethod Pipeline.from_yaml(name: str, config: str)` | Builds a complete pipeline from YAML configuration content. `config` is the YAML text itself, not a path — read the file first. See [Describing a pipeline in YAML](#describing-a-pipeline-in-yaml). |
| `Pipeline.add(factory_or_element, name=None, *args, **props) -> Pipeline` | Creates an element from a factory name (or adds an externally constructed `Element`) under a unique instance name, and applies any trailing key/value properties immediately. Required for the specialized wrappers (`AppSrc`, `AppSink`, `CamSrc`, `MLVConverter`, `MLPostprocess`, the ML bins) so callbacks can be attached before the element joins the pipeline. |
| `Pipeline.add_stream_filter(name, caps=None) -> Pipeline` | Inserts a caps filter between the previous and next element. Accepts either `(name, caps)` or a single `StreamFilter` argument, in which case the element name is derived from the filter's class. See [StreamFilter](#streamfilter). |
| `Pipeline.link(*names) -> Pipeline` | Links the named elements in the given order. Needed only for non-linear graphs. |

`add()`'s optional key/value pairs are the element's properties, validated as soon as the element is added — an unknown property raises `ValueError` naming the property and the element type, instead of being silently ignored:

```python theme={null}
# "fulscreen" is not a waylandsink property -> raises ValueError
pipeline.add("waylandsink", "display", "fulscreen", True)
```

**Branching with explicit links**

Add order cannot express a branch, so name the chains explicitly. Element instance names are the identifiers.

```python theme={null}
pipeline = (
    Pipeline("branching-pipeline")
    .add("filesrc", "src", "location", path)
    .add("qtdemux", "demux")
    .add("h264parse", "parse")
    .add("v4l2h264dec", "decoder", "output-io-mode", 4, "capture-io-mode", 4)
    .add_stream_filter("vf", VideoFilter().format("NV12"))
    .add("tee", "split")
    .add("queue", "q1")
    .add("qtimlvconverter", "preprocessing")
    .add("qtimltflite", "inferencing", "model", model_path)
    .add("qtimlpostprocess", "postprocessing", "module", "yolov8")
    .add_stream_filter("mlf", TextFilter())
    .add("qtimetamux", "mlmuxer")
    .add("qtivoverlay", "overlay")
    .add("waylandsink", "display", "fullscreen", True)
    # trunk, up to the tee
    .link("src", "demux", "parse", "decoder", "vf", "split")
    # branch 1: raw video to the metadata muxer
    .link("split", "mlmuxer")
    # branch 2: inference path, rejoining at the muxer
    .link("split", "q1", "preprocessing", "inferencing", "postprocessing",
          "mlf", "mlmuxer", "overlay", "display")
)

pipeline.execute()
```

<Tip>
  Elements whose src pads are `SOMETIMES` or `REQUEST` — `qtdemux`, `tsdemux`, `tee`, `qtivsplit` — cannot be linked at construction time, because the pads do not exist yet. The pipeline detects this from the pad templates, defers the link, and completes it from a `pad-added` handler it installs itself. Request pads on the sink side (for example `qtimetamux`, `qtivcomposer`) are requested and released automatically. No `pad-added` callback is needed in application code.
</Tip>

<Note>
  A link between two elements that both have only static pads and no common caps fails immediately with `RuntimeError` naming both elements. A deferred link that never resolves surfaces at run time instead; on error the pipeline logs which pads stayed unlinked.
</Note>

**Running the pipeline**

A pipeline is always in one of three states, and the lifecycle methods are the transitions between them:

<img alt="Introduction" />

| State | Meaning |
| - | - |
| `NULL` | No resources allocated. Initial state, and the state after `stop()`. Devices are closed, buffer pools released. |
| `PAUSED` | Elements are allocated and caps are negotiated between linked pads, but the clock is not running and no data flows. Sinks have pre-rolled. This is the state in which elements can be added, removed, or relinked. |
| `PLAYING` | The clock is running and buffers flow through the graph. |

`execute()` performs the whole `NULL → PLAYING → NULL` cycle, so most applications never call the individual transitions. `prepare()`, `activate()`, and `deactivate()` — the transitions through `PAUSED` — are covered in [Advanced state control](#advanced-state-control).

| Method | Description |
| - | - |
| `Pipeline.execute() -> None` | `start()`, `wait()`, `stop()` in sequence. The normal entry point. |
| `Pipeline.start() -> Pipeline` | Moves the pipeline to `PLAYING`. Non-blocking: the state change is asynchronous and this returns without waiting for it. Raises `RuntimeError` if the transition fails. |
| `Pipeline.wait() -> Pipeline` | Blocks until the pipeline is done: `EOS` on the bus, `ERROR` on the bus, `stop()` from another thread, or `SIGINT`/`SIGTERM`. |
| `Pipeline.stop() -> Pipeline` | Moves the pipeline to `NULL` and releases resources. Blocks until teardown completes. |
| `Pipeline.eos(enabled: bool) -> Pipeline` | When enabled, `stop()` first sends an end-of-stream event through the pipeline and waits for it to come back from the sinks, so that data already inside the pipeline is processed before teardown. Off by default. See [Draining the pipeline on shutdown](#draining-the-pipeline-on-shutdown). |

<Warning>
  `start()` is non-blocking, so it must be paired with `wait()` (or your own blocking logic). A script that calls `start()` and exits destroys the pipeline before buffers flow.
</Warning>

To run the pipeline while the application does other work, split `execute()` apart:

```python theme={null}
pipeline.start()   # returns immediately

# ... application work, runtime property changes, snapshots ...

pipeline.wait()    # block until EOS / error / Ctrl+C
pipeline.stop()
```

**Draining the pipeline on shutdown**

By default `stop()` tears the pipeline down as fast as it can. Every element goes to `NULL` immediately and whatever buffers are still in flight — queued between elements, held inside an encoder, not yet written by a sink — are discarded. This is usually what you want: the pipeline stops without delay.

Some pipelines must not lose those buffers, and for them `eos(True)` changes the shutdown sequence. Instead of stopping straight away, `stop()` sends an end-of-stream event into the pipeline. The event travels downstream element by element; each element finishes processing the data it has already accepted, passes the event on, and the sinks report it back on the bus. Only once that end-of-stream notification comes back does `stop()` tear the pipeline down. Nothing that had already entered the pipeline is dropped.

The trade-off is shutdown latency: `stop()` now blocks until the last buffer has made it through the whole graph, which takes as long as the remaining data needs.

Use `eos(True)` when the tail of the stream is part of the result:

* **Muxed recordings.** Containers such as MP4 and MKV are written by a muxer (`mp4mux`, `matroskamux`) that writes its index and header metadata — the `moov` atom, in MP4's case — only on end-of-stream. Cut it off early and the file is left incomplete, often unplayable rather than merely short.
* **Encoders with buffered frames.** Encoders keep several frames in flight for B-frames, lookahead, and similar optimizations. End-of-stream is what makes the encoder flush those frames downstream; without it the tail of the recording is simply missing.
* **Any sink where losing the final buffered data is unacceptable** — file sinks, muxed outputs, and applications that expect to receive every frame through `AppSink`.

Not every sink needs draining, though. A live network stream can usually just be cut off, and a receiver will not care that the last few buffers never arrived. The question is not whether a sink writes somewhere, but whether anything downstream depends on the data still sitting in the pipeline.

Leave `eos()` off (the default) when the newest data is all that matters and stopping promptly is preferable — live preview to a display, a camera feed on screen, or writing self-contained files where each buffer is already complete on its own (JPEG frames through `multifilesink`, for example, which needs no finalization).

```python theme={null}
pipeline = (
    Pipeline("cam-encoder-pipeline")
    .add("qtiqmmfsrc", "source")
    .add_stream_filter("vf", VideoFilter().format("NV12")
                                          .resolution(1920, 1080)
                                          .framerate(30))
    .add("v4l2h264enc", "encoder",
         "output-io-mode", "dmabuf-import", "capture-io-mode", "dmabuf")
    .add("h264parse", "parser")
    .add("mp4mux", "muxer")
    .add("filesink", "sink", "location", f"{HOME_PATH}/media/output.mp4")
    .eos(True)          # drain, so mp4mux can finalize the container
)

pipeline.execute()
```

<Note>
  If the pipeline has already reached end-of-stream on its own — a file source that ran out of data, for instance — `stop()` detects it and tears down immediately instead of sending a second event and waiting again. Enabling `eos(True)` therefore costs nothing for pipelines that end naturally; it only matters when you stop a pipeline that is still running.
</Note>

**Advanced state control**

For reconfiguring a loaded pipeline — adding, removing, or relinking elements without returning to `NULL`.

| Method | Description |
| - | - |
| `Pipeline.prepare() -> Pipeline` | `NULL` → `PAUSED`. Allocates resources and negotiates caps without running the clock. Blocking. |
| `Pipeline.activate() -> Pipeline` | `PAUSED` → `PLAYING`. |
| `Pipeline.deactivate() -> Pipeline` | `PLAYING` → `PAUSED`. Resources stay allocated. |

**Accessing elements at run time**

| Method | Description |
| - | - |
| `Pipeline.get(name: str, as_type=None) -> Any` | Returns the named element. With `as_type` omitted, returns the generic `Element` wrapper. With `as_type` set to a specialized class (for example `CamSrc`), returns that class's `cast()` of the element instead. Raises `KeyError` if the name is not in the pipeline. |

Unlike the C++ SDK's `get<T>()`, Python has no compile-time generics, so the type is passed as an ordinary argument. To reach a specialized element's own API (for example `CamSrc.image_capture()`), pass the class as `as_type`:

```python theme={null}
from qimsdk import CamSrc

# qtiqmmfsrc was added as "source"; CamSrc exposes its capture API.
cam = pipeline.get("source", CamSrc)
cam.image_capture()
```

**Runtime property and per-pad configuration** — `get()` reaches any element by name while the pipeline is loaded or playing:

```python theme={null}
# Change a property at run time.
pipeline.get("display").set("fullscreen", False)

# Per-pad properties on a request pad of qtivcomposer.
pipeline.get("composer").input(1).set("alpha", 0.5)
```

See the sample applications for complete image-capture pipelines.

<Note>
  `get()` returns a new `Element` wrapper on every call, each referencing the same underlying `Gst.Element`. There is no cached identity to keep in sync — call it again whenever you need the element.
</Note>

**Describing a pipeline in YAML**

The graph — elements, properties, stream filters, and links — can be described in YAML instead of in code. This separates *what the pipeline is* from *what the application does with it*: the same script can run a different topology, a different model, or a different source by pointing it at another configuration file.

That makes it useful where the pipeline is not fixed up front:

* **Deploying one script across variants** — the same application against a built-in camera, a USB camera, or an RTSP stream, switching source and format per device.
* **Tuning without code changes** — resolution, framerate, model path, or delegate changed on the target, without touching the script.
* **Keeping topology under review** — the pipeline shape lives in a file that can be diffed and versioned on its own.

Loading from YAML does not give up application logic: the elements are ordinary pipeline members afterwards, so `get()` reaches them as usual.

```yaml theme={null}
pipeline:
  elements:
    - type: filesrc
      name: src
      location: ~/media/video.mp4

    - type: qtdemux
      name: demux

    - type: h264parse
      name: parse

    - type: v4l2h264dec
      name: decoder
      output-io-mode: 4
      capture-io-mode: 4

    - type: filter
      name: videofilter
      video:
        format: NV12

    - type: waylandsink
      name: display
      fullscreen: true
```

Everything lives under a single root key, `pipeline:`, which takes three independent child keys:

```yaml theme={null}
pipeline:
  eos: true             # optional — EOS-on-shutdown, same as .eos(True) in code
  elements:             # required — the elements, in add order
    - type: ...
  links:                # optional — explicit links, for non-linear graphs
    - [ ... ]
```

| Key | Required | Description |
| - | - | - |
| `eos` | No | Boolean. Equivalent to calling [`eos()`](#running-the-pipeline). Defaults to false when omitted. |
| `elements` | Yes | Sequence of elements. Add order is link order, exactly as with `add()`. |
| `links` | No | Sequence of chains, each a list of element names. Equivalent to one `link()` call per entry. Omit it for linear pipelines. |

Within `elements:`, each entry needs `type` (the factory name) and `name` (the instance name); every remaining key is applied as a property of that element. An entry with `type: filter` inserts a stream filter instead, described by a `video:`, `image:`, `h264:`, `tensor:`, `text:`, `audio:`, or raw `caps:` block.

`links:` mirrors [`link()`](#branching-with-explicit-links) — one sequence per chain, so the branching pipeline shown earlier becomes:

```yaml theme={null}
  links:
    - [split, mlmuxer]
    - [source, videostream, split, q1, preprocessing, inferencing,
       postprocessing, mlf, mlmuxer, overlay, display]
```

`from_yaml()` takes the configuration content itself, so read the file first:

```python theme={null}
from qimsdk import Pipeline

with open(config_path, "r", encoding="utf-8") as f:
    config = f.read()

pipeline = Pipeline.from_yaml("demo-pipeline", config)
pipeline.execute()
```

The names in the configuration are the handles the application uses afterwards. Given a config that declares a `qtimlpostprocess` element named `postprocessing`, the application can attach its post-processing callback without the topology being hard-coded:

```python theme={null}
from qimsdk import Pipeline, MLPostprocess, ObjectDetections

with open(config_path, "r", encoding="utf-8") as f:
    config = f.read()

pipeline = Pipeline.from_yaml("demo-pipeline", config)

def detection_callback(mlframe, mlparams, detections: ObjectDetections):
    ...
    return True

postprocessing = pipeline.get("postprocessing", MLPostprocess)
postprocessing.set_handler(detection_callback)

pipeline.execute()
```

### Element

**Purpose**

Generic wrapper around a `Gst.Element` with fluent, variadic property setting. Can be instantiated at create time or used at runtime to update a live pipeline. Serves as the base class for all specialized element types.

**Key Responsibilities**

* Create GStreamer elements from a factory name.
* Provide fluent, variadic property setting for all common types and `StreamFilter` (caps).
* Implement pad linking/unlinking with error checking.
* Support state publication to pipeline (`sync`, `deactivate`, `stop`).
* Expose per-pad configuration through `Port`.
* Serve as the base class for specialized wrappers.

```python theme={null}
from qimsdk import Element

display = Element("waylandsink", "display")
display.set("fullscreen", True)
display.set("sync", False)
```

| Method | Description |
| - | - |
| `Element(factory: str, name: str \| None = None)` | Creates the element from a GStreamer factory name. |
| `Element.set(*args, **props) -> Element` | Sets one or more properties. Accepts key/value pairs positionally, keyword arguments, or both, and returns `self` for chaining. |
| `Element.link(downstream, src_pad="src", sink_pad="sink") -> Element` | Links this element's src pad to `downstream`'s sink pad. |
| `Element.unlink(downstream=None, src_pad="src", sink_pad="sink") -> Element` | Unlinks this element's src pad from `downstream`'s sink pad. |
| `Element.sync() -> Element` | Synchronizes the element's state with its parent pipeline's state. For runtime link/unlink. |
| `Element.deactivate() -> Element` | Sets the element's state to `PAUSED`. |
| `Element.stop() -> Element` | Moves the element to `NULL` state. For runtime link/unlink. |
| `Element.input(name_or_id=0, id=None) -> Port` | Returns a reference to an input port, for per-pad configuration. |
| `Element.output(name_or_id=0, id=None) -> Port` | Returns a reference to an output port, for per-pad configuration. |

**Per-pad configuration with Port**

Some elements are configured per pad rather than per element — a compositor's inputs each have their own position, size, and alpha. `input()` and `output()` return a `Port` for one pad, which carries the same fluent, validated `set()`:

```python theme={null}
# Second input of qtivcomposer: blend at 50% alpha.
pipeline.get("composer").input(1).set("alpha", 0.5)
```

| Method | Description |
| - | - |
| `Port.set(*args, **props) -> Port` | Sets one or more pad properties, same as `Element.set()`. |

### AppSrc

**Purpose**

`AppSrc` wraps the `appsrc` element, the entry point for feeding application-produced buffers into a pipeline. It manages the `need-data`/`enough-data` signal plumbing so the application only has to supply buffers, either from a callback or by pushing them directly.

**Key Responsibilities**

* Validate that the wrapped element is indeed an `appsrc`.
* Register and dispatch the underlying `need-data` and `enough-data` signals.
* Let the application supply buffers on demand through `set_buffer_producer()`, or push them directly with `push_buffer()`.
* Signal end-of-stream on this element's output.
* Manage lifetime of the connected signal handlers.

```python theme={null}
from qimsdk import AppSrc, VideoFilter

appsrc = AppSrc("src")
appsrc.set("is-live", True, "block", True)
appsrc.set("format", AppSrc.Format.TIME, "do-timestamp", True)
appsrc.set("caps", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
```

| Method | Description |
| - | - |
| `AppSrc(name: str \| None = None)` | Instantiates an application source element with an optional instance name. |
| `classmethod AppSrc.cast(element) -> AppSrc` | Wraps an existing `Element` (for example one returned by `Pipeline.get()`) as an `AppSrc`. |
| `AppSrc.set(*args, **props) -> AppSrc` | Sets one or more properties, same as `Element.set()`. |
| `AppSrc.set_caps(caps) -> AppSrc` | Sets the `caps` property from a `StreamFilter`, a raw caps string, or a `Gst.Caps` object. |
| `AppSrc.set_buffer_producer(producer) -> AppSrc` | Attaches a callback executed when more data is needed. Data control is driven by the pipeline. |
| `AppSrc.set_enough_handler(enough) -> AppSrc` | Attaches a callback executed when the data source should be stopped. Data control is driven by the pipeline. |
| `AppSrc.push_buffer(buffer: Buffer) -> bool` | Pushes a buffer into the AppSrc/pipeline stream. Data control is driven by the application. |
| `AppSrc.end_of_stream() -> None` | Sends an end-of-stream event to the AppSrc/pipeline stream. |

`AppSrc.Format` mirrors appsrc's own `format` property — an `IntEnum` with `DEFAULT`, `BYTES`, `TIME`, `BUFFERS`, `PERCENT`.

### AppSink

**Purpose**

`AppSink` wraps the `appsink` element, the exit point that delivers pipeline output back to application code. It pulls samples through appsink's action signals and hands them to the application as `Buffer`.

**Key Responsibilities**

* Validate that the wrapped element is indeed an `appsink`.
* Pull samples via appsink's action signals and convert them into `Buffer`.
* Deliver each buffer to the application through `set_buffer_consumer()`.
* Deliver the first (preroll) sample separately through `set_preroll_handler()`.
* Notify the application of end-of-stream through `set_eos_handler()`.
* Manage lifetime of the connected signal handlers.

```python theme={null}
from qimsdk import AppSink

appsink = AppSink("sink")
appsink.set("emit-signals", True, "max-buffers", 5, "drop", True)
appsink.set_buffer_consumer(lambda buffer: consume(buffer))
```

| Method | Description |
| - | - |
| `AppSink(name: str \| None = None)` | Instantiates an application sink element with an optional instance name. |
| `classmethod AppSink.cast(element) -> AppSink` | Wraps an existing `Element` (for example one returned by `Pipeline.get()`) as an `AppSink`. |
| `AppSink.set(*args, **props) -> AppSink` | Sets one or more properties, same as `Element.set()`. |
| `AppSink.set_buffer_consumer(consumer) -> AppSink` | Attaches a listener for buffers pulled from the pipeline. |
| `AppSink.set_preroll_handler(preroll) -> AppSink` | Attaches a listener for the first (preroll) sample only. |
| `AppSink.set_eos_handler(eos) -> AppSink` | Attaches a listener for end-of-stream. |

### CamSrc

**Purpose**

`CamSrc` wraps the `qtiqmmfsrc` camera source element and exposes its still/burst image-capture action signals as ordinary method calls, so an application can trigger a snapshot without touching GStreamer signals directly.

**Key Responsibilities**

* Validate that the wrapped element is indeed a `qtiqmmfsrc`.
* Trigger still or burst image capture through `image_capture()`.
* Cancel any pending capture request through `cancel_capture()`.

| Method | Description |
| - | - |
| `CamSrc(name: str \| None = None)` | Instantiates a camera source element with an optional instance name. |
| `classmethod CamSrc.cast(element) -> CamSrc` | Wraps an existing `Element` (for example one returned by `Pipeline.get()`) as a `CamSrc`. |
| `CamSrc.image_capture(*args, **kwargs) -> bool` | Captures still or burst images. Accepts `count`, `mode`, and `metadata` positionally or as keyword arguments. |
| `CamSrc.cancel_capture() -> bool` | Cancels any capture requests still pending. |

Reach a `qtiqmmfsrc` already in the pipeline through `Pipeline.get()`'s `as_type` argument to use its capture API:

```python theme={null}
from qimsdk import CamSrc

cam = pipeline.get("source", CamSrc)
cam.image_capture()
```

### MLVConverter

**Purpose**

`MLVConverter` wraps the `qtimlvconverter` element, which converts raw video frames into the input tensor format expected by a model. Beyond the properties available on the generic `Element`, it lets the application replace the built-in conversion with its own implementation through a callback.

**Key Responsibilities**

* Validate that the wrapped element is a `qtimlvconverter`.
* Convert incoming video frames into model input tensors.
* Optionally delegate the conversion to an application-supplied pre-process callback.

```python theme={null}
from qimsdk import MLVConverter

preprocessing = MLVConverter("preprocessing")
# engine=none disables internal preprocessing path in qtimlvconverter.
preprocessing.set("engine", "none")
preprocessing.set_handler(preprocess_callback)
```

| Method | Description |
| - | - |
| `MLVConverter(name: str \| None = None)` | Instantiates a `qtimlvconverter` element with an optional instance name. |
| `classmethod MLVConverter.cast(element) -> MLVConverter` | Wraps an existing `Element` (for example one returned by `Pipeline.get()`) as an `MLVConverter`. |
| `MLVConverter.set(*args, **props) -> MLVConverter` | Sets one or more properties, same as `Element.set()`. |
| `MLVConverter.set_handler(handler) -> MLVConverter` | Registers the external pre-process callback. Requires `engine=none` to take effect. |

### MLPostprocess

**Purpose**

`MLPostprocess` wraps the `qtimlpostprocess` element, which decodes model output tensors into ML metadata attached to the frame. It lets the application replace the built-in decoding module with its own implementation through a callback.

**Key Responsibilities**

* Validate that the wrapped element is a `qtimlpostprocess`.
* Decode output tensors into ML metadata (classifications, detections, poses, depth maps, segmentations, or raw tensors).
* Optionally delegate the decoding to an application-supplied post-process callback.
* Select the callback's ML task automatically from its output-parameter type annotation.

**Post-process callback**

`set_handler()` takes a single callback and, unlike a per-task setter, infers which ML task it handles from the type annotation on the callback's output parameter. The callback takes either 3 parameters (`mlframe, mlparams, results`) or 4 (`mlpostprocess, mlframe, mlparams, results`); the output parameter must be annotated with one of the marker types below.

| Output annotation | ML task |
| - | - |
| `ImageClassifications` | Image classification |
| `AudioClassifications` | Audio classification |
| `ObjectDetections` | Object detection |
| `Poses` | Pose estimation |
| `DepthMaps` | Depth estimation |
| `Segmentations` | Image segmentation |
| `Tensors` | Raw output tensors |

```python theme={null}
from qimsdk import MLPostprocess, ImageClassifications

def classification_callback(mlframe, mlparams, classifications: ImageClassifications):
    ...
    return True

postprocessing = MLPostprocess("postprocessing")
postprocessing.set("results", 1)
postprocessing.set_handler(classification_callback)
```

| Method | Description |
| - | - |
| `MLPostprocess(name: str \| None = None)` | Instantiates a `qtimlpostprocess` element with an optional instance name. |
| `classmethod MLPostprocess.cast(element) -> MLPostprocess` | Wraps an existing `Element` (for example one returned by `Pipeline.get()`) as an `MLPostprocess`. |
| `MLPostprocess.set(*args, **props) -> MLPostprocess` | Sets one or more properties, same as `Element.set()`. |
| `MLPostprocess.set_handler(handler, kind: str \| None = None) -> MLPostprocess` | Registers a callback that decodes tensors into the ML task selected by `kind`, or inferred from the callback's output-parameter type annotation if `kind` is omitted. |

### ML inference bins

**Purpose**

The ML video bins wrap a full pre-process → inference → post-process chain into a single element, so a whole AI branch can be added to the pipeline as one node instead of being assembled from separate elements. Four bins are available, one per inference runtime:

| Python class | GStreamer element | Runtime |
| - | - | - |
| `MLVideoONNXBin` | `qtimlvideoonnxbin` | ONNX Runtime |
| `MLVideoQNNBin` | `qtimlvideoqnnbin` | Qualcomm® AI Engine Direct (QNN) |
| `MLVideoSNPEBin` | `qtimlvideosnpebin` | Snapdragon® Neural Processing Engine (SNPE) |
| `MLVideoTFLiteBin` | `qtimlvideotflitebin` | LiteRT (TensorFlow Lite) |

All four expose an identical API and differ only in the wrapped element and the runtime-specific properties. Each accepts a custom pre-process callback, a custom post-process callback, or both.

**Key Responsibilities**

* Validate that the wrapped element matches the expected bin factory.
* Run pre-processing, inference, and post-processing inside a single element.
* Optionally delegate pre-processing to an application-supplied callback via `set_preprocess_handler()`.
* Optionally delegate post-processing to an application-supplied callback via `set_postprocess_handler()`.

**Properties**

Because the bin embeds several stages, its properties are prefixed by the stage they configure — `preprocess-*`, `inference-*`, and `postprocess-*`. For example `inference-model` on the bin corresponds to `model` on a standalone inference element, and `postprocess-labels` corresponds to `labels` on a standalone `qtimlpostprocess`.

<Note>
  Set `preprocess-engine` to `none` to disable the internal pre-processing path when you supply your own pre-process callback.
</Note>

<Note>
  The bin's callbacks are `set_preprocess_handler()` and `set_postprocess_handler()` — distinct names from the standalone `MLVConverter.set_handler()` and `MLPostprocess.set_handler()`, since a bin exposes both stages on one object.
</Note>

**Usage**

**Custom post-processing** — the bin handles pre-processing and inference internally, the application decodes the output tensors:

```python theme={null}
from qimsdk import MLVideoTFLiteBin, VideoFilter, ObjectDetections

def detection_callback(mlframe, mlparams, detections: ObjectDetections):
    ...
    return True

mlbin = MLVideoTFLiteBin("mlbin")
mlbin.set("inference-delegate", "external",
          "inference-external-delegate-path", "libQnnTFLiteDelegate.so",
          "inference-external-delegate-options",
          "QNNExternalDelegate,backend_type=htp;",
          "inference-model", f"{HOME_PATH}/models/yolov8_det_quantized.tflite")
mlbin.set_postprocess_handler(detection_callback)

pipeline = (
    Pipeline("mlbin-external-detection")
    .add("filesrc", "src", "location", f"{HOME_PATH}/media/video.mp4")
    .add("qtdemux", "demux")
    .add("h264parse", "parse")
    .add("v4l2h264dec", "decoder", "output-io-mode", 4, "capture-io-mode", 4)
    .add_stream_filter("vf", VideoFilter().format("NV12"))
    .add(mlbin)
    .add("qtivoverlay", "overlay")
    .add("waylandsink", "display", "fullscreen", True)
)

pipeline.execute()
```

**Custom pre-processing** — the application converts frames into input tensors, the bin runs inference and its built-in post-process module:

```python theme={null}
mlbin = MLVideoTFLiteBin("mlbin")

# preprocess-engine=none disables internal preprocessing path in mlbin.
mlbin.set("preprocess-engine", "none")
mlbin.set("inference-delegate", "external")
mlbin.set("inference-external-delegate-path", "libQnnTFLiteDelegate.so")
mlbin.set("inference-external-delegate-options",
          "QNNExternalDelegate,backend_type=htp;")
mlbin.set("inference-model", f"{HOME_PATH}/models/yolov8_det_quantized.tflite")
mlbin.set("postprocess-module", "yolov8")
mlbin.set("postprocess-labels", f"{HOME_PATH}/labels/yolov8.json")

def preprocess_callback(blits, output) -> bool:
    ...
    return True

mlbin.set_preprocess_handler(preprocess_callback)
```

Swapping runtimes is a matter of changing the class and the runtime-specific properties; the callback registration is unchanged.

```python theme={null}
from qimsdk import MLVideoQNNBin, MLVideoSNPEBin, MLVideoONNXBin

qnn_bin = MLVideoQNNBin("mlbin")     # qtimlvideoqnnbin
snpe_bin = MLVideoSNPEBin("mlbin")   # qtimlvideosnpebin
onnx_bin = MLVideoONNXBin("mlbin")   # qtimlvideoonnxbin
```

**APIs**

The table below applies to each of `MLVideoONNXBin`, `MLVideoQNNBin`, `MLVideoSNPEBin`, and `MLVideoTFLiteBin`; substitute the concrete class name for `<Bin>`.

| Method | Description |
| - | - |
| `<Bin>(name: str \| None = None)` | Creates the corresponding `qtimlvideo*bin` element with an optional instance name. |
| `<Bin>.set(*args, **props) -> <Bin>` | Sets one or more properties, same as `Element.set()`. Properties are stage-prefixed: `preprocess-*`, `inference-*`, `postprocess-*`. |
| `<Bin>.set_preprocess_handler(handler) -> <Bin>` | Registers the external pre-process callback. Requires `preprocess-engine=none` to take effect. |
| `<Bin>.set_postprocess_handler(handler, kind: str \| None = None) -> <Bin>` | Registers the external post-process callback. Accepts the same callback shapes as `MLPostprocess.set_handler()`. |

### Buffer

**Purpose**

Unified buffer abstraction over native GStreamer memory. Provides read/write access, timestamp metadata (pts, dts, duration), resize support, and zero-copy transfer between AppSink and AppSrc.

**Key Responsibilities**

* Own and manage mapping/unmapping of underlying `Gst.Buffer` or `Gst.Sample`.
* Provide read/write access to buffer memory.
* Support resizing and writable allocation for AppSrc.
* Carry timestamps (PTS/DTS/duration) and apply them back to `Gst.Buffer`.
* Enable zero-copy extraction using `take_gst_buffer()`.

```python theme={null}
def on_buffer(buf):
    data = buf.data()
    pts = buf.pts

appsink.set_buffer_consumer(on_buffer)
```

| Method | Description |
| - | - |
| `Buffer(gst_buffer=None, size=None)` | Wraps an existing `Gst.Buffer`, or allocates a new one of `size` bytes. |
| `classmethod Buffer.from_readable_sample(sample) -> Buffer` | Wraps the buffer inside a `Gst.Sample`, as delivered by `AppSink`'s preroll handler. |
| `Buffer.data() -> memoryview \| None` | Maps the buffer for reading (or writing, if writable) and returns a `memoryview` onto it. |
| `Buffer.size() -> int` | Returns the buffer's size in bytes. |
| `Buffer.resize(size: int) -> None` | Reallocates the buffer to `size` bytes, preserving existing data. |
| `Buffer.is_writable() -> bool` | Returns whether the buffer can be mapped for writing. |
| `Buffer.is_readonly() -> bool` | Returns `True` if the buffer is valid but not writable. |
| `Buffer.valid() -> bool` | Returns `True` if the buffer wraps a `Gst.Buffer`. |
| `Buffer.take_gst_buffer() -> Any \| None` | Transfers ownership of the underlying `Gst.Buffer` and clears this wrapper, enabling zero-copy hand-off (used internally by `AppSrc.push_buffer()`). |
| `Buffer.set_pts(pts_ns)` / `Buffer.set_dts(dts_ns)` / `Buffer.set_duration(duration_ns)` | Sets the buffer's timestamp fields, in nanoseconds. |
| `Buffer.pts` / `Buffer.dts` / `Buffer.duration` | Read-only properties for the buffer's timestamp fields; `None` if the buffer is empty. |

### StreamFilter

**Purpose**

StreamFilter describes the characteristics of a media stream between two pipeline stages so they can negotiate concrete parameters rather than relying on defaults. Without explicit parameters, adjacent components may fall back to suboptimal formats, trigger implicit conversions, or produce mismatched configurations.

Specialized subtypes provide fluent, type-safe setters for each domain. Raw caps strings are also accepted for cases not covered by typed setters or when copying directly from gst-launch output.

**When to Use StreamFilter**

* When two adjacent stages can interoperate in multiple ways, and you want to select explicit parameters instead of relying on defaults.
* When you need to optimize latency or performance by avoiding implicit conversions (e.g., choose a hardware-friendly pixel format).
* When you want predictable behavior across platforms (e.g., lock color range/primaries/transfer/matrix).
* When you integrate tensors for AI/ML and must precisely specify dimensions and element types.
* When you need a human-readable summary of the stream parameters for logs and debugging.

A `StreamFilter` is not an `Element` and is never added to a pipeline with `add()`. It is always attached between stages with `Pipeline.add_stream_filter(name, filter)`, which internally creates a `capsfilter` element and applies the filter's caps to it. The base `StreamFilter` class itself only exposes a raw-caps-string constructor and `to_string()`; the fluent per-domain setters (`.format()`, `.resolution()`, `.framerate()`, ...) live on the typed subclasses such as `VideoFilter`.

**Usage**

**Typed subclass**

```python theme={null}
from qimsdk import Pipeline, VideoFilter

pipeline = (
    Pipeline("video-pipeline")
    .add("videotestsrc", "src", "pattern", "ball")
    .add_stream_filter("filter", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
    .add("waylandsink", "sink")
)

pipeline.execute()
```

**Raw caps string**

```python theme={null}
from qimsdk import Pipeline, StreamFilter

pipeline = (
    Pipeline("video-pipeline-streamfilter-string")
    .add("videotestsrc", "src", "pattern", "ball")
    .add_stream_filter("filter", StreamFilter(
        "video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1"))
    .add("waylandsink", "sink")
)

pipeline.execute()
```

**APIs**

| Method | Description |
| - | - |
| `StreamFilter(caps: str)` | Creates a filter from a raw caps or media-type string, e.g. `"video/x-raw,format=NV12"`. |
| `StreamFilter.to_string() -> str` | Serializes the current filter expression to a caps string, for logging/debugging. |

### VideoFilter

**Purpose**

Configures a raw video stream (YUV, RGB, Bayer) between a camera source and any downstream processing or display element. Supports format, resolution, framerate, colorimetry, interlace mode, and pixel aspect ratio.

**Key Responsibilities**

* Produce caps with standard raw video attributes: format, resolution, framerate, colorimetry, interlace mode, pixel aspect ratio, etc.

**APIs**

| Method | Description |
| - | - |
| `format(fmt: str \| VideoFormat)` | Sets the stream's pixel format, for example: `NV12`, `YUY2`, `RGB`, `RGBA`, etc. Determines how pixel data is laid out in memory; essential for hardware compatibility and color-conversion paths. |
| `resolution(w: int, h: int)` | Specifies the frame dimensions in pixels. |
| `framerate(num: int \| float, den: int = 1)` | Sets the frame rate as a fraction, for example: `30/1`, `30000/1001` frames-per-second. You can pass only `num` if you do not need a fractional frame rate, like `30.0` fps. |
| `colorimetry(value: str)` | Defines the colorimetry standard, for example: `bt601`, `bt709`, `bt2020`. |
| `range(value: str)` | Selects the quantization range, for example: `full` or `limited`. |
| `interlace(mode: str)` | Sets the interlacing mode, for example: `progressive`, `interleaved`, etc. Progressive is standard for modern pipelines; interlaced is rare but still used in some camera/legacy systems. |
| `pixel_aspect_ratio(num: int, den: int = 1)` | Specifies the pixel aspect ratio, also called PAR. Normally `1/1`; used for anamorphic or non-square-pixel formats. |
| `add(expr: str)` | Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API. |

**Usage**

We can achieve the same results in two different ways:

**Typed (enum) API** - if you want IDE autocompletion and guardrails against typos. Ideal for most application code.

```python theme={null}
from qimsdk import VideoFilter, VideoFormat

v1 = (
    VideoFilter()
    .format(VideoFormat.NV12)
    .resolution(1920, 1080)
    .framerate(30)
    .colorimetry("bt709")
    .range("limited")
    .interlace("progressive")
)
```

**String API** - if you need quick iteration or to pass values not (yet) covered by the enum — e.g., a freshly added GStreamer value on a target platform.

```python theme={null}
from qimsdk import VideoFilter

v2 = (
    VideoFilter()
    .format("NV12")
    .resolution(1920, 1080)
    .framerate(30)
    .colorimetry("bt709")
    .range("limited")
    .interlace("progressive")
)
```

**Raw caps string** - very useful for setting an arbitrary stream (caps) filter, or if you want to lift an existing caps line from logs, gst-launch, or a pad template and drop it as-is — great for debugging, prototyping, or reproducing pipelines exactly.

```python theme={null}
from qimsdk import StreamFilter

v3 = StreamFilter("video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1,colorimetry=bt709,range=limited")
```

### H264Filter

**Purpose**

Configures an encoded H.264 stream between an encoder and a downstream parser or muxer. Supports resolution, framerate, stream-format, alignment, profile, and level.

**Key Responsibilities**

* Specify resolution, framerate, stream-format, alignment, profile, level.
* Insert `codec_data` buffers.
* Attach arbitrary name/value string properties.

**APIs**

| Method | Description |
| - | - |
| `resolution(w: int, h: int)` | Specifies the frame dimensions in pixels. |
| `framerate(num: int \| float, den: int = 1)` | Sets the frame rate as a fraction. |
| `profile(profile: str)` | Sets the H.264 profile caps field, for example `baseline`, `main`, `high`. |
| `level(level: str)` | Sets the H.264 level caps field. |
| `stream_format(fmt: str)` | Sets the H.264 stream-format caps field, for example `byte-stream`, `avc`. |
| `alignment(value: str)` | Sets the H.264 alignment caps field, for example `au`, `nal`. |
| `codec_data(value: str)` | Sets the H.264 `codec_data` caps field. |
| `set(key: str, value: str)` | Shorthand for `add(f"{key}={value}")`. |
| `add(expr: str)` | Appends an arbitrary custom caps field using a `key=value` expression. |

**Usage**

```python theme={null}
from qimsdk import Pipeline, H264Filter

h264f = (
    H264Filter()
    .resolution(1920, 1080)
    .framerate(30)
    .profile("baseline")
    .stream_format("byte-stream")
    .alignment("au")
)

pipeline = Pipeline("h264-pipeline").add_stream_filter("h264f", h264f)
```

### TensorFilter

**Purpose**

Configures an ML tensor stream with explicit data type and shape (e.g., UINT8, NHWC). Not needed in a full inference pipeline — when preprocess, inference, and postprocess elements are all present, tensor parameters are negotiated automatically. Use TensorFilter when:

* The preprocessing stage does not have model dimension information

* You need exact tensor data output from the inference plugin

**Key Responsibilities**

* Declare element type (e.g., `UINT8`, `FLOAT32`) for tensors.
* Specify shape (dimensions). Supports:
  * Single tensor: a flat list of ints (e.g., `1, 520, 520, 3`).
  * Multiple tensors: a list of lists (e.g., `[[1,3,H,W], [1,32]]`) when needed.

**APIs**

| Method | Description |
| - | - |
| `type(t: TensorFilter.Type \| str)` | Specify tensor data type, for example: `UINT8`, `FLOAT32`. |
| `dimensions(*dims)` | Specify tensor dimensions. Pass ints directly for a single tensor, or one or more `list`/`tuple` arguments for multiple tensors. |
| `add(expr: str)` | Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API. |

**Usage**

We can achieve the same results in two different ways:

**Typed (enum) API** - if you want IDE autocompletion and guardrails against typos. Ideal for most application code.

```python theme={null}
from qimsdk import TensorFilter

t1 = (
    TensorFilter()
    .type(TensorFilter.Type.UINT8)
    .dimensions(1, 520, 520, 3)       # NHWC, for example
)
```

**String API** - if you need quick iteration or to pass values not (yet) covered by the enum.

```python theme={null}
from qimsdk import TensorFilter

t2 = (
    TensorFilter()
    .type("FLOAT32")
    .dimensions(1, 3, 640, 640)       # NCHW, for example
)
```

**Raw caps string** - very useful for setting an arbitrary stream (caps) filter, or if you want to lift an existing caps line from logs, gst-launch, or a pad template and drop it as-is.

```python theme={null}
from qimsdk import StreamFilter

t3 = StreamFilter("neural-network/tensors,type=UINT8,dimensions=<<1,520,520,3>>")
```

### TextFilter

**Purpose**

Configures a raw text stream (text/x-raw) for pipelines that carry textual data such as metadata, subtitles, or logs. Minimal by design — attach only the fields you need using `.add("key=value")`.

**Key Responsibilities**

* Provide a lightweight caps descriptor for text (`text/x-raw`) without enforcing a schema.
* Let callers attach arbitrary properties (e.g., encoding/charset, language, packetization hints) using `.add("key=value")`.

**APIs**

| Method | Description |
| - | - |
| `add(expr: str)` | Appends an arbitrary custom caps field using a `"key=value"` expression. Escape-hatch for advanced/experimental fields not covered by the typed API. |

**Usage**

```python theme={null}
from qimsdk import TextFilter

txt = TextFilter().add("charset=utf8")
```

### ImageFilter

**Overview**

Used exclusively with CamSrc (`qtiqmmfsrc`) to enable and link its dedicated image capture pad. Without ImageFilter, the image pad is not requested, not activated, and produces no buffers. It is a structural enabler, not a processing stage — it is not intended for general image processing or frame extraction from video sources.

`ImageFilter` is used to:

* Enable the image output pad exposed by `CamSrc` (`qtiqmmfsrc`)
* Attach that pad to the pipeline so the source can activate image capture
* Provide a concrete sink target required by GStreamer for pad activation

It is not intended for:

* Generic image processing
* Frame extraction from arbitrary video sources
* Replacement of `VideoFilter` or `AppSink`

**Relationship to CamSrc (qtiqmmfsrc)**

* `CamSrc` (`qtiqmmfsrc`) exposes a dedicated image pad that remains inactive unless linked.
* `ImageFilter` explicitly exists to satisfy this requirement.
* Without `ImageFilter`, the image pad:
  * Will not be requested
  * Will not be activated
  * Will not produce buffers

This makes `ImageFilter` a structural enabler, not a processing stage.

**Usage**

```python theme={null}
from qimsdk import ImageFilter

imagefilter1 = (
    ImageFilter()
    .format("JPEG")
    .resolution(3840, 2160)
)
```

### AudioFilter

**Purpose**

Configures a raw PCM audio stream with sample format, channel count, sample rate, and layout. Use this to ensure audio converters and sinks negotiate exactly the format your hardware supports rather than inserting an implicit resampler.

**Key Responsibilities**

* Set audio sample format (e.g., `S16LE`, `F32LE`) to match HW/SW capabilities.
* Define channels and rate for correct resampling/mixing.
* Choose layout (`interleaved` vs `non-interleaved`) to align with downstream expectations.

**APIs**

| Method | Description |
| - | - |
| `format(fmt: str \| AudioFormat)` | Specifies the audio sample format, e.g. `S16LE`, `F32LE`. |
| `channels(n: int)` | Specifies the number of audio channels. |
| `rate(hz: int)` | Specifies the audio sample rate in Hz. |
| `layout(layout: str \| AudioLayout)` | Specifies the audio layout, `interleaved` or `non-interleaved`. |
| `add(expr: str)` | Appends an arbitrary custom caps field using a `"key=value"` expression. Escape-hatch for advanced/experimental fields not covered by the typed API. |

**Usage**

We can achieve the same results in two different ways:

**Typed (enum) API** - if you want IDE autocompletion and guardrails against typos. Ideal for most application code.

```python theme={null}
from qimsdk import AudioFilter, AudioFormat, AudioLayout

a1 = (
    AudioFilter()
    .format(AudioFormat.S16LE)
    .rate(48000)
    .channels(2)
    .layout(AudioLayout.INTERLEAVED)
)
```

**String API** - if you need quick iteration or to pass values not (yet) covered by the enums.

```python theme={null}
from qimsdk import AudioFilter

a2 = (
    AudioFilter()
    .format("F32LE")
    .rate(44100)
    .channels(1)
    .layout("non-interleaved")
)
```

**Raw caps string** - very useful for setting an arbitrary stream (caps) filter, or if you want to lift an existing caps line from logs, gst-launch, or a pad template and drop it as-is.

```python theme={null}
from qimsdk import StreamFilter

a3 = StreamFilter("audio/x-raw,format=S16LE,rate=48000,channels=2,layout=interleaved")
```

### Logging

The SDK can either simplify GStreamer's log stream into a compact, uniform format or step aside and let GStreamer log raw, with `GST_DEBUG` fully in control. Both modes, the verbosity levels, the integration with Python's `logging` module, and the ordering constraint on configuring them are covered in [Logging and Diagnostics](/app-builder/python-logging).

## QIM SDK Nodes

All nodes are executed by GStreamer elements under the hood. Most elements are handled through the `Element` base class. However, elements that provide additional functionality — such as application buffer interfaces in appsink and appsrc, or image-capture signals in camera sources like `qtiqmmfsrc` — have a specialized Python wrapper class.

If you do not need these extra capabilities (for example, image capture), you can still use these elements via the generic `Element` base class without any limitations for standard data processing.

Below is a table listing all supported elements and the corresponding QIM SDK Python wrapper classes used to work with them.

**Sources**

| GStreamer | QIM SDK Python Class |
| - | - |
| `qtiqmmfsrc` | `CamSrc` (also usable as `Element`) |
| `v4l2src` | `Element` |
| `filesrc` | `Element` |
| `multifilesrc` | `Element` |
| `appsrc` | `AppSrc` |
| `appsink` | `AppSink` |
| `qtisocketsrc` | `Element` |

**AI**

| GStreamer | QIM SDK Python Class |
| - | - |
| `qtimlaconverter` | `Element` |
| `qtimlvconverter` | `MLVConverter` (also usable as `Element`) |
| `qtimlqnn` | `Element` |
| `qtimlsnpe` | `Element` |
| `qtimltflite` | `Element` |
| `qtimlpostprocess` | `MLPostprocess` (also usable as `Element`) |
| `qtimlvideoonnxbin` | `MLVideoONNXBin` (also usable as `Element`) |
| `qtimlvideoqnnbin` | `MLVideoQNNBin` (also usable as `Element`) |
| `qtimlvideosnpebin` | `MLVideoSNPEBin` (also usable as `Element`) |
| `qtimlvideotflitebin` | `MLVideoTFLiteBin` (also usable as `Element`) |
| `qtimlaic` | `Element` |
| `qtimetamux` | `Element` |
| `qtimlmetaextractor` | `Element` |
| `qtimlmetaparser` | `Element` |
| `qtimldemux` | `Element` |
| `qtimetatransform` | `Element` |
| `mlonnx` | `Element` |

**Multimedia**

| GStreamer | QIM SDK Python Class |
| - | - |
| `qtiobjtracker` | `Element` |
| `qtismartvencbin` | `Element` |
| `qtivoverlay` | `Element` |
| `qtivsplit` | `Element` |
| `qtivcomposer` | `Element` |
| `qtibatch` | `Element` |
| `qticamreproc` | `Element` |
| `qtic2adec` | `Element` |
| `qtic2aenc` | `Element` |
| `qtic2vdec` | `Element` |
| `qtic2venc` | `Element` |
| `qtijpegenc` | `Element` |
| `qtivtransform` | `Element` |
| `qticvimgpyramid` | `Element` |
| `qticvoptclflow` | `Element` |
| `qtidfs` | `Element` |
| `v4l2h264dec` | `Element` |
| `v4l2h264enc` | `Element` |
| `v4l2h265dec` | `Element` |
| `v4l2h265enc` | `Element` |
| `tee` | `Element` |
| `h264parse` | `Element` |
| `rtpptdemux` | `Element` |
| `rtph264depay` | `Element` |
| `queue` | `Element` |
| `qtdemux` | `Element` |
| `tsdemux` | `Element` |

## Common Application Design

### Create pipeline with config file

The SDK supports fully declarative pipeline construction using YAML. A YAML pipeline describes elements and links between them, which are translated at runtime into a GStreamer pipeline.

**TopLevel Structure**

A YAML pipeline file has a single top-level key:

```yaml theme={null}
pipeline:
  elements: [...]
  links: [...]
```

**Elements Section**

`elements` is an ordered list of pipeline elements. Each element entry defines:

* The GStreamer element type
* A unique name used for linking
* Optional properties
* Optional filter configuration

**Common Fields**

```yaml theme={null}
- type: <element-type>
  name: <unique-name>
  <property>: <value>
```

* `type`: GStreamer element factory name, or `filter`
* `name`: Unique identifier within the pipeline
* properties: Any remaining key/value pairs are passed directly to the element as GStreamer properties

**Filter Elements**

Filters are declared using `type: filter` and are further specialized by their configuration block.

**Stream Filter**

```yaml theme={null}
- type: capsfilter
  name: raw_caps
  caps: "video/x-raw,format=NV12,width=1280,height=720,framerate=30/1"
```

**Video Filter**

```yaml theme={null}
- type: filter
  name: vf
  video:
    format: NV12
    width: 1920
    height: 1080
    framerate: 30
```

**Links Section**

`links` defines explicit connection paths between elements.

Each entry is an ordered list of element names. Elements are linked sequentially from left to right.

```yaml theme={null}
links:
  - [src, demux, parse, decoder, vf, split]
```

**Example yaml files**

**YAML config file**

```yaml theme={null}
pipeline:
  elements:
    - type: qtiqmmfsrc
      name: source
      camera: "0"

    - type: filter
      name: filter
      video:
        format: NV12
        width: 1920
        height: 1080
        framerate: 30.0

    - type: tee
      name: split

    - type: qtimlvconverter
      name: preprocessing

    - type: qtimltflite
      name: inferencing
      delegate: gpu
      model: /etc/models/model.tflite

    - type: qtimlpostprocess
      name: postprocessing

    - type: qtimetamux
      name: mlmuxer

    - type: qtivoverlay
      name: overlay

    - type: waylandsink
      name: display
      sync: true
      async: true
      fullscreen: true

  links:
    - [source, filter, split, preprocessing, inferencing, postprocessing, mlmuxer, overlay, display]
    - [split, mlmuxer]
```

### Multi-Model Daisy Chain

Below is shown a logical diagram of an AI daisy chain pipeline with sequential and parallel inferencing.

The same could be handled by a straight sequential pipeline of inference bins. But we must set manually the source for every inference bin.

<img alt="Introduction" />

<Accordion title="Try me">
  <Note>
    Several `.add(...)` calls in this example had missing/mismatched quotes in the source PDF (e.g. a stray closing paren instead of a closing quote, or a missing leading quote). These have been corrected below where unambiguous. One inline comment truncated at the page margin was completed using the identical sentence found in the prose above.
  </Note>

  ```cpp theme={null}
  Element ml_car("qtimlvideotflitebin", "car")
    .add("performance-profile", "high-performance")
    .add("inference-delegate", "dsp")
    .add("inference-model", "/opt/car_detect.dlc")
    .add("inference-layers", "< /model.22/Mul_2, /model.22/Sigmoid >")
    .add("postprocess-threshold", 60.0)
    .add("postprocess-results", 4)
    .add("postprocess-module", "yolov8")
    .add("postprocess-labels", "/opt/car_detect_model.labels");
    // The same could be handle by straight sequential pipeline of inference bins. But we must set manually source to every inference bin.

  Element ml_plate("qtimlvideotflitebin", "plated")
    .add("source", 1)
    .add("image-disposition", "centre")
    .add("performance-profile", "high-performance")
    .add("inference-delegate", "dsp")
    .add("inference-model", "/opt/plate_detect.dlc")
    .add("inference-layers", "< /model.22/Mul_2, /model.22/Sigmoid >")
    .add("postprocess-threshold", 40.0)
    .add("postprocess-results", 1)
    .add("postprocess-module", "yolov8")
    .add("postprocess-labels", "/opt/plate_detect_model.labels");

  Element ml_anpr("qtimlvideotflitebin", "anpr")
    .add("source", 2)
    .add("image-disposition", "centre")
    .add("inference-delegate", "dsp")
    .add("inference-model", "/opt/anpr.dlc")
    .add("inference-layers", "< /decode_head2/classifier/Conv, /decode_head/classifier/Conv >")
    .add("postprocess-threshold", 15.0)
    .add("postprocess-results", 1)
    .add("postprocess-module", "anpr")
    .add("postprocess-labels", "/opt/anpr_model.labels");

  Element ml_siglip("qtimlvideotflitebin", "siglip")
    .add("source", 1)
    .add("image-disposition", "centre");
    .add("inference-delegate", "dsp");
    .add("inference-model", "/opt/car_make_model.dlc");
    .add("inference-layers", "< /classifier/Gemm >");
    .add("postprocess-threshold", 15.0);
    .add("postprocess-results", 1);
    .add("postprocess-module", "mobilenet");
    .add("postprocess-labels", "/opt/car_make_model.labels");

  Pipeline pipeline("SmartVehicleIdentification");
  pipeline.add("camsrc", "source", "camera", "0")
              .add_stream_filter("filter", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
              .add(ml_car)
              .add(ml_plate)
              .add(ml_anpr)
              .add(ml_siglip)
              .add("qtivoverlay", "overlay")
              .add("waylandsink", "display", "sync", "true", "async", "true", "fullscreen")
              .execute();
  ```
</Accordion>

### Custom Postprocessing for YOLOv8 (Detection)

QIM SDK implements AI model post-processing through dedicated modules that take tensors as input and produce predictions. All remaining complexity — such as batching, daisy-chaining, image mask support — is encapsulated within the post-processing plugin. This makes the post-processing module API an ideal integration point for application-specific post-processing logic. As a result, applications do not need to handle batching, chaining, image masks, or ML metadata themselves. We will implement a specialized post-processing module to manage bindings with the application.

<img alt="Introduction" />

Custom postprocessing callback is attached to a [`qtimlpostprocess`](/plugin-reference/qtimlpostprocess) element via [`MLPostprocess.set_handler(...)`](/plugin-reference/qtimlpostprocess).

Typical pipeline segment:

1. Preprocess frame/tensors ([`qtimlvconverter`](/plugin-reference/qtimlvconverter))
2. Run ML inference ([`qtimltflite`](/plugin-reference/qtimltflite))
3. Convert raw tensors in custom callback ([`qtimlpostprocess`](/plugin-reference/qtimlpostprocess))

**Supported output types**

* `ImageClassifications`
* `ObjectDetections`
* `Poses`
* `DepthMaps`
* `Segmentations`
* `Tensors`

**Minimal usage**

```python theme={null}
from qimsdk import MLPostprocess, ObjectDetections

def my_callback(mlframe, mlparams, out: ObjectDetections):
    ...
    return True

postprocessing = MLPostprocess("postprocessing")
postprocessing.set_handler(my_callback)

pipeline = (
    Pipeline("my-pipeline")
    .add("qtimltflite", "inferencing", ...)
    .add(postprocessing)
    .add_stream_filter("mlf", TextFilter())
    ...
)
```

**Input and output explained**

`mlframe` (read-only model tensors)

Passed to every callback. Its `tensors` list holds one entry per model output tensor, each exposing:

* `type`: tensor element type
* `dimensions`: tensor shape
* `data`: raw tensor memory (as a `memoryview`)
* `size`: tensor bytes

Common access pattern:

```python theme={null}
t = mlframe.tensors[0]
data = t.data
```

`mlparams` (runtime metadata and context)

`mlparams` carries key/value fields provided by the pipeline and preprocessing/postprocess context. Commonly used keys:

* `input-tensor-width`
* `input-tensor-height`
* `input-tensor-region` (an `(x, y, width, height)` region)

Example:

```python theme={null}
source_width = mlparams.get("input-tensor-width")
source_height = mlparams.get("input-tensor-height")
region = mlparams.get("input-tensor-region")
```

**Callback signatures**

* Callbacks annotated with `ImageClassifications`, `AudioClassifications`, `ObjectDetections`, `Poses`, `DepthMaps`, or `Segmentations` fill the corresponding output metadata container.
* A callback annotated with `Tensors` fills an output `Tensors` object for downstream tensor stages.
* `mlframe` is read-only and must not be modified.

**Classification callback**

```python theme={null}
def classification_callback(mlframe, mlparams, out: ImageClassifications):
    ...
    return True
```

Output: append classification items to `out`. Field meanings:

* `name`: class label text
* `confidence`: prediction confidence
* `color`: optional UI/overlay color
* `extra`: optional metadata map

**Object detection callback**

```python theme={null}
def detection_callback(mlframe, mlparams, out: ObjectDetections):
    ...
    return True
```

Output: append detection items to `out`. Field meanings:

* `left`, `top`, `right`, `bottom`: normalized box coordinates
* `name`, `confidence`, `color`: class info
* `landmarks`: optional keypoints
* `extra`: optional metadata map

Implemented example includes confidence thresholding, region coordinate transform, and NMS.

**Segmentation callback**

```python theme={null}
def segmentation_callback(mlframe, mlparams, segmentations: Segmentations):
    ...
    return True
```

Input parameters

* `mlframe`: read-only model output tensors.
* `mlparams`: runtime metadata (for example: `input-tensor-width`, `input-tensor-height`, `input-tensor-region`).
* `segmentations`: output container to fill.

Output contract

* `labels`: per-pixel semantic labels.
* `colors`: per-pixel colors; index-aligned with `labels`.
* `n_rows`, `n_columns`: segmentation map dimensions.
* `extra`: optional metadata fields.

**Pose estimation callback**

```python theme={null}
def pose_callback(mlframe, mlparams, out: Poses):
    ...
    return True
```

Output: append pose items to `out`. Field meanings:

* `keypoints`: list of keypoints (`name`, `x`, `y`, `confidence`, `color`)
* `links`: list of keypoint links describing skeleton connections
* `name`, `confidence`, `extra`: pose-level metadata

**Depth estimation callback**

```python theme={null}
def depth_callback(mlframe, mlparams, out: DepthMaps):
    ...
    return True
```

Output: append depth map items to `out`. Field meanings:

* `values`: per-pixel depth values
* `colors`: per-pixel pseudo-color values
* `n_rows`, `n_columns`: depth map shape
* `extra`: optional metadata map

**Tensor-to-tensor callback**

```python theme={null}
def tensor_callback(mlframe, mlparams, out: Tensors):
    ...
    return True
```

`out` is owned and writable; callback typically:

1. validates input tensors from `mlframe`
2. copies/selects required tensors into `out`
3. updates output tensor dimensions/order if needed

**Semantic callbacks** (classification, detection, pose, depth, segmentation)

```python theme={null}
def callback(mlframe, mlparams, output: OutputType) -> bool:
    ...
```

* `mlframe`: input tensors from inference (read-only)
* `mlparams`: runtime metadata (input sizes, region, etc.)
* `output`: semantic objects to be filled by callback
* return `True` on valid output
* return `False` on validation/decode failure

**Tensor callback**

```python theme={null}
def callback(mlframe, mlparams, output: Tensors) -> bool:
    ...
```

* `mlframe`: input tensors from inference (read-only)
* `mlparams`: runtime metadata/context
* `output`: writable output tensors for downstream tensor stage
* return `True` on valid output
* return `False` on validation/decode failure

**Copy-ready callback skeletons**

Classification

```python theme={null}
def classification_callback(mlframe, mlparams, out: ImageClassifications):
    # decode tensors -> classes
    return True
```

Detection

```python theme={null}
def detection_callback(mlframe, mlparams, out: ObjectDetections):
    # decode tensors -> detections
    return True
```

Pose estimation

```python theme={null}
def pose_callback(mlframe, mlparams, out: Poses):
    # decode tensors -> poses
    return True
```

Depth estimation

```python theme={null}
def depth_callback(mlframe, mlparams, out: DepthMaps):
    # decode tensors -> depthmaps
    return True
```

Segmentation

```python theme={null}
def segmentation_callback(mlframe, mlparams, out: Segmentations):
    # decode tensors -> segmentations
    return True
```

Tensors

```python theme={null}
def tensor_callback(mlframe, mlparams, out: Tensors):
    # map/copy selected tensors from mlframe -> out
    return True
```

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_detection_yolov8.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_detection_yolov8.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_detection_yolov8.py`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
  | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      Also copy an MP4 (H.264) file for the pipeline's video input, or update the pipeline's `location` property to point to your own file.

      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp video.mp4                   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_external_postprocess_detection_yolov8.py
      ```

      The application runs inference on the video file and uses the custom, application-defined postprocessing callback to decode detections, which are then overlaid and displayed fullscreen.

      To stop the application, press **CTRL + C**.
    </Step>
  </Steps>
</Accordion>


# Coding Agent
Source: https://imsdkdocs.qualcomm.com/app-builder/python-coding-agent

Use the QIM SDK Python App Builder skill to generate Python pipeline apps with an AI coding agent.

## What is the Python App Builder Skill?

The **QIM SDK Python App Builder** is an AI coding skill that generates Python applications using `QIM SDK Python API`. You describe the pipeline behavior and configuration in natural language, and the agent produces a ready-to-run `main.py` built on the `QIM SDK Python API` — along with a `README.md` that documents information about the generated application.

**Flow**

<img alt="QIM SDK coding agent workflow" />

**What it generates**

For each request, the agent produces an artifact folder named `qimsdk-python-<name>/` containing:

* **`main.py`** — a complete, runnable Python application using qimsdk python package, including pipeline construction, element wiring, stream filters, and ML inference configuration
* **`README.md`** — a detailed document covering:
  * Purpose and pipeline behavior summary
  * Configuration placeholders (input, model, labels, output paths)
  * A step-by-step pipeline flow (text summary + Mermaid diagram)
  * Steps to run the app on device
* **`A YAML config file`** — only when you request declarative YAML pipeline mode

**Supported use cases:**

| Category | Details |
| - | - |
| **AI pipelines** | Object detection, image classification, semantic segmentation, face detection, pose estimation, super resolution, depth estimation, audio classification, object tracking, gesture recognition |
| **Inputs** | ISP camera, USB camera, file source (MP4/H.264), RTSP stream |
| **Outputs** | HDMI display, encoded file (MP4), RTSP stream, Redis/MQTT metadata |
| **ML backends** | TFLite (HTP/NPU), QNN, SNPE |
| **Pipeline patterns** | Single-stream, multi-stream, daisy-chained models, ML-bin fused inference, custom pre/post-processing, AppSrc/AppSink bridges, YAML config mode |

## Prerequisites

### For generating code

<Note>
  Qualcomm dev kits are not needed for code generation.
</Note>

**Host Machine**: Any PC with internet access

**Coding Agent**: Any AI coding agent installed (Claude Code, Cursor, Codex, etc.) which supports skills — the skills provided are agent-agnostic.

<Note>
  This skill is designed to work well even with low-reasoning / smaller models.

  Eg: A Sonnet 4.5 model is sufficient.
</Note>

### For deploying and running apps

The target devices where QIM SDK is supported (see [QIM SDK Installation Guide](/installation))

## How to Use

### Step 1: Get the skill

Choose one installation method:

<Tabs>
  <Tab title="Install Skill">
    Clone the skill repository and copy the **QIM SDK Python App Builder** and **qimsdk-deploy** skills into your coding agent's skills directory:

    ```bash theme={null}
    git clone https://github.com/qualcomm/qimsdk-agentic-skills

    # The following commands are for Claude Code.
    # Set this to your coding agent's skills directory.
    cp -r qimsdk-agentic-skills/skills/qimsdk-python-app-builder ~/.claude/skills
    cp -r qimsdk-agentic-skills/skills/qimsdk-deploy ~/.claude/skills
    ```

    The skills directory is agent-specific. Check your coding agent's documentation for its skills location.

    Restart or reload your coding agent after installation. Verify that `qimsdk-python-app-builder` and `qimsdk-deploy` are available with `/skills`.
  </Tab>

  <Tab title="Install Plugin">
    **Claude Code** — run inside a Claude Code session:

    ```text theme={null}
    /plugin marketplace add https://github.com/qualcomm/qimsdk-agentic-skills.git
    /plugin install qimsdk-agentic-skills@qimsdk-marketplace
    /reload-plugins
    ```

    **Codex** — run in your terminal:

    ```bash theme={null}
    codex plugin marketplace add https://github.com/qualcomm/qimsdk-agentic-skills.git
    codex plugin add qimsdk-agentic-skills@qimsdk-marketplace
    ```
  </Tab>
</Tabs>

***

### Step 2: Generate the application

Describe your pipeline to the agent. It will automatically activate the skill and generate a complete `main.py` and `README.md`.

#### Sample Prompt

**Prompt format — describe pipeline behavior and configuration:**

```
Create a QIM SDK Python app for single-stream YOLOX object detection from an
MP4 file, encoding the result to a file.

## Pipeline Behavior
- Decode an MP4 file using Qualcomm hardware decoder
- Run YOLOX object detection on full frames using TFLite external delegate on HTP/NPU
- Merge metadata with the video stream before overlay
- Overlay bounding boxes and labels
- Encode the overlaid video and save to an output MP4 file

## Configuration
- Input:   ~/Downloads/qimsdk_samples/media/ai_demo_sample.mp4
- Model:   ~/Downloads/qimsdk_samples/models/yolox_w8a8.tflite
- Labels:  ~/Downloads/qimsdk_samples/labels/yolov8.json
- Backend: TFLite external delegate, HTP/NPU
- Confidence: 51.0
- Output:  ~/Downloads/qimsdk_samples/media/obj_detect_out.mp4
```

More sample prompts can be found in the [Available Sample Prompts](#available-sample-prompts) section below.

#### Generated `README.md`

The README documents the generated app in full:

* **Purpose** — a plain-English summary of what the pipeline does
* **Files** — lists `main.py` and `README.md` with descriptions
* **Assumptions** — codec format, quantization requirements, camera defaults, output directory pre-conditions
* **Configuration** — all user-supplied paths (`INPUT_FILE`, `MODEL_PATH`, `LABELS_PATH`, `OUTPUT_FILE`) with instructions on where to change them
* **Placeholders to Fill** — any values you still need to supply, or a note that none remain when the request was fully specified
* **Pipeline Flow** — a `Text Summary` walkthrough of every element plus a `Mermaid Diagram` of the full pipeline
* **Steps to Run** — exact commands to run on device

<Accordion title="README.md">
  # QIM SDK Python App — Single-Stream YOLOX Object Detection (MP4 → MP4)

  ## Purpose

  Decode an MP4 file with the Qualcomm hardware decoder, run **YOLOX** object
  detection on full frames using the **TFLite external delegate on the HTP/NPU**,
  merge the detection metadata with the video stream, overlay bounding boxes and
  class labels, then hardware-encode the annotated video to an output MP4 file.
  Headless (no display branch).

  ## Files

  * `main.py` — the qimsdk pipeline app
  * `README.md` — this file

  ## Assumptions

  * Input is H.264-in-MP4, decoded with `v4l2h264dec` (`capture-io-mode=4`,
    `output-io-mode=4` — file source decoded through the hardware decoder).
  * YOLOX (`yolox_w8a8.tflite`) uses the `yolov8` postprocess module with the
    `yolov8.json` labels, per the model catalog.
  * Confidence threshold is applied as inline postprocess settings
    `{"confidence": 51.0}`.
  * No display: the annotated stream is encoded and written straight to the
    output MP4 (`v4l2h264enc` → `h264parse` → `mp4mux` → `filesink`).
  * `pipeline.eos(True)` is set so `mp4mux` finalizes the container on EOS.

  ## Configuration (fixed constants in `main.py`)

  | Value | Path |
  | - | - |
  | Input | `$HOME/Downloads/qimsdk_samples/media/ai_demo_sample.mp4` |
  | Model | `$HOME/Downloads/qimsdk_samples/models/yolox_w8a8.tflite` |
  | Labels | `$HOME/Downloads/qimsdk_samples/labels/yolov8.json` |
  | Output | `$HOME/Downloads/qimsdk_samples/media/obj_detect_out.mp4` |

  `$HOME` is expanded in Python (`os.environ['HOME']`) before being passed to any
  element property, so the element receives a fully-resolved absolute path.

  ## Placeholders to Fill

  None. All paths, model, labels, confidence, and delegate options are concrete.

  ## Pipeline Flow

  ### Text Summary

  `filesrc` reads the MP4 and `qtdemux` extracts the H.264 elementary stream,
  which `h264parse` prepares and `v4l2h264dec` hardware-decodes. A `queue`
  (`q_dec`) decouples the decoder thread, then a `VideoFilter` constrains the
  stream to NV12. A `tee` (`split`) branches:

  * **Passthrough / video branch** → `qtimetamux` (`obj_mux`).
  * **AI branch** → `queue` → `qtimlvconverter` (preprocess to model tensor) →
    `queue` → `qtimltflite` (YOLOX, external delegate, HTP/NPU) → `queue` →
    `qtimlpostprocess` (`module=yolov8`, labels, `{"confidence": 51.0}`) →
    `TextFilter` → `queue` → `qtimetamux` (`obj_mux`).

  `qtimetamux` merges the detection metadata back onto the video frames,
  `qtivoverlay` draws the bounding boxes and class labels, and the annotated NV12
  stream is hardware-encoded by `v4l2h264enc`, prepared by `h264parse`, muxed by
  `mp4mux`, and written by `filesink` to the output MP4.

  ### Mermaid Diagram

  ```mermaid theme={null}
  flowchart TD
    SRC[filesrc] --> DEMUX[qtdemux]
    DEMUX --> PARSE[h264parse]
    PARSE --> DEC[v4l2h264dec]
    DEC --> QDEC[queue q_dec]
    QDEC --> NV12[NV12 VideoFilter]
    NV12 --> SPLIT[tee split]
    SPLIT -->|passthrough| MUX[qtimetamux obj_mux]
    SPLIT -->|AI| Q1[queue q1]
    Q1 --> PRE[qtimlvconverter]
    PRE --> Q2[queue q2]
    Q2 --> INFER[qtimltflite YOLOX HTP]
    INFER --> Q4[queue q4]
    Q4 --> POST[qtimlpostprocess module=yolov8]
    POST --> TEXT[TextFilter]
    TEXT --> Q5[queue q5]
    Q5 --> MUX
    MUX --> OVL[qtivoverlay]
    OVL --> ENC[v4l2h264enc]
    ENC --> PARSEOUT[h264parse parse_out]
    PARSEOUT --> MP4[mp4mux]
    MP4 --> SINK[filesink]
  ```

  ## Steps to Run

  First ensure all referenced files — the input MP4, the `yolox_w8a8.tflite`
  model, and the `yolov8.json` labels — are already present on the device at the
  paths configured in `main.py`.

  ```bash theme={null}
  # Copy the app to device
  scp main.py <user>@<device-ip>:~/

  # SSH into device and run
  ssh <user>@<device-ip>
  python3 ~/main.py
  ```

  The input MP4 must exist at
  `$HOME/Downloads/qimsdk_samples/media/ai_demo_sample.mp4` on the device, and the
  output directory (`$HOME/Downloads/qimsdk_samples/media/`) must be writable so
  the app can create `obj_detect_out.mp4`. The app runs to end-of-file, finalizes
  the MP4, and exits.
</Accordion>

#### Generated `main.py`

* The generated application constructs a tee-split pipeline: decoded frames are split into a passthrough branch and an AI branch.
* The AI branch runs `qtimlvconverter` (preprocess) → `qtimltflite` (YOLOX inference via HTP/NPU) → `qtimlpostprocess` → `TextFilter`, then merges back into `qtimetamux` with the passthrough video.
* `qtivoverlay` draws bounding boxes, and the result is hardware-encoded and written to an MP4 file via `filesink`.
* Pipeline construction and `pipeline.execute()` live inside `create_and_execute_pipeline()`; `main()` handles logging setup and calls it.
* All paths use `$HOME` expansion via `os.environ['HOME']` — element properties do not perform shell expansion.

<Accordion title="main.py">
  ```python theme={null}
  #!/usr/bin/env python3

  """Single-stream YOLOX object detection from an MP4 file, encoded to an output MP4 file."""

  import os

  from qimsdk import Element, Pipeline, TextFilter, VideoFilter

  #  Example pipeline:
  #
  #    src -> demux -> parse -> decoder -> q_dec -> [vf:NV12] -> tee name=split
  #      split. -> obj_mux -> qtivoverlay -> encoder -> parse_out -> mux -> filesink
  #      split. -> q1 -> preprocessing -> q2 -> inferencing -> q4
  #             -> postprocessing -> [mlf:text] -> q5 -> obj_mux.
  #
  #  The pipeline reads an MP4/H.264 file, decodes it through the hardware decoder,
  #  runs YOLOX (yolov8 postprocess module) object detection on full frames using
  #  the HTP/NPU external delegate, merges the detection metadata back with the
  #  video, overlays the detected bounding boxes and class labels on each frame,
  #  and hardware-encodes the annotated stream to an output MP4 file.

  INPUT_FILE = f"{os.environ['HOME']}/Downloads/qimsdk_samples/media/ai_demo_sample.mp4"
  MODEL_PATH = f"{os.environ['HOME']}/Downloads/qimsdk_samples/models/yolox_w8a8.tflite"
  LABELS_PATH = f"{os.environ['HOME']}/Downloads/qimsdk_samples/labels/yolov8.json"
  OUTPUT_FILE = f"{os.environ['HOME']}/Downloads/qimsdk_samples/media/obj_detect_out.mp4"


  def create_and_execute_pipeline() -> None:

      # Reads the input MP4 file as raw bytes.
      source = (
          Element("filesrc", "source")
          .set("location", INPUT_FILE)
      )

      # Extracts the elementary H.264 stream from the MP4 container.
      demux = Element("qtdemux", "demux")

      # Prepares the H.264 bitstream for the decoder.
      parser = Element("h264parse", "parser")

      # Decodes the compressed H.264 stream into raw video frames.
      #
      # DMA I/O modes are used to avoid unnecessary buffer copies. This is a
      # file source decoded through the hardware decoder, so both io-modes are 4.
      decoder = (
          Element("v4l2h264dec", "decoder")
          .set("capture-io-mode", 4)
          .set("output-io-mode", 4)
      )

      # Queue immediately after hardware decode to decouple the decoder thread.
      q_dec = Element("queue", "q_dec")

      # Restricts the decoded stream to NV12 before branching for AI/overlay.
      videofilter = VideoFilter().format("NV12")

      # Splits decoded frames into a video/overlay branch and an AI branch.
      split = Element("tee", "split")

      # Queues frames from the tee into the AI branch.
      q1 = Element("queue", "q1")

      # Converts raw NV12 video frames into the model input tensor format.
      preprocessing = Element("qtimlvconverter", "preprocessing")

      # Queues converted tensors before inference.
      q2 = Element("queue", "q2")

      # Runs YOLOX object detection on full frames using the TFLite external
      # delegate targeting the HTP/NPU.
      inferencing = (
          Element("qtimltflite", "inferencing")
          .set("model", MODEL_PATH)
          .set("delegate", "external")
          .set("external-delegate-path", "libQnnTFLiteDelegate.so")
          .set("external-delegate-options", "QNNExternalDelegate,backend_type=htp,log_level=(string)1;")
      )

      # Queues tensor outputs before postprocessing.
      q4 = Element("queue", "q4")

      # Decodes YOLOX model output tensors into detection metadata.
      # YOLOX detection uses the yolov8 postprocess module per the model catalog.
      # The confidence threshold (51.0) is applied via inline JSON settings.
      postprocessing = (
          Element("qtimlpostprocess", "postprocessing")
          .set("module", "yolov8")
          .set("labels", LABELS_PATH)
          .set("settings", '{"confidence": 51.0}')
      )

      # Stream filter marking the AI branch output as text metadata.
      mlf = TextFilter()

      # Queues metadata before it is merged back with the video branch.
      q5 = Element("queue", "q5")

      # Merges detection metadata produced by the AI branch with the original
      # video frames from the passthrough branch.
      obj_mux = Element("qtimetamux", "obj_mux")

      # Renders detected bounding boxes and class labels on top of the frame.
      overlay = Element("qtivoverlay", "overlay")

      # Hardware-encodes the annotated NV12 stream back to H.264.
      #
      # File source path decoded via hardware decoder: capture/output io-mode 4.
      encoder = (
          Element("v4l2h264enc", "encoder")
          .set("capture-io-mode", 4)
          .set("output-io-mode", 4)
      )

      # Prepares the encoded H.264 stream for muxing.
      parse_out = Element("h264parse", "parse_out")

      # Multiplexes the H.264 stream into an MP4 container.
      mux = Element("mp4mux", "mux")

      # Writes the final annotated video to the output MP4 file.
      sink = (
          Element("filesink", "sink")
          .set("location", OUTPUT_FILE)
      )

      # Creates the pipeline, adds and links elements, and executes it.
      pipeline = (
          Pipeline("obj-detect-file-pipeline")
          .add(source)
          .add(demux)
          .add(parser)
          .add(decoder)
          .add(q_dec)
          .add_stream_filter("videofilter", videofilter)
          .add(split)
          .add(q1)
          .add(preprocessing)
          .add(q2)
          .add(inferencing)
          .add(q4)
          .add(postprocessing)
          .add_stream_filter("mlf", mlf)
          .add(q5)
          .add(obj_mux)
          .add(overlay)
          .add(encoder)
          .add(parse_out)
          .add(mux)
          .add(sink)
          .link("source", "demux", "parser", "decoder", "q_dec", "videofilter", "split")
          .link("split", "obj_mux")
          .link("split", "q1", "preprocessing", "q2", "inferencing", "q4",
                "postprocessing", "mlf", "q5", "obj_mux")
          .link("obj_mux", "overlay", "encoder", "parse_out", "mux", "sink")
      )

      # Ensures the MP4 muxer finalizes the container correctly on EOS.
      pipeline.eos(True)

      pipeline.execute()


  def main() -> None:

      from qimsdk import ImsdkGstLogMode, ImsdkLogLevel, SetImsdkGstLogMode, SetImsdkLogLevel
      SetImsdkGstLogMode(ImsdkGstLogMode.ImsdkLog)
      SetImsdkLogLevel(ImsdkLogLevel.Debug)

      create_and_execute_pipeline()


  if __name__ == "__main__":
      main()
  ```
</Accordion>

***

### Step 3: Running the applications

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy the assets to the device">
    The input video, model, and labels must be present on the device before running — regardless of how you run the app. Copy them to the paths `main.py` expects:

    ```bash theme={null}
    # Replace <user> and <device-ip> with your device credentials.
    ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"

    # Copy the assets to device (paths must match main.py)
    scp ai_demo_sample.mp4 <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
    scp yolox_w8a8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
    scp yolov8.json <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
    ```
  </Step>

  <Step title="Run the application">
    With the assets in place, you have two options to deploy and run the agent-generated `main.py`:

    **Option A — Use the QIM SDK Deploy skill (Recommended):**

    The `qimsdk-deploy` skill pushes `main.py` and runs it directly on the target device over SSH. It copies only `main.py` — the assets from the previous step must already be on the device.

    **Option B — Follow the generated README:**

    Follow the **steps to run** in the [generated `README.md`](#generated-readme-md) to manually copy `main.py` to device and run the app:

    ```bash theme={null}
    # Copy the app to device
    scp main.py <user>@<device-ip>:~/

    # SSH into device and run
    ssh <user>@<device-ip>
    python3 ~/main.py
    ```
  </Step>
</Steps>

***

## Available Sample Prompts

| # | Prompt |
| - | - |
| 1 | [Camera YOLOv8 object detection using fused TFLite ML-bin](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-python-app-builder/AI_01_mlbin_yolov8_camera_overlay.md) |
| 2 | [Two-stage PPE daisy-chain — person detection → PPE detection → overlay display](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-python-app-builder/AI_02_mlbin_ppe_daisy_chain_display.md) |
| 3 | [Object detection with custom postprocess placeholder](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-python-app-builder/AI_03_custom_postprocess_detection_placeholder.md) |
| 4 | [Pose estimation with custom postprocess placeholder](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-python-app-builder/AI_04_custom_postprocess_pose_placeholder.md) |
| 5 | [Inference app with tensor output via custom callback placeholder](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-python-app-builder/AI_05_tensor_dump_callback_placeholder.md) |
| 6 | [YOLOv8 object detection pipeline with generated YAML config](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-python-app-builder/AI_06_yaml_yolov8_generated_config.md) |
| 7 | [Object detection with custom preprocess placeholder](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-python-app-builder/AI_07_custom_preprocess_detection_placeholder.md) |
| 8 | [Object detection with ML-bin custom preprocess placeholder](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-python-app-builder/AI_08_mlbin_custom_preprocess_detection_placeholder.md) |
| 9 | [Load existing YOLOv8 detection pipeline from external YAML config](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-python-app-builder/AI_09_yaml_yolov8_external_config.md) |
| 10 | [Single-stream YOLOX object detection from MP4, encoded to file](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-python-app-builder/AI_10_single_stream_object_detection.md) |
| 11 | [AI wall — four parallel AI inference streams composed into a 2×2 grid display](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-python-app-builder/AI_11_ai_wall.md) |
| 12 | [Four-stage gesture recognition daisy-chain pipeline](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-python-app-builder/AI_12_gesture_recognition.md) |
| 13 | [Audio classification — classify audio events from video and overlay results](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-python-app-builder/AI_13_audio_classification.md) |
| 14 | [Two pipelines bridged through application-managed buffers (AppSrc/AppSink)](https://github.com/qualcomm/qimsdk-agentic-skills/blob/main/sample-prompts/qimsdk-python-app-builder/MM_01_appsrc_appsink_bridge.md) |


# Demo Applications
Source: https://imsdkdocs.qualcomm.com/app-builder/python-demo-applications

Ready-to-run Python App Builder demo applications

The table below lists the Python App Builder demo applications built on top of the [Python App Builder](/app-builder/python-app-builder) SDK.

| # | Application | Source Code | Description |
| - | - | - | - |
| 1 | [Product Counting Application](/blogs/product-counting-application) | [qimsdk\_demo\_product\_counting.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/demo-apps/qimsdk_demo_product_counting.py) | A YOLOv8-based object detection pipeline that counts products crossing a center region of interest in real time. |
| 2 | [Smart Board Application](/blogs/smart-board-application) | [qimsdk\_demo\_smartboard.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/demo-apps/qimsdk_demo_smartboard.py) | A touchless virtual whiteboard controlled entirely by hand gestures, supporting freehand drawing, shape recognition, erasing, and dragging content via a two-stage palm detection and hand landmark pipeline. |
| 3 | [Touchless Wrist Steering Car Game](/blogs/touchless-wrist-steering-car-game) | [qimsdk\_demo\_car\_game.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/demo-apps/qimsdk_demo_car_game.py) | A two-stage palm detection and hand landmark pipeline turns wrist movement into a virtual steering wheel for a real-time, touchless split-screen driving game. |


# Quick start guide
Source: https://imsdkdocs.qualcomm.com/app-builder/python-quickstart

A Quick demonstration of Python SDK for building AI application on Qualcomm platforms

## What is the QIM SDK Python App Builder?

### Overview

The **QIM SDK Python App Builder** is a modern Python SDK for building, running, and managing multimedia and AI pipelines on Qualcomm platforms. It sits as a high-level abstraction layer on top of GStreamer, taking care of framework-level concerns such as element negotiation, pad linking, signal management, event loops, and state transitions, so developers can stay focused on application logic instead of framework boilerplate.

<img alt="Introduction" />

### Key Capabilities

* **No GStreamer expertise required** — the SDK manages all underlying GStreamer mechanics internally, with no compile-time dependency on GStreamer headers.
* **Less code, no GStreamer boilerplate** — a fluent, high-level API bundles away the pipeline construction, element linking, and state management that GStreamer's native API would otherwise require, so a functional camera-to-display pipeline can be written in just a few lines of Python.
* **Custom pre/post-processing** — plug in your own pre- and post-processing logic around the built-in ML bins without writing a custom GStreamer element.
* **AI inference support** — built-in ML bins for TFLite, QNN, SNPE, and ONNX models.
* **Broad multimedia support** — video capture, signal processing, ML inference, overlay rendering, and display output.

### API Reference

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>API</th>
      <th>Description</th>
      <th>API Code Reference</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        **Pipeline**
      </td>

      <td>
        Builds, links, and manages the full lifecycle of a GStreamer pipeline. Supports both programmatic (Python fluent API) and declarative (YAML) pipeline definition. Key methods: `add()`, `link()`, `add_stream_filter()`, `execute()`, `prepare()`, `start()`, `stop()`, `get()`.
      </td>

      <td>
        ```python theme={null}
        p = Pipeline("cam-pipeline")
        p.add("qtiqmmfsrc", "src").add("waylandsink", "sink").execute()
        ```
      </td>
    </tr>

    <tr>
      <td>
        **Element**
      </td>

      <td>
        Generic wrapper around a GStreamer element. Supports fluent property setting at create time or at runtime to update a live pipeline.
      </td>

      <td>
        ```python theme={null}
        sink = Element("waylandsink", "display")
        sink.set("fullscreen", True)
        sink.set("sync", False)
        ```
      </td>
    </tr>

    <tr>
      <td>
        **Buffer**
      </td>

      <td>
        Unified buffer abstraction over native GStreamer memory. Provides read/write access, timestamp metadata, and zero-copy transfer paths. Key methods: `data()`, `size()`, `resize()`, `pts()`, `dts()`, `duration()`.
      </td>

      <td>
        ```python theme={null}
        def consume_buffer(buf):
            data = buf.data()
            pts = buf.pts()

        sink.set_buffer_consumer(consume_buffer)
        ```
      </td>
    </tr>

    <tr>
      <td>
        **AppSrc**
      </td>

      <td>
        App-facing source. Feeds buffers from application code into the pipeline using lambda callbacks or a push API.
      </td>

      <td>
        ```python theme={null}
        def produce_buffer(buf):
            # fill buf.data()
            return True

        src = AppSrc(pipeline.get("src"))
        src.set_buffer_producer(produce_buffer)
        ```
      </td>
    </tr>

    <tr>
      <td>
        **AppSink**
      </td>

      <td>
        App-facing sink. Delivers pipeline output buffers to application code with zero-copy.
      </td>

      <td>
        ```python theme={null}
        def consume_buffer(buf):
            # consume buf.data()
            pass

        sink = AppSink(pipeline.get("sink"))
        sink.set_buffer_consumer(consume_buffer)
        ```
      </td>
    </tr>

    <tr>
      <td>
        **CamSrc**
      </td>

      <td>
        Built-in ISP camera source. Exposes an API for runtime image capture and snapshots.
      </td>

      <td>
        ```python theme={null}
        cam = CamSrc(pipeline.get("source"))
        cam.capture_image("snapshot.jpg")
        ```
      </td>
    </tr>

    <tr>
      <td>
        **MLPreProcess**
      </td>

      <td>
        Attaches a custom Python callback for pre-processing input frames before ML inference.
      </td>

      <td>
        ```python theme={null}
        def preprocess(buf):
            # transform input frame
            pass

        pre = MLPreProcess(pipeline.get("preproc"))
        pre.set_handler(preprocess)
        ```
      </td>
    </tr>

    <tr>
      <td>
        **MLPostProcess**
      </td>

      <td>
        Attaches a custom Python callback for post-processing raw inference tensor output.
      </td>

      <td>
        ```python theme={null}
        def postprocess(buf):
            # parse inference output
            pass

        pp = MLPostProcess(pipeline.get("postproc"))
        pp.set_handler(postprocess)
        ```
      </td>
    </tr>

    <tr>
      <td>
        **MLVideoTFLiteBin** / **MLVideoQNNBin** / **MLVideoSNPEBin** / **MLVideoONNXBin**
      </td>

      <td>
        All-in-one preprocess + inference + postprocess bins. Supports TFLite, QNN, SNPE, and ONNX models.
      </td>

      <td>
        ```python theme={null}
        pipeline.add("MLVideoTFLiteBin", "ml", "model", "/models/model.tflite", "delegate", "htp")
        ```
      </td>
    </tr>

    <tr>
      <td>
        **VideoFilter** / **AudioFilter** / **ImageFilter** / **H264Filter** / **TensorFilter** / **TextFilter**
      </td>

      <td>
        Fluent caps builders for `add_stream_filter()`. Describes stream characteristics for caps negotiation between pipeline stages.
      </td>

      <td>
        ```python theme={null}
        pipeline.add_stream_filter("vf", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
        ```
      </td>
    </tr>
  </tbody>
</table>

***

## Build your application

<Note>
  Ensure you have QIM SDK installed. See [QIM SDK Installation Guide](../installation)
</Note>

Before running any of the applications below, create the sample directory tree on the target device:

```bash theme={null}
# ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{media,models,labels}"
```

This is required even for apps that don't download any media, model, or label files (such as the camera capture and recording apps), since they still write their output under this path.

### USB Camera Source

USB (UVC) cameras are handled by the `v4l2src` node. Each USB camera comes with its own format and frame constraints. Because of this, we always place `qtivtransform` right after the USB camera to ensure that the rest of the pipeline receives a hardware-friendly NV12 (Semi-planar YUV420) video format. `qtivtransform` automatically operates in passthrough mode if the USB camera already supports NV12, so you don't need to worry about any performance overhead.

<img alt="Introduction" />

<Tabs>
  <Tab title="Explicit coding style">
    <Info>
      Check application source code on GitHub: [`qimsdk_ref_usb_camera`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_usb_camera.py)

      Pre-built application on device: `/usr/bin/qimsdk_ref_usb_camera.py`
    </Info>

    ```python theme={null}
    """USB camera preview using explicit linking."""

    import argparse

    from qimsdk import Element, Pipeline, VideoFilter

    #  Example pipeline:
    #
    #    source -> transform -> [videofilter] -> display
    #
    #  The pipeline reads frames from a USB (V4L2) camera, rotates them, restricts
    #  the stream to NV12/1080p/30fps, and displays the result through Wayland.


    def create_and_execute_pipeline(device: str = "/dev/video2") -> None:

        # Captures frames from the USB camera source.
        source = (
            Element("v4l2src", "source")
            .set("device", device)
        )

        # Applies geometric transforms to video frames.
        transform = Element("qtivtransform", "transform")

        # Stream filters used in branch links.
        # They define specific stream characteristics from the supported options.
        videofilter = VideoFilter().format("NV12").resolution(1920, 1080).framerate(30)

        # Render video stream on display.
        #
        # sync=false disables strict rendering synchronization to the pipeline clock.
        # fullscreen=true renders the output fullscreen on the target display.
        display = (
            Element("waylandsink", "display")
            .set("sync", False)
            .set("fullscreen", True)
        )

        # Creates the pipeline, adds elements, links them explicitly, and executes it.
        pipeline = (
            Pipeline("usb-cam-pipeline")
            .add(source)
            .add(transform)
            .add_stream_filter("videofilter", videofilter)
            .add(display)
            .link("source", "transform", "videofilter", "display")
        )

        pipeline.execute()


    def main() -> None:

        from qimsdk import ImsdkGstLogMode, ImsdkLogLevel, SetImsdkGstLogMode, SetImsdkLogLevel
        SetImsdkGstLogMode(ImsdkGstLogMode.ImsdkLog)
        SetImsdkLogLevel(ImsdkLogLevel.Debug)

        parser = argparse.ArgumentParser(description=__doc__)
        parser.add_argument(
            "-d", "--device", default="/dev/video2",
            help="V4L2 device node to capture from (default: %(default)s)",
        )
        args = parser.parse_args()

        create_and_execute_pipeline(args.device)


    if __name__ == "__main__":
        main()
    ```

    <Steps>
      <Step title="Connect a USB camera">
        Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the pipeline's `device` property if your camera is exposed on a different node).
      </Step>

      <Step title="Run the application">
        ```bash theme={null}
        python3 /usr/bin/qimsdk_ref_usb_camera.py --input-config /dev/video2
        ```
      </Step>

      <Step title="Expected Output">
        The live camera feed is rotated and rendered fullscreen on the display in NV12 format at 1080p/30fps.

        <img alt="Expected Output" />

        To stop the application, press **CTRL + C**.
      </Step>

      <Step title="Customize application">
        * **Source code:** [`qimsdk_ref_usb_camera`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_usb_camera.py)

        Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Implicit coding style">
    <Info>
      Check application source code on GitHub: [`qimsdk_ref_usb_camera_impl`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_usb_camera_impl.py)

      Pre-built application on device: `/usr/bin/qimsdk_ref_usb_camera_impl.py`
    </Info>

    ```python theme={null}
    """USB camera preview using auto-linking."""

    import argparse

    from qimsdk import Pipeline, VideoFilter

    #  Example pipeline:
    #
    #    source -> transform -> [videofilter] -> display
    #
    #  The pipeline reads frames from a USB (V4L2) camera, rotates them, restricts
    #  the stream to NV12/1080p/30fps, and displays the result through Wayland.


    def create_and_execute_pipeline(device: str = "/dev/video2") -> None:

        # Creates the pipeline, adds and links elements, and executes it.
        #
        # Elements are created on the fly as they are added, and linking is
        # implicit, following the order in which elements are added.
        #
        # sync=false disables strict rendering synchronization to the pipeline clock.
        # fullscreen=true renders the output fullscreen on the target display.
        pipeline = (
            Pipeline("usb-cam-pipeline")
            # Captures frames from the USB camera source.
            .add("v4l2src", "source", "device", device)
            # Applies geometric transforms to video frames.
            .add("qtivtransform", "transform")
            # Stream filter used in the branch link.
            # It defines specific stream characteristics from the supported options.
            .add_stream_filter(
                "videofilter",
                VideoFilter().format("NV12").resolution(1920, 1080).framerate(30),
            )
            # Render video stream on display.
            .add("waylandsink", "display", "sync", False, "fullscreen", True)
        )

        pipeline.execute()


    def main() -> None:

        from qimsdk import ImsdkGstLogMode, ImsdkLogLevel, SetImsdkGstLogMode, SetImsdkLogLevel
        SetImsdkGstLogMode(ImsdkGstLogMode.ImsdkLog)
        SetImsdkLogLevel(ImsdkLogLevel.Debug)

        parser = argparse.ArgumentParser(description=__doc__)
        parser.add_argument(
            "-d", "--device", default="/dev/video2",
            help="V4L2 device node to capture from (default: %(default)s)",
        )
        args = parser.parse_args()

        create_and_execute_pipeline(args.device)


    if __name__ == "__main__":
        main()
    ```

    <Steps>
      <Step title="Connect a USB camera">
        Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the pipeline's `device` property if your camera is exposed on a different node).
      </Step>

      <Step title="Run the application">
        ```bash theme={null}
        python3 /usr/bin/qimsdk_ref_usb_camera_impl.py --input-config /dev/video2
        ```
      </Step>

      <Step title="Expected Output">
        The live camera feed is rotated and rendered fullscreen on the display in NV12 format at 1080p/30fps.

        <img alt="Expected Output" />

        To stop the application, press **CTRL + C**.
      </Step>

      <Step title="Customize application">
        * **Source code:** [`qimsdk_ref_usb_camera_impl`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_usb_camera_impl.py)

        Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
      </Step>
    </Steps>
  </Tab>

  <Tab title="YAML config style">
    <Info>
      See the full YAML configuration here: [qimsdk\_ref\_usb\_camera.yaml](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_usb_camera.yaml)

      Check application source code on GitHub: [qimsdk\_ref\_yml.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yml.py)

      Pre-built application on device: `/usr/bin/qimsdk_ref_yml.py`

      YAML configuration saved on device: `/etc/qimsdk/qimsdk_ref_usb_camera.yaml`
    </Info>

    ```yaml theme={null}
    pipeline:
      elements:
        - type: v4l2src
          name: source
          device: /dev/video2

        - type: qtivtransform
          name: transform

        - type: filter
          name: videostream
          video:
            format: NV12
            width: 1920
            height: 1080
            framerate: 30

        - type: waylandsink
          name: display
          sync: false
          fullscreen: true
    ```

    <Steps>
      <Step title="Connect a USB camera">
        Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the pipeline's `device` property if your camera is exposed on a different node).
      </Step>

      <Step title="Run the application">
        ```bash theme={null}
        qimsdk_ref_yml.py /etc/qimsdk/qimsdk_ref_usb_camera.yaml
        ```
      </Step>

      <Step title="Expected Output">
        The live camera feed is rotated and rendered fullscreen on the display in NV12 format at 1080p/30fps.

        <img alt="Expected Output" />

        To stop the application, press **CTRL + C**.
      </Step>

      <Step title="Customize application">
        * **Source code:** [qimsdk\_ref\_yml.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yml.py)

        Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
      </Step>
    </Steps>
  </Tab>
</Tabs>

### Object Detection

This pipeline captures from a built-in camera, runs an object detection model through an ML inference bin, renders bounding box overlays on the output frames, and displays the result. The ML bin handles preprocessing, inference, and postprocessing internally. The stream filter pins the input video format expected by the model. The overlay element draws detection results on each frame before display.

<img alt="Introduction" />

<Tabs>
  <Tab title="Explicit coding style">
    <Info>
      Check application source code on GitHub: [`qimsdk_ref_camera_yolov8`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_camera_yolov8.py)

      Pre-built application on device: `/usr/bin/qimsdk_ref_camera_yolov8.py`
    </Info>
  </Tab>
</Tabs>

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOv8 detection model | [Qualcomm AI Hub — YOLOv8](https://aihub.qualcomm.com/iot/models/yolov8_det) | `yolov8_det_quantized.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |

    <Note>
      If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

      Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
    </Note>
  </Step>

  <Step title="Copy the assets to the device">
    The model and labels must be present on the device before running. Copy them to the paths the application expects:

    ```bash theme={null}
    # Replace <user> and <device-ip> with your device credentials.
    # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

    ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"

    # Copy the assets to device
    scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
    scp yolov8.json <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
    ```
  </Step>

  <Step title="Connect an IMX camera">
    This application uses the built-in ISP camera. On platforms without an onboard ISP camera (e.g. IQ9), attach an external IMX camera (e.g. from RB3 Gen2) before running the application — see [ISP Camera (Config #2 / qticamsrc)](/advanced/debugging#isp-camera-config-2--qticamsrc) for the procedure to switch from `libcamera` to `qticamsrc`.
  </Step>

  <Step title="Run the application">
    ```bash theme={null}
    python3 /usr/bin/qimsdk_ref_camera_yolov8.py
    ```
  </Step>

  <Step title="Expected Output">
    The live camera feed is rendered fullscreen on the display with YOLOv8 bounding boxes and class labels overlaid on each detected object in real time.

    <img alt="Expected Output" />

    To stop the application, press **CTRL + C**.
  </Step>

  <Step title="Customize application">
    * **Source code:** [`qimsdk_ref_camera_yolov8`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_camera_yolov8.py)

    Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
  </Step>
</Steps>

### Custom Post-Processing

The QIM SDK implements AI model post-processing through dedicated modules that take tensors as input and produce predictions. All underlying complexity — batching, daisy-chaining, image mask support, and ML metadata handling — is encapsulated within the post-processing plugin. This makes `MLPostProcess` the ideal integration point for application-specific logic, since the application does not need to handle any of that complexity directly. A specialized post-processing module manages the bindings between the pipeline and the application.

<img alt="Introduction" />

<Tabs>
  <Tab title="Explicit coding style">
    <Info>
      Check application source code on GitHub: [`qimsdk_ref_external_postprocess_mlbin_yolov8.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_mlbin_yolov8.py)

      Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_mlbin_yolov8.py`
    </Info>
  </Tab>
</Tabs>

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOv8 detection model | [Qualcomm AI Hub — YOLOv8](https://aihub.qualcomm.com/iot/models/yolov8_det) | `yolov8_det_quantized.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

      Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
    </Note>
  </Step>

  <Step title="Copy the assets to the device">
    The input video, model, and labels must be present on the device before running. Copy them to the paths the application expects:

    ```bash theme={null}
    # Replace <user> and <device-ip> with your device credentials.
    # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

    ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"

    # Copy the assets to device
    scp ai_demo_sample.mp4 <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
    scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
    scp yolov8.json <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
    ```
  </Step>

  <Step title="Run the application">
    ```bash theme={null}
    python3 /usr/bin/qimsdk_ref_external_postprocess_mlbin_yolov8.py
    ```
  </Step>

  <Step title="Expected Output">
    The video plays back with YOLOv8 bounding boxes and class labels decoded by the custom postprocessing callback, overlaid and rendered fullscreen on the display.

    <img alt="Expected Output" />

    To stop the application, press **CTRL + C**.
  </Step>

  <Step title="Customize application">
    * **Source code:** [`qimsdk_ref_external_postprocess_mlbin_yolov8.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_mlbin_yolov8.py)

    Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
  </Step>
</Steps>

***

## Benefits of QIM SDK Python App Builder

* Significantly reduced source code
* Improved readability
* Internal GStreamer state transition handling
* Hidden End-of-Stream handling
* Automatic error and state event handling
* Automatic resource cleanup
* Simplified AppSink/AppSrc interaction using lambda functions
* SDK-managed buffer mapping and queries
* Simplified buffer access anywhere in the pipeline using wrapped GST probes
* YAML-based pipeline description


# Reference Applications
Source: https://imsdkdocs.qualcomm.com/app-builder/python-reference-applications

Reference Python App Builder pipeline examples

## Multimedia Pipeline

Before running any of the applications below, create the sample directory tree on the target device:

```bash theme={null}
# ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{media,models,labels}"
```

This is required even for apps that don't download any media, model, or label files (such as the camera capture and recording apps), since they still write their output under this path.

### USB Camera Preview

USB (UVC) cameras are handled by the `v4l2src` node. Each USB camera comes with its own format and frame constraints. Because of this, we always place `qtivcomposer` right after the USB camera to ensure that the rest of the pipeline receives a hardware-friendly NV12 (Semi-planar YUV420) video format. However, `qtivcomposer` will automatically operate in passthrough mode if the USB camera already supports NV12, so you don't need to worry about any performance overhead.

<Note>
  Some USB webcams (e.g. Logitech Brio 4K) may render a green/garbled frame at 30fps depending on platform/build. If this happens, lower the pipeline's framerate (e.g. to 15fps) as a workaround.
</Note>

<img alt="Introduction" />

<Accordion title="Try me">
  <Tabs>
    <Tab title="Explicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_usb_camera.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_usb_camera.py)

        Pre-built application on device: `/usr/bin/qimsdk_ref_usb_camera.py`
      </Info>

      ```python theme={null}
      """USB camera preview using explicit linking."""

      import argparse

      from qimsdk import Element, Pipeline, VideoFilter

      #  Example pipeline:
      #
      #    source -> transform -> [videofilter] -> display
      #
      #  The pipeline reads frames from a USB (V4L2) camera, rotates them, restricts
      #  the stream to NV12/1080p/30fps, and displays the result through Wayland.


      def create_and_execute_pipeline(device: str) -> None:

          # Captures frames from the USB camera source.
          source = (
              Element("v4l2src", "source")
              .set("device", device)
          )

          # Applies geometric transforms to video frames.
          transform = Element("qtivtransform", "transform")

          # Stream filters used in branch links.
          # They define specific stream characteristics from the supported options.
          videofilter = VideoFilter().format("NV12").resolution(1920, 1080).framerate(30)

          # Render video stream on display.
          #
          # sync=false disables strict rendering synchronization to the pipeline clock.
          # fullscreen=true renders the output fullscreen on the target display.
          display = (
              Element("waylandsink", "display")
              .set("sync", False)
              .set("fullscreen", True)
          )

          # Creates the pipeline, adds elements, links them explicitly, and executes it.
          pipeline = (
              Pipeline("usb-cam-pipeline")
              .add(source)
              .add(transform)
              .add_stream_filter("videofilter", videofilter)
              .add(display)
              .link("source", "transform", "videofilter", "display")
          )

          pipeline.execute()


      def main() -> None:

          from qimsdk import ImsdkGstLogMode, ImsdkLogLevel, SetImsdkGstLogMode, SetImsdkLogLevel
          SetImsdkGstLogMode(ImsdkGstLogMode.ImsdkLog)
          SetImsdkLogLevel(ImsdkLogLevel.Debug)

          parser = argparse.ArgumentParser(description=__doc__)
          parser.add_argument(
              "-d", "--device", default="/dev/video2",
              help="V4L2 device node to capture from (default: %(default)s)",
          )
          args = parser.parse_args()

          create_and_execute_pipeline(args.device)


      if __name__ == "__main__":
          main()
      ```

      <Steps>
        <Step title="Connect a USB camera">
          Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the pipeline's `device` property if your camera is exposed on a different node).
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          /usr/bin/qimsdk_ref_usb_camera.py --input-config /dev/video2
          ```
        </Step>

        <Step title="Expected Output">
          The live USB camera feed is rendered fullscreen on the display, rotated and scaled to 1080p/30fps NV12 as configured by the pipeline's stream filter.

          <img alt="Expected Output" />

          To stop the application, press **CTRL + C**.
        </Step>

        <Step title="Customize application">
          * **Source code:** [`qimsdk_ref_usb_camera.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_usb_camera.py)

          Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
        </Step>
      </Steps>
    </Tab>

    <Tab title="Implicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_usb_camera_impl.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_usb_camera_impl.py)

        Pre-built application on device: `/usr/bin/qimsdk_ref_usb_camera_impl.py`
      </Info>

      ```python theme={null}
      """USB camera preview using auto-linking."""

      import argparse

      from qimsdk import Pipeline, VideoFilter

      #  Example pipeline:
      #
      #    source -> transform -> [videofilter] -> display
      #
      #  The pipeline reads frames from a USB (V4L2) camera, rotates them, restricts
      #  the stream to NV12/1080p/30fps, and displays the result through Wayland.


      def create_and_execute_pipeline(device: str) -> None:

          # Creates the pipeline, adds and links elements, and executes it.
          #
          # Elements are created on the fly as they are added, and linking is
          # implicit, following the order in which elements are added.
          #
          # sync=false disables strict rendering synchronization to the pipeline clock.
          # fullscreen=true renders the output fullscreen on the target display.
          pipeline = (
              Pipeline("usb-cam-pipeline")
              # Captures frames from the USB camera source.
              .add("v4l2src", "source", "device", device)
              # Applies geometric transforms to video frames.
              .add("qtivtransform", "transform")
              # Stream filter used in the branch link.
              # It defines specific stream characteristics from the supported options.
              .add_stream_filter(
                  "videofilter",
                  VideoFilter().format("NV12").resolution(1920, 1080).framerate(30),
              )
              # Render video stream on display.
              .add("waylandsink", "display", "sync", False, "fullscreen", True)
          )

          pipeline.execute()


      def main() -> None:

          from qimsdk import ImsdkGstLogMode, ImsdkLogLevel, SetImsdkGstLogMode, SetImsdkLogLevel
          SetImsdkGstLogMode(ImsdkGstLogMode.ImsdkLog)
          SetImsdkLogLevel(ImsdkLogLevel.Debug)

          parser = argparse.ArgumentParser(description=__doc__)
          parser.add_argument(
              "-d", "--device", default="/dev/video2",
              help="V4L2 device node to capture from (default: %(default)s)",
          )
          args = parser.parse_args()

          create_and_execute_pipeline(args.device)


      if __name__ == "__main__":
          main()
      ```

      <Steps>
        <Step title="Connect a USB camera">
          Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the pipeline's `device` property if your camera is exposed on a different node).
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          /usr/bin/qimsdk_ref_usb_camera_impl.py --input-config /dev/video2
          ```
        </Step>

        <Step title="Expected Output">
          The live USB camera feed is rendered fullscreen on the display, rotated and scaled to 1080p/30fps NV12 as configured by the pipeline's stream filter.

          <img alt="Expected Output" />

          To stop the application, press **CTRL + C**.
        </Step>

        <Step title="Customize application">
          * **Source code:** [`qimsdk_ref_usb_camera_impl.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_usb_camera_impl.py)

          Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
        </Step>
      </Steps>
    </Tab>

    <Tab title="YAML config style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_usb_camera.yaml`](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_usb_camera.yaml)

        Pre-built application on device: `qimsdk_ref_yml.py`
      </Info>

      ```yaml theme={null}
      pipeline:
        elements:
          - type: v4l2src
            name: source
            device: /dev/video2

          - type: qtivtransform
            name: transform

          - type: filter
            name: videostream
            video:
              format: NV12
              width: 1920
              height: 1080
              framerate: 30

          - type: waylandsink
            name: display
            sync: false
            fullscreen: true
      ```

      <Steps>
        <Step title="Connect a USB camera">
          Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the pipeline's `device` property if your camera is exposed on a different node).
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          qimsdk_ref_yml.py /etc/qimsdk/qimsdk_ref_usb_camera.yaml
          ```
        </Step>

        <Step title="Expected Output">
          The live USB camera feed is rendered fullscreen on the display, rotated and scaled to 1080p/30fps NV12 as configured by the YAML pipeline description.

          <img alt="Expected Output" />

          To stop the application, press **CTRL + C**.
        </Step>

        <Step title="Customize application">
          * **Source code:** [`qimsdk_ref_usb_camera.yaml`](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_usb_camera.yaml)

          Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
        </Step>
      </Steps>
    </Tab>
  </Tabs>
</Accordion>

### Built-in Camera Preview

This pipeline includes a built-in camera node, a stream filter, and a display node. If the device has more than one built-in camera, you can specify which one to use by providing the camera's unique ID. The stream filter defines the camera's output format, resolution, and framerate. You can also configure the display settings, such as enabling fullscreen rendering.

<Note>
  IQ9 has no onboard ISP camera. You can attach an external IMX camera (e.g. from RB3 Gen2) instead — see [ISP Camera (Config #2 / qticamsrc)](/advanced/debugging#isp-camera-config-2--qticamsrc) for the procedure to switch from `libcamera` to `qticamsrc`.
</Note>

<img alt="Introduction" />

<Accordion title="Try me">
  <Tabs>
    <Tab title="Explicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_camera.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_camera.py)

        Pre-built application on device: `/usr/bin/qimsdk_ref_camera.py`
      </Info>

      ```python theme={null}
      #!/usr/bin/env python3

      """Basic camera preview using explicit linking."""

      from qimsdk import Element, Pipeline, VideoFilter

      #  Example pipeline:
      #
      #    source -> [videofilter] -> display
      #
      #  The pipeline reads camera frames and displays the result through Wayland.


      def create_and_execute_pipeline() -> None:

          # Captures frames from the camera source.
          source = Element("qtiqmmfsrc", "source")

          # Stream filters used in branch links.
          # They define specific stream characteristics from the supported options.
          videofilter = VideoFilter().format("NV12").resolution(1920, 1080).framerate(30)

          # Render video stream on display.
          #
          # async=false enforce state transition to ensure the buffers are returned on time.
          # sync=false disables strict rendering synchronization to the pipeline clock.
          # fullscreen=true renders the output fullscreen on the target display.
          display = (
              Element("waylandsink", "display")
              .set("sync", False)
              .set("fullscreen", True)
          )

          # Creates the pipeline, adds elements, links them explicitly, and executes it.
          pipeline = (
              Pipeline("cam-pipeline")
              .add(source)
              .add_stream_filter("videofilter", videofilter)
              .add(display)
              .link("source", "videofilter", "display")
          )

          pipeline.execute()


      def main() -> None:

          from qimsdk import ImsdkGstLogMode, ImsdkLogLevel, SetImsdkGstLogMode, SetImsdkLogLevel
          SetImsdkGstLogMode(ImsdkGstLogMode.ImsdkLog)
          SetImsdkLogLevel(ImsdkLogLevel.Debug)

          create_and_execute_pipeline()


      if __name__ == "__main__":
          main()
      ```
    </Tab>

    <Tab title="Implicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_camera_impl.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_camera_impl.py)

        Pre-built application on device: `/usr/bin/qimsdk_ref_camera_impl.py`
      </Info>

      ```python theme={null}
      #!/usr/bin/env python3

      """Basic camera preview using auto-linking."""

      from qimsdk import Pipeline, VideoFilter

      #  Example pipeline:
      #
      #    source -> [videofilter] -> display
      #
      #  The pipeline reads camera frames and displays the result through Wayland.


      def create_and_execute_pipeline() -> None:

          # Creates the pipeline, adds and links elements, and executes it.
          #
          # Elements are created on the fly as they are added, and linking is
          # implicit, following the order in which elements are added.
          #
          # async=false enforce state transition to ensure the buffers are returned on time.
          # sync=false disables strict rendering synchronization to the pipeline clock.
          # fullscreen=true renders the output fullscreen on the target display.
          pipeline = (
              Pipeline("cam-pipeline")
              # Captures frames from the camera source.
              .add("qtiqmmfsrc", "source")
              # Stream filter used in the branch link.
              # It defines specific stream characteristics from the supported options.
              .add_stream_filter(
                  "videofilter",
                  VideoFilter().format("NV12").resolution(1920, 1080).framerate(30),
              )
              # Render video stream on display.
              .add("waylandsink", "display", "sync", False, "fullscreen", True)
          )

          pipeline.execute()


      def main() -> None:

          from qimsdk import ImsdkGstLogMode, ImsdkLogLevel, SetImsdkGstLogMode, SetImsdkLogLevel
          SetImsdkGstLogMode(ImsdkGstLogMode.ImsdkLog)
          SetImsdkLogLevel(ImsdkLogLevel.Debug)

          create_and_execute_pipeline()


      if __name__ == "__main__":
          main()
      ```
    </Tab>
  </Tabs>

  <Steps>
    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_camera.py
      ```
    </Step>

    <Step title="Expected Output">
      The live camera feed is rendered fullscreen on the display at 1080p/30fps NV12 as configured by the pipeline's stream filter.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_camera.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_camera.py), [`qimsdk_ref_camera_impl.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_camera_impl.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Built-in Camera Preview with Still Capture

<Note>
  The application captures a still image a few seconds after starting and then stops automatically — this is expected behavior, not a hang.
</Note>

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_camera_and_capture.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_camera_and_capture.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_camera_and_capture.py`
  </Info>

  ```python theme={null}
  #!/usr/bin/env python3

  """Camera preview + still image capture using CamSrc and explicit links."""

  import time
  import os

  from qimsdk import CamSrc, Element, ImageFilter, Pipeline, VideoFilter

  #  Example pipeline:
  #
  #    source -> [vf] -> display
  #    source -> [if] -> imagesink
  #
  #  The pipeline reads camera frames, displays them through Wayland, and on
  #  request captures a still image to a JPEG file.


  def create_and_execute_pipeline() -> None:

      # Captures frames from the camera source.
      source = CamSrc("source")

      # Restricts the preview stream to NV12/1080p/30fps before display.
      vf = VideoFilter().format("NV12").resolution(1920, 1080).framerate(30)

      # Render video stream on display.
      display = (
          Element("waylandsink", "display")
          .set("sync", False)
          .set("async", False)
          .set("fullscreen", True)
      )

      # Restricts the still capture stream to a 4K JPEG format.
      imagefilter = ImageFilter().format("JPEG").resolution(3840, 2160)

      # Writes output buffers to files.
      imagesink = (
          Element("multifilesink", "imagesink")
          .set("enable-last-sample", False)
          .set("location", f"{os.environ['HOME']}/Downloads/qimsdk_samples/media/image_%d.jpeg")
      )

      # Creates the pipeline, adds and links elements, and executes it.
      #
      # Explicit linking is applied.
      pipeline = (
          Pipeline("cam-capture-pipeline")
          .add(source)
          .add_stream_filter("vf", vf)
          .add(display)
          .add_stream_filter("if", imagefilter)
          .add(imagesink)
          .link("source", "vf", "display")
          .link("source", "if", "imagesink")
      )

      pipeline.start()
      time.sleep(2)

      # image capture trigger
      cam = pipeline.get("source", CamSrc)
      cam.image_capture()

      time.sleep(2)
      pipeline.stop().wait()


  def main() -> None:

      if "HOME" not in os.environ:
          raise EnvironmentError("Error: HOME environment variable is not set.")

      from qimsdk import ImsdkGstLogMode, ImsdkLogLevel, SetImsdkGstLogMode, SetImsdkLogLevel
      SetImsdkGstLogMode(ImsdkGstLogMode.ImsdkLog)
      SetImsdkLogLevel(ImsdkLogLevel.Debug)

      create_and_execute_pipeline()


  if __name__ == "__main__":
      main()
  ```

  <Steps>
    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_camera_and_capture.py
      ```

      The application displays a live preview and captures a still JPEG image a few seconds after starting, saving it under `~/media/image_%d.jpeg`.
    </Step>

    <Step title="Expected Output">
      The live camera preview is rendered briefly on the display, and a still JPEG image is saved to `~/Downloads/qimsdk_samples/media/image_%d.jpeg`. The application then exits automatically.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_camera_and_capture.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_camera_and_capture.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Camera Recording to MP4

This pipeline captures frames from the built-in camera, encodes them into an H.264 stream using the hardware encoder, parses the bitstream, muxes it into an MP4 container, and writes the resulting file to disk.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_camera_encoder.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_camera_encoder.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_camera_encoder.py`
  </Info>

  ```python theme={null}
  #!/usr/bin/env python3

  """Camera preview + still image capture using CamSrc and explicit links."""

  import time
  import os

  from qimsdk import CamSrc, Element, Pipeline, VideoFilter

  #  Example pipeline:
  #
  #    source -> [vf] -> encoder -> parser -> muxer -> sink
  #
  #  The pipeline reads camera frames, encodes them with H.264, muxes into MP4,
  #  and writes the output to disk.


  def create_and_execute_pipeline() -> None:

      # Captures frames from the camera source.
      source = CamSrc("source")

      # Stream filters used in branch links.
      # They define specific stream characteristics from the supported options.
      vf = VideoFilter().format("NV12").resolution(1920, 1080).framerate(30)

      # Encodes raw video frames into H.264 stream.
      encoder = (
          Element("v4l2h264enc", "encoder")
          .set("output-io-mode", "dmabuf-import")
          .set("capture-io-mode", "dmabuf")
      )

      # Parses H.264 bitstream for downstream muxing.
      parser = Element("h264parse", "parser")

      # Muxes encoded stream into MP4 container.
      muxer = Element("mp4mux", "muxer")

      # Writes output stream to a file.
      sink = (
          Element("filesink", "sink")
          .set("location", f"{os.environ['HOME']}/Downloads/qimsdk_samples/media/encoder_output.mp4")
      )

      # Creates the pipeline, adds and links elements, and executes it.
      #
      # Explicit linking is applied.
      pipeline = (
          Pipeline("cam-encoder-pipeline")
          .add(source)
          .add_stream_filter("vf", vf)
          .add(encoder)
          .add(parser)
          .add(muxer)
          .add(sink)
      )

      pipeline.eos(True).execute()


  def main() -> None:

      if "HOME" not in os.environ:
          raise EnvironmentError("Error: HOME environment variable is not set.")

      from qimsdk import ImsdkGstLogMode, ImsdkLogLevel, SetImsdkGstLogMode, SetImsdkLogLevel
      SetImsdkGstLogMode(ImsdkGstLogMode.ImsdkLog)
      SetImsdkLogLevel(ImsdkLogLevel.Debug)

      create_and_execute_pipeline()


  if __name__ == "__main__":
      main()
  ```

  <Steps>
    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_camera_encoder.py
      ```

      The application records camera frames encoded as H.264 into an MP4 file saved to `~/Downloads/qimsdk_samples/media/encoder_output.mp4`.
    </Step>

    <Step title="Expected Output">
      The application records silently with no display output. An MP4 file is written to `~/Downloads/qimsdk_samples/media/encoder_output.mp4`, which you can copy off the device and play back to verify the recording.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_camera_encoder.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_camera_encoder.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### MP4 File Playback

This pipeline ingests an MP4 file, demultiplexes it to extract the video track, decodes the compressed stream (e.g., H.264/H.265) into raw frames, converts (or directly negotiates) the output to NV12 — a hardware-friendly YUV 4:2:0 semi-planar format — and then renders the frames to the display using a sink that supports zero-copy or hardware buffers.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_qtdemux_decode_display.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_qtdemux_decode_display.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_qtdemux_decode_display.py`
  </Info>

  ```python theme={null}
  #!/usr/bin/env python3

  """Transport stream demux sample that exercises H264Filter."""

  from qimsdk import Element, Pipeline, VideoFilter
  import os

  #  Example pipeline:
  #
  #    src -> demux -> parse -> decoder -> [videofilter] -> display
  #
  #  The pipeline reads an MP4/H.264 file, decodes it through the hardware decoder,
  #  and displays the result through Wayland.


  def create_and_execute_pipeline() -> None:

      # Reads the input media file as raw bytes.
      src = (
          Element("filesrc", "src")
          .set("location", f"{os.environ['HOME']}/Downloads/qimsdk_samples/media/ai_demo_sample.mp4")
      )

      # Extracts elementary streams from the MP4 container.
      demux = Element("qtdemux", "demux")

      # Prepares the H.264 bitstream for the decoder.
      parse = Element("h264parse", "parse")

      # Decodes the compressed H.264 stream into raw video frames.
      #
      # The I/O mode is configured to enforce DMA buffer usage,
      # avoiding unnecessary buffer copies.
      decoder = (
          Element("v4l2h264dec", "decoder")
          .set("output-io-mode", 4)
          .set("capture-io-mode", 4)
      )

      # Restricts the decoded stream to NV12 before it reaches the display.
      videofilter = VideoFilter().format("NV12")

      # Render video stream on display.
      display = (
          Element("waylandsink", "display")
          .set("fullscreen", True)
      )

      # Creates the pipeline, adds and links elements, and executes it.
      #
      # Linking is implicit and follows the order in which elements are added.
      pipeline = (
          Pipeline("video-pipeline")
          .add(src)
          .add(demux)
          .add(parse)
          .add(decoder)
          .add_stream_filter("videofilter", videofilter)
          .add(display)
      )

      pipeline.execute()


  def main() -> None:

      if "HOME" not in os.environ:
          raise EnvironmentError("Error: HOME environment variable is not set.")

      from qimsdk import ImsdkGstLogMode, ImsdkLogLevel, SetImsdkGstLogMode, SetImsdkLogLevel
      SetImsdkGstLogMode(ImsdkGstLogMode.ImsdkLog)
      SetImsdkLogLevel(ImsdkLogLevel.Debug)

      create_and_execute_pipeline()


  if __name__ == "__main__":
      main()
  ```

  <Steps>
    <Step title="Download Required Files">
      | File | Download | Save as |
      | - | - | - |
      | Sample video | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | `ai_demo_sample.mp4` |

      You can also use your own MP4 (H.264) file, or update the pipeline's `location` property to point to it.
    </Step>

    <Step title="Copy the video file to the device">
      ```bash theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media"
      scp ai_demo_sample.mp4 <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_qtdemux_decode_display.py
      ```

      The application decodes the video file and renders it fullscreen on the display.
    </Step>

    <Step title="Expected Output">
      The MP4 file's video track is decoded and rendered fullscreen on the display until the end of the file is reached.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_qtdemux_decode_display.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_qtdemux_decode_display.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Transport Stream (.ts) File Playback

This pipeline is similar to the Offline Video Source pipeline, but ingests an MPEG transport stream (`.ts`) file instead of an MP4 container. It demultiplexes the transport stream using `tsdemux` to extract the video track, decodes the compressed H.264 stream into raw frames, converts (or directly negotiates) the output to NV12, and renders the frames to the display.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_tsdemux_decode_display.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_tsdemux_decode_display.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_tsdemux_decode_display.py`
  </Info>

  ```python theme={null}
  #!/usr/bin/env python3

  """Transport stream demux sample that exercises H264Filter."""

  from qimsdk import Element, H264Filter, Pipeline, VideoFilter
  import os

  #  Example pipeline:
  #
  #    src -> demux -> [vf1] -> parse -> decoder -> [vf2] -> display
  #
  #  The pipeline reads a sequence of transport-stream files, extracts the H.264
  #  elementary stream, decodes it through the hardware decoder, and displays
  #  the result through Wayland.


  def create_and_execute_pipeline() -> None:

      # Reads a sequence of input files as stream data.
      src = (
          Element("multifilesrc", "src")
          .set("location", f"{os.environ['HOME']}/Downloads/qimsdk_samples/media/ai_demo_sample.ts")
          .set("stop-index", 0)
      )

      # Extracts elementary streams from the transport stream.
      demux = Element("tsdemux", "demux")

      # Restricts the extracted stream to a fixed framerate before parsing.
      vf1 = H264Filter().framerate(30)

      # Prepares the H.264 bitstream for the decoder.
      parse = Element("h264parse", "parse")

      # Decodes the compressed H.264 stream into raw video frames.
      #
      # The I/O mode is configured to enforce DMA buffer usage,
      # avoiding unnecessary buffer copies.
      decoder = (
          Element("v4l2h264dec", "decoder")
          .set("output-io-mode", 4)
          .set("capture-io-mode", 4)
      )

      # Restricts the decoded stream to NV12 before it reaches the display.
      vf2 = VideoFilter().format("NV12")

      # Render video stream on display.
      display = (
          Element("waylandsink", "display")
          .set("fullscreen", True)
      )

      # Creates the pipeline, adds and links elements, and executes it.
      #
      # Linking is implicit and follows the order in which elements are added.
      pipeline = (
          Pipeline("video-pipeline")
          .add(src)
          .add(demux)
          .add_stream_filter("vf1", vf1)
          .add(parse)
          .add(decoder)
          .add_stream_filter("vf2", vf2)
          .add(display)
      )

      pipeline.execute()


  def main() -> None:

      if "HOME" not in os.environ:
          raise EnvironmentError("Error: HOME environment variable is not set.")

      from qimsdk import ImsdkGstLogMode, ImsdkLogLevel, SetImsdkGstLogMode, SetImsdkLogLevel
      SetImsdkGstLogMode(ImsdkGstLogMode.ImsdkLog)
      SetImsdkLogLevel(ImsdkLogLevel.Debug)

      create_and_execute_pipeline()


  if __name__ == "__main__":
      main()
  ```

  <Steps>
    <Step title="Download Required Files">
      | File | Download | Save as |
      | - | - | - |
      | Sample transport stream | [ai\_demo\_sample.ts](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.ts) | `ai_demo_sample.ts` |
    </Step>

    <Step title="Copy the transport stream file to the device">
      ```bash theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media"
      scp ai_demo_sample.ts <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_tsdemux_decode_display.py
      ```

      The application decodes the transport stream file and renders it fullscreen on the display.
    </Step>

    <Step title="Expected Output">
      The transport stream file's video track is decoded and rendered fullscreen on the display until the end of the file is reached.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_tsdemux_decode_display.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_tsdemux_decode_display.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Synthetic Test Pattern (VideoFilter API)

This pipeline generates synthetic test video frames using `videotestsrc` (a bouncing ball pattern), instead of reading from a camera or file. The frames are restricted to NV12/1080p/30fps, rotated 180 degrees by [`qtivtransform`](/plugin-reference/qtivtransform), downscaled to 1280x720 by a second stream filter, and rendered to the display. This is a useful pipeline for testing display and transform stages without any camera or media file dependency.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_videotestsrc_videofilter_display.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_videotestsrc_videofilter_display.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_videotestsrc_videofilter_display.py`
  </Info>

  ```python theme={null}
  #!/usr/bin/env python3

  """Test video test source with stream filter."""

  from qimsdk import Element, Pipeline, VideoFilter

  #  Example pipeline:
  #
  #    src -> [vf1] -> transform -> [vf2] -> display
  #
  #  The pipeline generates synthetic test video frames, restricts them to NV12
  #  at 1920x1080/30fps, rotates the frames, restricts the output to 1280x720,
  #  and displays the result through Wayland.


  def create_and_execute_pipeline() -> None:

      # Generates synthetic test video frames.
      src = (
          Element("videotestsrc", "src")
          .set("pattern", "ball")
      )

      # Restricts the generated stream to NV12 at 1920x1080, 30fps.
      vf1 = VideoFilter().format("NV12").resolution(1920, 1080).framerate(30)

      # Applies geometric transforms to video frames.
      transform = (
          Element("qtivtransform", "transform")
          .set("rotate", "180")
      )

      # Restricts the transformed stream to 1280x720.
      vf2 = VideoFilter().resolution(1280, 720)

      # Render video stream on display.
      display = (
          Element("waylandsink", "display")
          .set("fullscreen", True)
      )

      # Creates the pipeline, adds and links elements, and executes it.
      #
      # Linking is implicit and follows the order in which elements are added.
      pipeline = (
          Pipeline("video-pipeline")
          .add(src)
          .add_stream_filter("vf1", vf1)
          .add(transform)
          .add_stream_filter("vf2", vf2)
          .add(display)
      )

      pipeline.execute()


  def main() -> None:

      from qimsdk import ImsdkGstLogMode, ImsdkLogLevel, SetImsdkGstLogMode, SetImsdkLogLevel
      SetImsdkGstLogMode(ImsdkGstLogMode.ImsdkLog)
      SetImsdkLogLevel(ImsdkLogLevel.Debug)

      create_and_execute_pipeline()


  if __name__ == "__main__":
      main()
  ```

  <Steps>
    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_videotestsrc_videofilter_display.py
      ```

      The application generates a synthetic bouncing-ball test pattern, rotates and rescales it, and renders it fullscreen on the display.
    </Step>

    <Step title="Expected Output">
      A synthetic bouncing-ball test pattern is rendered fullscreen on the display, rotated 180 degrees and rescaled to 1280x720 — with no camera or media file required.

      <video />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_videotestsrc_videofilter_display.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_videotestsrc_videofilter_display.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Synthetic Test Pattern (Raw Caps StreamFilter)

This pipeline is functionally identical to the previous example, but it defines its stream constraints using raw GStreamer caps strings via `StreamFilter` (e.g. `"video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1"`), instead of the structured `VideoFilter` builder API. This is useful when you need to express filter capabilities that aren't covered by the structured builder, or when porting caps strings directly from existing GStreamer pipelines.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_videotestsrc_streamfilter_display.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_videotestsrc_streamfilter_display.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_videotestsrc_streamfilter_display.py`
  </Info>

  ```python theme={null}
  #!/usr/bin/env python3

  """Test video test source with stream filter."""

  from qimsdk import Element, Pipeline, StreamFilter

  #  Example pipeline:
  #
  #    src -> [vf1] -> transform -> [vf2] -> display
  #
  #  The pipeline generates synthetic test video frames, applies a caps-string-based
  #  stream filter, rotates the frames, applies a second caps-string-based stream
  #  filter, and displays the result through Wayland.


  def create_and_execute_pipeline() -> None:

      # Generates synthetic test video frames.
      src = (
          Element("videotestsrc", "src")
          .set("pattern", "ball")
      )

      # Restricts the generated stream to NV12 at 1920x1080, 30fps.
      vf1 = StreamFilter("video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1")

      # Applies geometric transforms to video frames.
      transform = (
          Element("qtivtransform", "transform")
          .set("rotate", "180")
      )

      # Restricts the transformed stream to 1280x720.
      vf2 = StreamFilter("video/x-raw,width=1280,height=720")

      # Render video stream on display.
      display = (
          Element("waylandsink", "display")
          .set("fullscreen", True)
      )

      # Creates the pipeline, adds and links elements, and executes it.
      #
      # Linking is implicit and follows the order in which elements are added.
      pipeline = (
          Pipeline("video-pipeline-streamfilter-string")
          .add(src)
          .add_stream_filter("vf1", vf1)
          .add(transform)
          .add_stream_filter("vf2", vf2)
          .add(display)
      )

      pipeline.execute()


  def main() -> None:

      from qimsdk import ImsdkGstLogMode, ImsdkLogLevel, SetImsdkGstLogMode, SetImsdkLogLevel
      SetImsdkGstLogMode(ImsdkGstLogMode.ImsdkLog)
      SetImsdkLogLevel(ImsdkLogLevel.Debug)

      create_and_execute_pipeline()


  if __name__ == "__main__":
      main()
  ```

  <Steps>
    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_videotestsrc_streamfilter_display.py
      ```

      The application generates a synthetic bouncing-ball test pattern, rotates and rescales it, and renders it fullscreen on the display.
    </Step>

    <Step title="Expected Output">
      A synthetic bouncing-ball test pattern is rendered fullscreen on the display, rotated 180 degrees and rescaled to 1280x720, identical in appearance to the `VideoFilter` API example above.

      <video />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_videotestsrc_streamfilter_display.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_videotestsrc_streamfilter_display.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### RTSP Camera Stream Playback

This pipeline connects to a network RTSP camera stream instead of a local (USB/built-in) camera or file. `rtspsrc` establishes the RTSP session and negotiates the H.264 RTP stream, a caps filter restricts it to H.264 video, `rtph264depay` extracts the H.264 payload from the RTP packets, `h264parse` prepares the bitstream for the hardware decoder, and the decoded frames are converted to NV12 and rendered to the display.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    See the full YAML configuration here: [qimsdk\_ref\_rtsp\_camera.yaml](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_rtsp_camera.yaml)

    Check application source code on GitHub: [qimsdk\_ref\_yml.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yml.py)

    Pre-built application on device: `qimsdk_ref_yml.py`

    YAML configuration saved on device: `/etc/qimsdk/qimsdk_ref_rtsp_camera.yaml`
  </Info>

  ```yaml theme={null}
  pipeline:
    elements:
      - type: rtspsrc
        name: source
        location: rtsp://username:password@192.168.1.188:554/Streaming/Channels/101
        latency: 0

      - type: capsfilter
        name: rtp_caps
        caps: application/x-rtp,media=video,encoding-name=H264

      - type: rtph264depay
        name: depay

      - type: h264parse
        name: parse

      - type: v4l2h264dec
        name: decoder
        capture-io-mode: 4
        output-io-mode: 4

      - type: filter
        name: videostream
        video:
          format: NV12

      - type: waylandsink
        name: display
        sync: false
        fullscreen: true
  ```

  <Steps>
    <Step title="Update the RTSP camera credentials">
      On the device, edit `/etc/qimsdk/qimsdk_ref_rtsp_camera.yaml` and update the `location` property with your RTSP camera's username, password, IP address, and stream path.
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      qimsdk_ref_yml.py /etc/qimsdk/qimsdk_ref_rtsp_camera.yaml
      ```

      The application connects to the RTSP camera, decodes the H.264 stream, and renders it fullscreen on the display.
    </Step>

    <Step title="Expected Output">
      The RTSP camera's live H.264 stream is decoded and rendered fullscreen on the display, for as long as the network stream remains available.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [qimsdk\_ref\_yml.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yml.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

## AI Pipeline

### Video-to-Tensor Dump (Offline Preprocessing)

This pipeline ingests an MP4 file, demultiplexes it to extract the video track, decodes the compressed stream (e.g., H.264/H.265) into raw frames, converts (or directly negotiates) the output to NV12 — a hardware-friendly YUV 4:2:0 semi-planar format — and then goes through [`qtimlvconverter`](/plugin-reference/qtimlvconverter) (ML preprocessing). The `TensorFilter` stream filter describes the tensor format and shape as UINT8 with dimensions \[1, 520, 520, 3] (batch, height, width, channels). Finally, `multifilesink` writes the preprocessed tensors out as sequential `.rgb` files to `~/Downloads/qimsdk_samples/media/tensor_520_520_%d.rgb`, rolling after every 10 files.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_tensor_dump.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_tensor_dump.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_tensor_dump.py`
  </Info>

  <Steps>
    <Step title="Download Required Files">
      | File | Download | Save as |
      | - | - | - |
      | Sample video | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | `ai_demo_sample.mp4` |

      You can also use your own MP4 (H.264) file, or update the pipeline's `location` property to point to it.
    </Step>

    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media"
        scp ai_demo_sample.mp4 <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_tensor_dump.py
      ```

      The application writes preprocessed tensors to `~/Downloads/qimsdk_samples/media/tensor_520_520_%d.rgb`.
    </Step>

    <Step title="Expected Output">
      There is no display output. Sequential `.rgb` tensor files are written to `~/Downloads/qimsdk_samples/media/`, rolling after every 10 files, which you can inspect offline for use in other ML pipelines.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_tensor_dump.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_tensor_dump.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Live Camera YOLOv8 Detection (Manual Chain)

This pipeline captures live video from the on-device camera, then drives a full detection-and-visualization loop in real time: the raw NV12 stream is normalized and preprocessed into one or more tensors, fed to a TFLite inference stage (running on the external QNN delegate for efficient hardware execution), and the resulting tensors are postprocessed into semantic detections (classes, boxes, confidences). Those results are converted into QIM SDK-friendly metadata and attached back to the original video timeline, so the visual stream and its ML annotations stay synchronized. A parallel branch from the early tee ensures the original frames are available to a metadata muxer, which merges the detections with the video. Finally, the overlay stage renders the boxes/labels directly on the live frames, and the composed output is presented fullscreen to the display. In short, the graph turns a live camera feed into actionable on-screen insights by (1) creating a tensor view of the stream, (2) running detection, (3) translating inference outputs into QIM SDK metadata, (4) fusing metadata with the original frames, and (5) drawing the results on top of the video for low-latency, on-screen visualization.

<img alt="Introduction" />

<Accordion title="Try me">
  <Tabs>
    <Tab title="Explicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_camera_yolov8.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_camera_yolov8.py)

        Pre-built application on device: `/usr/bin/qimsdk_ref_camera_yolov8.py`
      </Info>

      #### Download Required Files

      | File | Save as |
      | - | - |
      | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |

      <Note>
        If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

        Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
      </Note>

      <Steps>
        <Step title="Copy Files to Device">
          <CodeGroup>
            ```bash SCP (SSH) theme={null}
            # Replace <user> and <device-ip> with your device credentials.
            # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

            ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
            scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            ```
          </CodeGroup>
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          python3 /usr/bin/qimsdk_ref_camera_yolov8.py
          ```

          The application overlays detected bounding boxes and labels on the live camera feed and renders it fullscreen on the display.
        </Step>

        <Step title="Expected Output">
          The live camera feed is rendered fullscreen on the display with YOLOv8 detection bounding boxes and class labels overlaid on each frame in real time.

          <img alt="Expected Output" />

          To stop the application, press **CTRL + C**.
        </Step>

        <Step title="Customize application">
          * **Source code:** [`qimsdk_ref_camera_yolov8.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_camera_yolov8.py)

          Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
        </Step>
      </Steps>
    </Tab>

    <Tab title="YAML config style">
      <Info>
        See the full YAML configuration here: [qimsdk\_ref\_camera\_yolov8.yaml](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_camera_yolov8.yaml)

        Check application source code on GitHub: [qimsdk\_ref\_yml.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yml.py)

        Pre-built application on device: `qimsdk_ref_yml.py`

        YAML configuration saved on device: `/etc/qimsdk/qimsdk_ref_camera_yolov8.yaml`
      </Info>

      ```yaml theme={null}
      pipeline:
        elements:
          - type: qtiqmmfsrc
            name: source
            camera: 0

          - type: filter
            name: videostream
            video:
              format: NV12
              width: 1920
              height: 1080
              framerate: 30

          - type: tee
            name: split

          - type: queue
            name: q1

          - type: qtimlvconverter
            name: preprocessing

          - type: queue
            name: q3

          - type: qtimltflite
            name: inferencing
            delegate: external
            external-delegate-path: libQnnTFLiteDelegate.so
            external-delegate-options: "QNNExternalDelegate,backend_type=htp;"
            model: ~/Downloads/qimsdk_samples/models/yolov8_det_quantized.tflite

          - type: queue
            name: q4

          - type: qtimlpostprocess
            name: postprocessing
            results: 5
            module: yolov8
            labels: ~/Downloads/qimsdk_samples/labels/yolov8.json
            settings: '{"confidence": 70.0}'

          - type: filter
            name: mlf
            text: {}

          - type: qtimetamux
            name: mlmuxer

          - type: queue
            name: q5

          - type: qtivoverlay
            name: overlay

          - type: waylandsink
            name: display
            sync: false
            fullscreen: true

        links:
          - [split, mlmuxer]
          - [source, videostream, split, q1, preprocessing, q3, inferencing, q4, postprocessing, mlf, mlmuxer, q5, overlay, display]
      ```

      #### Download Required Files

      | File | Save as |
      | - | - |
      | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |

      <Note>
        If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

        Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
      </Note>

      <Steps>
        <Step title="Copy Files to Device">
          <CodeGroup>
            ```bash SCP (SSH) theme={null}
            # Replace <user> and <device-ip> with your device credentials.
            # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

            ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
            scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            ```
          </CodeGroup>
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          qimsdk_ref_yml.py /etc/qimsdk/qimsdk_ref_camera_yolov8.yaml
          ```
        </Step>

        <Step title="Expected Output">
          The live camera feed is rendered fullscreen on the display with YOLOv8 detection bounding boxes and class labels overlaid on each frame in real time, identical to the explicit coding style example above.

          <img alt="Expected Output" />

          To stop the application, press **CTRL + C**.
        </Step>

        <Step title="Customize application">
          * **Source code:** [qimsdk\_ref\_yml.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yml.py)

          Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
        </Step>
      </Steps>
    </Tab>
  </Tabs>
</Accordion>

### Live Camera YOLOv8 Detection (ML Bin)

This solution is identical to the previous one. The only difference is that here we use `qtimlvideotflitebin` instead of manually linking the preprocessing, inference, postprocessing, and ML muxer components, which simplifies the pipeline.

<img alt="Introduction" />

<Accordion title="Try me">
  <Tabs>
    <Tab title="Explicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_camera_yolov8_mlbin.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_camera_yolov8_mlbin.py)

        Pre-built application on device: `/usr/bin/qimsdk_ref_camera_yolov8_mlbin.py`
      </Info>

      #### Download Required Files

      | File | Save as |
      | - | - |
      | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |

      <Note>
        If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

        Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
      </Note>

      <Steps>
        <Step title="Copy Files to Device">
          <CodeGroup>
            ```bash SCP (SSH) theme={null}
            # Replace <user> and <device-ip> with your device credentials.
            # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

            ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
            scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            ```
          </CodeGroup>
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          python3 /usr/bin/qimsdk_ref_camera_yolov8_mlbin.py
          ```

          The application overlays detected bounding boxes and labels on the live camera feed and renders it fullscreen on the display.
        </Step>

        <Step title="Expected Output">
          The live camera feed is rendered fullscreen on the display with YOLOv8 detection bounding boxes and class labels overlaid on each frame in real time.

          <img alt="Expected Output" />

          To stop the application, press **CTRL + C**.
        </Step>

        <Step title="Customize application">
          * **Source code:** [`qimsdk_ref_camera_yolov8_mlbin.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_camera_yolov8_mlbin.py)

          Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
        </Step>
      </Steps>
    </Tab>

    <Tab title="YAML config style">
      <Info>
        See the full YAML configuration here: [qimsdk\_ref\_mlbin\_yolov8.yaml](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_mlbin_yolov8.yaml)

        Check application source code on GitHub: [qimsdk\_ref\_yml.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yml.py)

        Pre-built application on device: `qimsdk_ref_yml.py`

        YAML configuration saved on device: `/etc/qimsdk/qimsdk_ref_mlbin_yolov8.yaml`
      </Info>

      ```yaml theme={null}
      pipeline:
        elements:
          - type: filesrc
            name: src
            location: ~/Downloads/qimsdk_samples/media/ai_demo_sample.mp4

          - type: qtdemux
            name: demux

          - type: h264parse
            name: parse

          - type: v4l2h264dec
            name: decoder
            output-io-mode: 4
            capture-io-mode: 4

          - type: filter
            name: videofilter
            video:
              format: NV12

          - type: qtimlvideotflitebin
            name: mlbin
            inference-delegate: "external"
            inference-external-delegate-path: "libQnnTFLiteDelegate.so"
            inference-external-delegate-options: "QNNExternalDelegate,backend_type=htp;"
            inference-model: "~/Downloads/qimsdk_samples/models/yolov8_det_quantized.tflite"
            postprocess-module: "yolov8"
            postprocess-labels: "~/Downloads/qimsdk_samples/labels/yolov8.json"

          - type: qtivoverlay
            name: overlay

          - type: waylandsink
            name: display
            fullscreen: true
      ```

      #### Download Required Files

      | File | Save as |
      | - | - |
      | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |

      <Note>
        If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

        Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
      </Note>

      <Steps>
        <Step title="Copy Files to Device">
          <CodeGroup>
            ```bash SCP (SSH) theme={null}
            # Replace <user> and <device-ip> with your device credentials.
            # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

            ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
            scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            ```
          </CodeGroup>
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          qimsdk_ref_yml.py /etc/qimsdk/qimsdk_ref_mlbin_yolov8.yaml
          ```

          The application overlays detected bounding boxes and labels on the live camera feed and renders it fullscreen on the display.
        </Step>

        <Step title="Expected Output">
          The live camera feed is rendered fullscreen on the display with YOLOv8 detection bounding boxes and class labels overlaid on each frame in real time, produced by the all-in-one ML bin instead of a manually chained pipeline.

          <img alt="Expected Output" />

          To stop the application, press **CTRL + C**.
        </Step>

        <Step title="Customize application">
          * **Source code:** [qimsdk\_ref\_yml.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yml.py)

          Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
        </Step>
      </Steps>
    </Tab>
  </Tabs>
</Accordion>

### Offline Video YOLOv8 Detection (Manual Chain, Metamux Overlay)

This pipeline is the offline-video counterpart to Live Camera YOLOv8 Detection (Manual Chain): instead of a live camera feed, an MP4 file is demuxed and decoded, then split by a `tee` into a display branch and an ML branch. The ML branch runs preprocessing, inference, and postprocessing manually, and the resulting detections are attached back onto the original frame via [`qtimetamux`](/plugin-reference/qtimetamux) before being overlaid and rendered. This Metamux-based approach is more flexible than the video composer approach (it allows chaining additional ML models or streaming metadata separately), whereas the composer overlay approach used in the next example offers better performance.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_yolov8_metamux_overlay.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yolov8_metamux_overlay.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_yolov8_metamux_overlay.py`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
  | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4          <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_yolov8_metamux_overlay.py
      ```

      The application decodes the video file, runs YOLOv8 detection, attaches the detections to the original frame via [`qtimetamux`](/plugin-reference/qtimetamux), and overlays the results, rendering them fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with YOLOv8 bounding boxes and class labels overlaid on each detected object, rendered fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_yolov8_metamux_overlay.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yolov8_metamux_overlay.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Offline Video YOLOv8 Detection (ML Bin, Composer Overlay)

The AI output can be overlaid directly on top of the frame instead of attaching AI metadata to the main frame. This approach provides better performance, whereas the Metamux solution is more flexible and can be used for chaining AI models or streaming AI metadata separately.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_yolov8_composer_overlay.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yolov8_composer_overlay.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_yolov8_composer_overlay.py`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
  | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4          <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_yolov8_composer_overlay.py
      ```

      The application composes the AI detection overlay directly on top of the decoded video frames and displays it fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with YOLOv8 bounding boxes and class labels composited directly onto each frame, rendered fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_yolov8_composer_overlay.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yolov8_composer_overlay.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Gesture Recognition

This pipeline demonstrates a daisy-chained, multi-model gesture recognition graph built entirely from standard SDK elements ([`qtimlvconverter`](/plugin-reference/qtimlvconverter), [`qtimltflite`](/plugin-reference/qtimltflite), [`qtimlpostprocess`](/plugin-reference/qtimlpostprocess), [`qtimetamux`](/plugin-reference/qtimetamux), `qtimetatransform`) rather than a custom postprocessing callback. Palm detection locates the hand region in the live camera frame; that region is transformed and fed into hand landmark detection; the resulting landmarks are passed through a gesture embedder followed by a canned gesture classifier to decode the recognized gesture. Two parallel [`qtimetamux`](/plugin-reference/qtimetamux) merge points combine metadata from each stage back onto the original frame before the final overlay is rendered to the display.

<img alt="Introduction" />

<Accordion title="Try me">
  <Tabs>
    <Tab title="Explicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_gesture_recognition.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_gesture_recognition.py)

        Pre-built application on device: `/usr/bin/qimsdk_ref_gesture_recognition.py`
      </Info>
    </Tab>

    <Tab title="YAML config style">
      <Info>
        See the full YAML configuration here: [qimsdk\_ref\_gesture\_recognition.yaml](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_gesture_recognition.yaml)

        Check application source code on GitHub: [qimsdk\_ref\_yml.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yml.py)

        Pre-built application on device: `qimsdk_ref_yml.py`

        YAML configuration saved on device: `/etc/qimsdk/qimsdk_ref_gesture_recognition.yaml`
      </Info>
    </Tab>
  </Tabs>

  #### Download Required Files

  Download the gesture recognizer task bundle from Google MediaPipe to obtain the palm detection, hand landmark, gesture embedder, and canned gesture classifier models:

  ```bash theme={null}
  # Download the gesture recognizer task bundle
  wget https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/latest/gesture_recognizer.task

  # Extract the top-level task
  unzip gesture_recognizer.task

  # Extract hand landmarker models
  unzip hand_landmarker.task
  # → hand_detector.tflite, hand_landmarks_detector.tflite

  # Extract gesture recognizer models
  unzip hand_gesture_recognizer.task
  # → gesture_embedder.tflite, canned_gesture_classifier.tflite
  ```

  | File | Save as |
  | - | - |
  | hand\_detector.tflite (see steps above) | palm\_detection\_full.tflite |
  | hand\_landmarks\_detector.tflite (see steps above) | hand\_landmark\_full.tflite |
  | gesture\_embedder.tflite (see steps above) | gesture\_embedder.tflite |
  | canned\_gesture\_classifier.tflite (see steps above) | canned\_gesture\_classifier.tflite |
  | <a href="../labels/palmd_labels.json">palmd\_labels.json</a> | palmd\_labels.json |
  | <a href="../labels/palmd_settings.json">palmd\_settings.json</a> | palmd\_settings.json |
  | <a href="../labels/hlandmarks.json">hlandmarks.json</a> | hlandmarks.json |
  | <a href="../labels/hlandmark_settings.json">hlandmark\_settings.json</a> | hlandmark\_settings.json |
  | <a href="../labels/gesture_rec.json">gesture\_rec.json</a> | gesture\_rec.json |

  <Note>
    If a downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      This pipeline's source is the on-device camera, so no video file needs to be copied.

      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
        scp palm_detection_full.tflite       <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp hand_landmark_full.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp gesture_embedder.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp canned_gesture_classifier.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp palmd_labels.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp palmd_settings.json              <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp hlandmarks.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp hlandmark_settings.json          <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp gesture_rec.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_gesture_recognition.py
      ```

      The application captures live video from the on-device camera, runs the daisy-chained palm detection → hand landmark → gesture embedder → canned gesture classifier models, and overlays the recognized gesture on the display fullscreen.
    </Step>

    <Step title="Expected Output">
      The live camera feed is rendered fullscreen on the display with the recognized gesture label overlaid on each frame in real time.

      <video />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_gesture_recognition.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_gesture_recognition.py), [qimsdk\_ref\_yml.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yml.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### PPE & Person/Foot Detection (ML Bin)

This pipeline demonstrates a two-model ML bin pipeline for personal protective equipment (PPE) and person/foot detection: `foot_track_net` detects persons and their feet in the frame, while `gear_guard_net` detects PPE gear. Both models run through `qtimlvideotflitebin`, which internally handles preprocessing, inference, and postprocessing, simplifying the pipeline compared to manually chaining each stage.

<img alt="Introduction" />

<Accordion title="Try me">
  <Tabs>
    <Tab title="Explicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_mlbin_ppe.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_mlbin_ppe.py)

        Pre-built application on device: `/usr/bin/qimsdk_ref_mlbin_ppe.py`
      </Info>
    </Tab>

    <Tab title="YAML config style">
      <Info>
        See the full YAML configuration here: [qimsdk\_ref\_mlbin\_ppe.yaml](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_mlbin_ppe.yaml)

        Check application source code on GitHub: [qimsdk\_ref\_yml.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yml.py)

        Pre-built application on device: `qimsdk_ref_yml.py`

        YAML configuration saved on device: `/etc/qimsdk/qimsdk_ref_mlbin_ppe.yaml`
      </Info>
    </Tab>
  </Tabs>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/models/gear_guard_net?searchTerm=gear) | gear\_guard\_net-ppe-detection-w8a8.tflite |
  | [Model](https://aihub.qualcomm.com/models/foot_track_net?searchTerm=foot) | foot\_track\_net-person-foot-detection-w8a8.tflite |
  | <a href="../labels/gear_guard_net.json">gear\_guard\_net.json</a> | gear\_guard\_net.json |
  | <a href="../labels/foot_track_net.json">foot\_track\_net.json</a> | foot\_track\_net.json |
  | <a href="../labels/foot_track_net_settings.json">foot\_track\_net\_settings.json</a> | foot\_track\_net\_settings.json |
  | [ppe\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/ppe_sample.mp4) | ppe\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp gear_guard_net-ppe-detection-w8a8.tflite      <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp foot_track_net-person-foot-detection-w8a8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp gear_guard_net.json                           <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp foot_track_net.json                           <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp foot_track_net_settings.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ppe_sample.mp4                                <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_mlbin_ppe.py
      ```

      The application runs inference on the video file, detecting persons/feet with `foot_track_net` and PPE gear with `gear_guard_net`, overlaying the resulting detections on the video and rendering it fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with person/foot and PPE-gear detection bounding boxes overlaid, rendered fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_mlbin_ppe.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_mlbin_ppe.py), [qimsdk\_ref\_yml.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yml.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### YOLOv8 Detection (ML Bin)

This pipeline reads an MP4/H.264 file, decodes it through the hardware decoder, and runs YOLOv8 object detection using `qtimlvideotflitebin`, which internally handles preprocessing, inference, and postprocessing. The detected objects are then overlaid on the video and displayed.

<img alt="Introduction" />

<Accordion title="Try me">
  <Tabs>
    <Tab title="Explicit coding style">
      <Info>
        Check application source code on GitHub: [`qimsdk_ref_mlbin_yolov8.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_mlbin_yolov8.py)

        Pre-built application on device: `/usr/bin/qimsdk_ref_mlbin_yolov8.py`
      </Info>
    </Tab>

    <Tab title="YAML config style">
      <Info>
        See the full YAML configuration here: [qimsdk\_ref\_mlbin\_yolov8.yaml](https://github.com/qualcomm/qimsdk/blob/main/tools/appbuilders-configs/qimsdk_ref_mlbin_yolov8.yaml)

        Check application source code on GitHub: [qimsdk\_ref\_yml.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yml.py)

        Pre-built application on device: `qimsdk_ref_yml.py`

        YAML configuration saved on device: `/etc/qimsdk/qimsdk_ref_mlbin_yolov8.yaml`
      </Info>
    </Tab>
  </Tabs>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
  | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4          <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_mlbin_yolov8.py
      ```

      The application decodes the video file, runs YOLOv8 detection through `qtimlvideotflitebin`, and overlays the detected objects on the video, rendering it fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with YOLOv8 bounding boxes and class labels overlaid on each detected object, rendered fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_mlbin_yolov8.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_mlbin_yolov8.py), [qimsdk\_ref\_yml.py](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_yml.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

## AppSink and AppSrc

<img alt="Introduction" />

AppSink and AppSrc are special elements because they expose buffers directly to and from the application. Because of this, we create and configure these elements separately, using their dedicated wrapper classes. Once configured, we insert them into the pipeline as fully constructed element objects.

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_appsrc_and_appsink.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_appsrc_and_appsink.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_appsrc_and_appsink.py`
  </Info>

  <Steps>
    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_appsrc_and_appsink.py
      ```

      The application generates synthetic test frames, pushes them through an `AppSink`/`AppSrc` bridge, and renders the resulting stream fullscreen on the display.
    </Step>

    <Step title="Expected Output">
      The synthetic test frames pushed through the `AppSink`/`AppSrc` bridge are rendered fullscreen on the display, confirming the buffers passed through application code unmodified.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_appsrc_and_appsink.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_appsrc_and_appsink.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

## Custom Pre and Post Processing

<img alt="Introduction" />

### Custom Preprocessing for YOLOv8 (Detection)

QIM SDK plugin support different backend: GLES, OpenCV, etc. This level of interface could be exposed to application. This API takes all input and output at once in case of batching or daisy for example. It can be simplified by calling application implementation for every input. Advantage of this approach is that all complexity of batching, depth, daisy chain support remains hidden in the plugin and application will do only pre-processing on a single frame/tensor at once.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_preprocess_detection_yolov8.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_preprocess_detection_yolov8.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_preprocess_detection_yolov8.py`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
  | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4          <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_external_preprocess_detection_yolov8.py
      ```
    </Step>

    <Step title="Expected Output">
      The video plays back with YOLOv8 bounding boxes and class labels overlaid on each detected object, rendered fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_preprocess_detection_yolov8.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_preprocess_detection_yolov8.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Custom Preprocessing for YOLOv8 (ML Bin variant)

This example demonstrates external preprocessing support inside `MLVideoTFLiteBin`. Registering a callback via `MLVideoTFLiteBin.set_preprocess_handler(...)` disables the bin's internal preprocessing path (the SDK sets `preprocess-engine=none` on the bin's internal `mlpreprocess` element automatically) and hands frame-to-tensor conversion off to an application-defined Python callback. The callback receives the decoded NV12 blit(s) as NumPy-backed buffers and writes directly into the bin's output tensor (a writable, in-place NumPy view), converting each pixel to the quantized int8 NHWC layout expected by the YOLOv8 model, with letterbox padding for any area the blit doesn't cover. Inference and postprocessing then run inside the same bin, and the detected objects are overlaid on the video and displayed.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_preprocess_detection_mlbin_yolov8.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_preprocess_detection_mlbin_yolov8.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_preprocess_detection_mlbin_yolov8.py`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
  | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4          <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_external_preprocess_detection_mlbin_yolov8.py
      ```

      The application decodes the video file, runs the custom, application-defined Python preprocessing callback inside `mlbin` to convert NV12 frames into the quantized int8 tensor expected by YOLOv8, then runs inference and postprocessing, overlaying the detected objects and rendering the result fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with YOLOv8 bounding boxes and class labels overlaid on each detected object, rendered fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_preprocess_detection_mlbin_yolov8.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_preprocess_detection_mlbin_yolov8.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Custom Preprocessing with Lightweight Face Detection

This example demonstrates external preprocessing with [`MLVConverter`](/plugin-reference/qtimlvconverter) (rather than `MLVideoTFLiteBin`), so the pipeline links preprocessing, inference, and postprocessing manually through explicit queues, and merges the detections back onto the original frame via [`qtimetamux`](/plugin-reference/qtimetamux) before overlaying. Setting `engine` to `"none"` on the [`MLVConverter`](/plugin-reference/qtimlvconverter) element disables its internal preprocessing path and hands frame-to-tensor conversion off to an application-defined callback registered via [`MLVConverter.set_handler(...)`](/plugin-reference/qtimlvconverter). Unlike the YOLOv8 example, the `face_det_lite_w8a8.tflite` model consumes raw uint8 grayscale, so the callback resizes only the NV12 luma (Y) plane into a single-channel `[1, H, W, 1]` NumPy tensor, skipping color conversion and normalization entirely and leaving the chroma plane unread. A `tee` right after decoding splits the stream into a display branch and an ML branch, and a queue on each branch (including the display one) keeps the two branches from stalling each other on a shared thread.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_preprocess_lightweight_face_detect.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_preprocess_lightweight_face_detect.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_preprocess_lightweight_face_detect.py`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/face_det_lite) | face\_det\_lite\_w8a8.tflite |
  | <a href="../labels/qfd-labels.json">qfd-labels.json</a> | qfd-labels.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | face\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp face_det_lite_w8a8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp qfd-labels.json            <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp face_sample.mp4            <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_external_preprocess_lightweight_face_detect.py
      ```

      The application decodes the video file, runs the custom, application-defined Python preprocessing callback to convert the NV12 luma plane into the grayscale tensor expected by the lightweight face detector, then runs inference and postprocessing, merging the detected faces onto the original frame via [`qtimetamux`](/plugin-reference/qtimetamux), overlaying them, and rendering the result fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with face-detection bounding boxes decoded by the custom grayscale preprocessing callback, overlaid and rendered fullscreen on the display.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_preprocess_lightweight_face_detect.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_preprocess_lightweight_face_detect.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Custom Preprocessing with Palm Detection

This example is the live-camera counterpart to the lightweight face detection pipeline: [`CamSrc`](/plugin-reference/qticamsrc) captures frames directly instead of decoding a file, but the branching structure is otherwise identical — a `tee` right after the stream filter splits the pipeline into a display branch and an ML branch, each guarded by its own queue, and [`qtimetamux`](/plugin-reference/qtimetamux) merges the ML branch's detections back onto the original frame before overlay. Setting `engine` to `"none"` on the [`MLVConverter`](/plugin-reference/qtimlvconverter) element disables its internal preprocessing path and hands frame-to-tensor conversion off to an application-defined callback registered via [`MLVConverter.set_handler(...)`](/plugin-reference/qtimlvconverter). Unlike the grayscale face-detection example, `palm_detection_full.tflite` expects a `[1, H, W, 3]` float32 RGB tensor normalized to `[0, 1]`, so the callback converts the full NV12 image (both luma and chroma planes) to RGB using NumPy and scales each channel into that range, with letterbox padding for any area the blit doesn't cover.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_preprocess_palm_detect.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_preprocess_palm_detect.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_preprocess_palm_detect.py`
  </Info>

  #### Download Required Files

  Download the gesture recognizer task bundle from Google MediaPipe to obtain the palm detection model:

  ```bash theme={null}
  # Download the gesture recognizer task bundle
  wget https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/latest/gesture_recognizer.task

  # Extract the top-level task
  unzip gesture_recognizer.task

  # Extract hand landmarker models
  unzip hand_landmarker.task
  # → hand_detector.tflite, hand_landmarks_detector.tflite
  ```

  | File | Save as |
  | - | - |
  | hand\_detector.tflite (see steps above) | palm\_detection\_full.tflite |
  | <a href="../labels/palmd_labels.json">palmd\_labels.json</a> | palmd\_labels.json |
  | <a href="../labels/palmd_settings.json">palmd\_settings.json</a> | palmd\_settings.json |

  <Note>
    If a downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      This pipeline's source is the on-device camera, so no video file needs to be copied.

      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
        scp palm_detection_full.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp palmd_labels.json          <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp palmd_settings.json        <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_external_preprocess_palm_detect.py
      ```

      The application captures live video from the on-device camera, runs the custom, application-defined Python preprocessing callback to convert NV12 frames into the normalized float32 RGB tensor expected by the palm detector, then runs inference and postprocessing, merging the detected palms onto the original frame via [`qtimetamux`](/plugin-reference/qtimetamux), overlaying them, and rendering the result fullscreen.
    </Step>

    <Step title="Expected Output">
      The live camera feed is rendered fullscreen on the display with palm-detection bounding boxes decoded by the custom float32 RGB preprocessing callback overlaid on each frame in real time.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_preprocess_palm_detect.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_preprocess_palm_detect.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Custom Postprocessing with YOLOv5 (Detection)

This example demonstrates the object detection postprocessing callback (`ObjectDetectionPostprocessCallback`) using a YOLOv5 model. Preprocessing and inference are handled by the SDK, and the raw output tensors are decoded into bounding boxes, class names, and confidence scores by an application-defined callback, which are then overlaid on the video and displayed.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_detection_yolov5.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_detection_yolov5.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_detection_yolov5.py`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/models/yolov5?searchTerm=yolov5) | yolov5m-320x320-int8.tflite |
  | <a href="../labels/yolov5m.json">yolov5m.json</a> | yolov5m.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov5m-320x320-int8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov5m.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4           <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_external_postprocess_detection_yolov5.py
      ```

      The application runs inference on the video file and uses the custom, application-defined postprocessing callback to decode detections, which are then overlaid and displayed fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with YOLOv5 bounding boxes and class labels decoded by the custom postprocessing callback, overlaid and rendered fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_postprocess_detection_yolov5.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_detection_yolov5.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Custom Post processing with ML Bin

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_mlbin_yolov8.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_mlbin_yolov8.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_mlbin_yolov8.py`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
  | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4          <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_external_postprocess_mlbin_yolov8.py
      ```

      The application runs preprocessing, inference, and the custom, application-defined postprocessing callback all inside `mlbin`, overlaying the resulting detections on the video and rendering it fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with YOLOv8 bounding boxes and class labels decoded by the custom postprocessing callback, overlaid and rendered fullscreen on the display.

      <img alt="Expected Output" />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_postprocess_mlbin_yolov8.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_mlbin_yolov8.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Custom Postprocessing with ResNet101 (Classification)

This example demonstrates the classification postprocessing callback (`ClassificationPostprocessCallback`). Preprocessing and inference are handled by the SDK, and the raw output tensors from a ResNet101 model are decoded into labeled classification results (name and confidence) by an application-defined callback.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_classification_resnet101.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_classification_resnet101.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_classification_resnet101.py`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/resnet101) | Resnet101\_Quantized.tflite |
  | <a href="../labels/resnet101.json">resnet101.json</a> | resnet101.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | classification\_sample.mp4 |

  <Note>
    No ideal sample video ships with the SDK for this use case. Supply your own content for best results.
  </Note>

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp Resnet101_Quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp resnet101.json             <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp classification_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_external_postprocess_classification_resnet101.py
      ```

      The application runs inference on the ResNet101 model and uses the custom, application-defined postprocessing callback to decode classification results, which are then displayed.
    </Step>

    <Step title="Expected Output">
      The video plays back with the top classification result and its confidence score decoded by the custom postprocessing callback, displayed as text overlaid on the frame.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_postprocess_classification_resnet101.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_classification_resnet101.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Custom Postprocessing with MiDaS v2 (Depth Estimation)

This example demonstrates the depth estimation postprocessing callback (`DepthEstimationPostprocessCallback`). Preprocessing and inference are handled by the SDK, and the raw output tensors from a MiDaS v2 model are decoded by an application-defined callback into a per-pixel depth map (values and pseudo-color) for downstream visualization.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_depth_estimation_midasv2.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_depth_estimation_midasv2.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_depth_estimation_midasv2.py`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/models/midas) | midas-tflite-w8a8.tflite |
  | <a href="../labels/midas-v2-labels.json">midas-v2-labels.json</a> | midas-v2-labels.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp midas-tflite-w8a8.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp midas-v2-labels.json      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4        <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_external_postprocess_depth_estimation_midasv2.py
      ```

      The application runs inference on the MiDaS v2 model and uses the custom, application-defined postprocessing callback to decode the depth map, which is then rendered.
    </Step>

    <Step title="Expected Output">
      The video plays back with a pseudo-colored depth map decoded by the custom postprocessing callback, rendered fullscreen on the display in place of the original frame.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_postprocess_depth_estimation_midasv2.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_depth_estimation_midasv2.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Custom Postprocessing with Gesture Recognition

This example demonstrates gesture recognition using a daisy-chained, multi-model pipeline: palm detection locates the hand region, hand landmark detection extracts keypoints from that region, and a gesture embedder followed by a canned gesture classifier decode the landmarks into a recognized gesture. Preprocessing and inference are handled by the SDK, and the raw output tensors are decoded by an application-defined postprocessing callback. The pipeline uses the on-device camera as its source.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_gesture_recognition.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_gesture_recognition.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_gesture_recognition.py`
  </Info>

  #### Download Required Files

  Download the gesture recognizer task bundle from Google MediaPipe to obtain the palm detection, hand landmark, gesture embedder, and canned gesture classifier models:

  ```bash theme={null}
  # Download the gesture recognizer task bundle
  wget https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/latest/gesture_recognizer.task

  # Extract the top-level task
  unzip gesture_recognizer.task

  # Extract hand landmarker models
  unzip hand_landmarker.task
  # → hand_detector.tflite, hand_landmarks_detector.tflite

  # Extract gesture recognizer models
  unzip hand_gesture_recognizer.task
  # → gesture_embedder.tflite, canned_gesture_classifier.tflite
  ```

  | File | Save as |
  | - | - |
  | hand\_detector.tflite (see steps above) | palm\_detection\_full.tflite |
  | hand\_landmarks\_detector.tflite (see steps above) | hand\_landmark\_full.tflite |
  | gesture\_embedder.tflite (see steps above) | gesture\_embedder.tflite |
  | canned\_gesture\_classifier.tflite (see steps above) | canned\_gesture\_classifier.tflite |
  | <a href="../labels/palmd_labels.json">palmd\_labels.json</a> | palmd\_labels.json |
  | <a href="../labels/hlandmarks.json">hlandmarks.json</a> | hlandmarks.json |
  | <a href="../labels/gesture_rec.json">gesture\_rec.json</a> | gesture\_rec.json |

  <Note>
    If a downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
        scp palm_detection_full.tflite       <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp hand_landmark_full.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp gesture_embedder.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp canned_gesture_classifier.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp palmd_labels.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp hlandmarks.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp gesture_rec.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_external_postprocess_gesture_recognition.py
      ```

      The application captures live video from the on-device camera, runs the daisy-chained palm detection → hand landmark → gesture embedder → canned gesture classifier models, and uses the custom, application-defined postprocessing callback to decode the recognized gesture, which is then overlaid and displayed fullscreen.
    </Step>

    <Step title="Expected Output">
      The live camera feed is rendered fullscreen on the display with the recognized gesture label, decoded by the custom postprocessing callback, overlaid on each frame in real time.

      <video />

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_postprocess_gesture_recognition.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_gesture_recognition.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Custom Postprocessing with HRNet (Pose Estimation)

This example demonstrates the pose estimation postprocessing callback (`PoseEstimationPostprocessCallback`) using a two-stage pipeline: a YOLOv5 model detects persons in the frame, and an HRNet model estimates keypoints for each detected person. Preprocessing and inference are handled by the SDK, and the raw output tensors are decoded into keypoints and skeleton links by an application-defined callback, which are then overlaid on the video and displayed.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_pose_estimation_hrnet.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_pose_estimation_hrnet.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_pose_estimation_hrnet.py`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/models/yolov5?searchTerm=yolov5) | yolov5m-320x320-int8.tflite |
  | [Model](https://aihub.qualcomm.com/models/hrnet_pose?searchTerm=hr) | hrnet\_pose\_w8a8.tflite |
  | <a href="../labels/yolov5m.json">yolov5m.json</a> | yolov5m.json |
  | <a href="../labels/hrnet.json">hrnet.json</a> | hrnet.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | pose\_sample.mp4 |

  <Note>
    No ideal sample video ships with the SDK for this use case. Supply your own content for best results.
  </Note>

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp yolov5m-320x320-int8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp hrnet_pose_w8a8.tflite      <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov5m.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp hrnet.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp pose_sample.mp4             <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_external_postprocess_pose_estimation_hrnet.py
      ```

      The application runs inference on the video file, first detecting persons with YOLOv5 and then estimating pose keypoints with HRNet, using the custom, application-defined postprocessing callback to decode the results, which are then overlaid and displayed fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with skeleton keypoints and links decoded by the custom postprocessing callback, overlaid on each detected person and rendered fullscreen on the display.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_postprocess_pose_estimation_hrnet.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_pose_estimation_hrnet.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>

### Custom Postprocessing with DeepLabv3 (Segmentation)

This example demonstrates the segmentation postprocessing callback (`SegmentationPostprocessCallback`) using a DeepLabv3 model. Preprocessing and inference are handled by the SDK, and the raw output tensors are decoded into per-pixel semantic labels and colors by an application-defined callback, which are then overlaid on the video and displayed.

<img alt="Introduction" />

<Accordion title="Try me">
  <Info>
    Check application source code on GitHub: [`qimsdk_ref_external_postprocess_segmentation_deeplab.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_segmentation_deeplab.py)

    Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_segmentation_deeplab.py`
  </Info>

  #### Download Required Files

  | File | Save as |
  | - | - |
  | [Model](https://aihub.qualcomm.com/iot/models/deeplabv3_plus_mobilenet) | dv3\_argmax\_int32.tflite |
  | <a href="../labels/dv3-argmax-labels.json">dv3-argmax-labels.json</a> | dv3-argmax-labels.json |
  | [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |

  <Note>
    If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`

    Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
  </Note>

  <Steps>
    <Step title="Copy Files to Device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Replace <user> and <device-ip> with your device credentials.
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
        scp dv3_argmax_int32.tflite    <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp dv3-argmax-labels.json     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4         <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      python3 /usr/bin/qimsdk_ref_external_postprocess_segmentation_deeplab.py
      ```

      The application runs inference on the video file, segmenting the frame with DeepLabv3, and uses the custom, application-defined postprocessing callback to decode the per-pixel segmentation results, which are then overlaid and displayed fullscreen.
    </Step>

    <Step title="Expected Output">
      The video plays back with a semantic segmentation color mask decoded by the custom postprocessing callback, overlaid on each frame and rendered fullscreen on the display.

      To stop the application, press **CTRL + C**.
    </Step>

    <Step title="Customize application">
      * **Source code:** [`qimsdk_ref_external_postprocess_segmentation_deeplab.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/reference-apps/qimsdk_ref_external_postprocess_segmentation_deeplab.py)

      Python applications run directly from source, so no build or compilation step is required. Edit the script directly and rerun it to apply your changes.
    </Step>
  </Steps>
</Accordion>


# Benefits
Source: https://imsdkdocs.qualcomm.com/benefits



### Why developers should use the Qualcomm Intelligent Multimedia SDK

<Card title="Single, Unified Multimedia + AI Framework" icon="arrows-to-circle">
  Combines multimedia pipelines and ML inference into a single, coherent developer experience.
</Card>

<Card title="Hardware-Accelerated by Default" icon="rabbit-running">
  QIM SDK natively exploits QCOM accelerators including CPU, GPU, NPU, ISP, VPU and DSP through optimized plugins
</Card>

<Card title="Gstreamer-Based, Industry-Standard Pipelines" icon="pipe-circle-check">
  Built on Gstreamer, providing a mature, composable pipeline model with 40+ Qualcomm-optimized plugins.
</Card>

<Card title="Flexible AI Model Support" icon="microchip-ai">
  SDK supports running models from common frameworks such as TFLite/LiteRT, ONNX, and QNN.
</Card>

<Card title="Qualcomm AI Hub Integration" icon="cloud-arrow-down">
  Browse, download, and deploy hundreds of pre-trained, quantized models directly from [Qualcomm AI Hub](https://aihub.qualcomm.com/). Models arrive ready to run — no manual conversion, calibration, or profiling required.
</Card>

***

## How QIM SDK Delivers These Benefits

<AccordionGroup>
  <Accordion title="Single, Unified Multimedia + AI Framework">
    **What developers get**

    * One pipeline for capture, preprocess, inference, overlay, encode, and streaming.
    * One metadata flow for pixels and AI results.
    * Less custom glue code between multimedia and ML stages.

    **How QIM SDK delivers it**

    * Builds on GStreamer, so media and AI elements live in the same pipeline graph.
    * Carries detections, labels, keypoints, and tensor info as frame metadata.
    * Supports sequential flows like detection → classification and parallel flows using `tee` + [`qtimetamux`](plugin-reference/qtimetamux).
    * Supports cross-pipeline designs with [`qtisocketsink`](plugin-reference/qtisocketsink) and [`qtisocketsrc`](plugin-reference/qtisocketsrc) when applications need modular stages.

    **Developer takeaway**
    You describe the full application as a pipeline instead of stitching together separate camera, inference, and rendering subsystems.
  </Accordion>

  <Accordion title="Hardware-Accelerated by Default">
    **What developers get**

    * Lower latency.
    * Lower CPU load.
    * Better throughput for multi-stream and real-time AI.

    **How QIM SDK delivers it**

    * **ISP:** camera ingest and imaging path use Qualcomm imaging hardware.
    * **GPU:** plugins such as [`qtimlvconverter`](plugin-reference/qtimlvconverter), [`qtivtransform`](plugin-reference/qtivtransform), [`qtivoverlay`](plugin-reference/qtivoverlay), and [`qtivcomposer`](plugin-reference/qtivcomposer) offload resize, color convert, crop, rotate, overlay, and composition work.
    * **NPU:** inference workloads run through supported runtimes on dedicated AI hardware.
    * **VPU:** [`v4l2h264dec`](plugin-reference/v4l2h264dec), `v4l2h265dec`, `v4l2h264enc`, and `v4l2h265enc` handle video decode/encode efficiently.
    * **Zero-copy + buffer pools:** frames move through the pipeline as DMA-buf handles with fewer copies and lower allocation overhead.

    **Developer takeaway**
    Most performance-critical stages are expressed by choosing the right plugins, not by writing accelerator-specific application code.
  </Accordion>

  <Accordion title="Qualcomm AI Hub Integration">
    **What developers get**

    * Hundreds of pre-trained, profiled, quantized models ready to deploy.
    * No manual conversion, calibration, or accuracy validation.
    * Models arrive as `.tflite`, `.dlc`, or `.bin` files compatible with QIM SDK runtimes.

    **How QIM SDK delivers it**

    * Models from AI Hub are exported targeting specific Qualcomm SoC backends (HTP, GPU, CPU).
    * [`qtimltflite`](plugin-reference/qtimltflite), `qtimlsnpe`, and [`qtimlqnn`](plugin-reference/qtimlqnn) load these models directly — no format adaptation layer.
    * Confidence thresholds, post-processing modules, and label files are supplied alongside the model, matching the QIM SDK [`qtimlpostprocess`](plugin-reference/qtimlpostprocess) interface.

    **Developer takeaway**
    Browse AI Hub, download a model, drop it into the pipeline — no custom integration work required.
  </Accordion>

  <Accordion title="DMA Zero-Copy Buffer Sharing">
    **What developers get**

    * Frames never copied through CPU memory between accelerator stages.
    * Lower peak memory bandwidth utilization.
    * Reduced end-to-end pipeline latency, especially at 1080p and above.

    **How QIM SDK delivers it**

    * Plugins negotiate DMA-buf allocation through GStreamer's allocation query mechanism.
    * ISP (`qticamsrc`) → GPU ([`qtimlvconverter`](plugin-reference/qtimlvconverter), [`qtivtransform`](plugin-reference/qtivtransform)) → NPU ([`qtimltflite`](plugin-reference/qtimltflite)) → display ([`waylandsink`](plugin-reference/waylandsink)) all share the same buffer handle.
    * The `capture-io-mode=dmabuf` and `output-io-mode=dmabuf-import` properties on decode/encode plugins enable hardware-to-hardware buffer passing.

    **Developer takeaway**
    Zero-copy is opt-in via plugin properties, not a framework limitation — the pipeline negotiates it automatically when all stages support it.
  </Accordion>

  <Accordion title="Loadable Post-Processing Modules">
    **What developers get**

    * Built-in modules for detection, classification, segmentation, pose, audio, and super-resolution.
    * Custom module support without SDK recompilation.
    * Runtime module swap — change post-processing logic by updating a `.so` file on the device.

    **How QIM SDK delivers it**

    * [`qtimlpostprocess`](plugin-reference/qtimlpostprocess) loads `libml-postprocess-<name>.so` at runtime via `dlopen`.
    * Each module implements `Caps()`, `Configure()`, and `Process()` from the `IModule` interface.
    * Modules are deployed to `/usr/lib/gstreamer-1.0/ml/modules/` — no pipeline changes required after deployment.

    **Developer takeaway**
    Swap or extend post-processing without touching the pipeline definition or recompiling the application.
  </Accordion>

  <Accordion title="ISP Camera Integration">
    **What developers get**

    * Hardware-demosaiced, ISP-processed frames delivered directly to downstream AI and multimedia stages.
    * Multi-camera support (primary + secondary sensor).
    * Zero-copy from ISP to GPU/NPU via GBM/DMA-buf.

    **How QIM SDK delivers it**

    * `qticamsrc` exposes a GStreamer source interface over the Qualcomm camera service.
    * Outputs `video/x-raw,format=NV12` — directly consumable by [`qtimlvconverter`](plugin-reference/qtimlvconverter), [`qtivtransform`](plugin-reference/qtivtransform), and encode plugins.
    * Camera properties (resolution, framerate, camera index) are set via standard GStreamer element properties.

    **Developer takeaway**
    A single pipeline element replaces the entire camera driver integration layer — no V4L2 or camera HAL code to manage.
  </Accordion>

  <Accordion title="GStreamer-Based, Industry-Standard Pipelines">
    **What developers get**

    * Familiar pipeline construction.
    * Easier integration with upstream plugins.
    * Better portability of application logic.

    **How QIM SDK delivers it**

    * Follows standard GStreamer plugin architecture and negotiation rules.
    * Uses caps negotiation for format compatibility.
    * Uses allocation queries for memory negotiation and zero-copy paths.
    * Reuses standard image formats such as `NV12`, `RGBA`, and `I420`.
    * Reuses standard metadata where possible and adds AI-specific metadata only when needed.

    **Developer takeaway**
    If your team already knows GStreamer, QIM SDK fits into the same workflow with Qualcomm-optimized building blocks.
  </Accordion>

  <Accordion title="Flexible AI Model Support">
    **What developers get**

    * Freedom to choose runtime per use case.
    * Support for common edge AI workloads.
    * A path from prototype to optimized deployment.

    **How QIM SDK delivers it**

    * Supports `TFLite`, `SNPE`, and `QNN` runtime paths.
    * Supports use cases such as detection, classification, segmentation, pose, super resolution, and audio AI.
    * Supports chained and parallel model execution in the same media flow.
    * Preserves model outputs as metadata so downstream stages can reuse them.

    **Developer takeaway**
    You can keep one pipeline architecture while changing the model, runtime, or accelerator strategy based on product needs.
  </Accordion>
</AccordionGroup>

## Developer Notes

<AccordionGroup>
  <Accordion title="Why zero-copy matters">
    Zero-copy pipelines keep the same frame buffer moving across decode, preprocess, inference, overlay, and encode stages. This reduces memory traffic, CPU overhead, and end-to-end latency — especially important for high-resolution and multi-stream workloads.
  </Accordion>

  <Accordion title="Why the GPU preprocessing path matters">
    A typical AI pipeline must resize, color-convert, and normalize frames before inference. QIM SDK offloads these steps through GPU-accelerated plugins instead of doing them in generic CPU-only stages, which helps preserve frame rate and lowers CPU utilization.
  </Accordion>

  <Accordion title="How to read the examples">
    Each example shows the same operation built with upstream GStreamer CPU elements and then with QIM SDK hardware-accelerated plugins. Timing numbers are measured on IQ9 with a USB camera at 1080p input.
  </Accordion>
</AccordionGroup>


# QIM SDK Tutorials
Source: https://imsdkdocs.qualcomm.com/blogs/blog-index



<hr />

<div>
  <span>
    Qualcomm QIM SDK — AI pipeline tutorials, guides, and deep dives.
  </span>

  <div>
    <span>🔍</span>
    <a href="#posts">All posts →</a>
    <a href="#posts">Guides →</a>
  </div>
</div>

<hr />

<div>
  <a href="montioring-ppe-compliance-with-ai-powered">
    <div>
      <div>
        <div>
          QIM SDK · Qualcomm
        </div>
      </div>

      <div>
        <div>Computer Vision</div>
        <div>Monitoring PPE Compliance with AI‑Powered Computer Vision</div>
        <p>Build a real-time PPE detection pipeline using Qualcomm QIM SDK with daisy-chained ML models for person detection and protective equipment recognition — running entirely on-device with hardware-accelerated inference via Qualcomm HTP.</p>

        <div>
          <span>QIM SDK Team · May 12, 2026</span>
          <span>Read more →</span>
        </div>
      </div>
    </div>
  </a>

  <a href="smart-hand-gesture-recognition">
    <div>
      <div>
        <video />

        <div>
          QIM SDK · Qualcomm
        </div>
      </div>

      <div>
        <div>Gesture Recognition</div>
        <div>Building a Real-Time Hand Gesture Recognition AI Pipeline Using the QIM SDK</div>
        <p>A four-stage sequential AI pipeline — palm detection, hand landmark estimation, gesture embedding, and gesture classification — built with Qualcomm QIM SDK for real-time edge deployment.</p>

        <div>
          <span>QIM SDK Team · Jun 10, 2026</span>
          <span>Read more →</span>
        </div>
      </div>
    </div>
  </a>

  <a href="multi-stream-ip-camera">
    <div>
      <div>
        <video />

        <div>
          QIM SDK · Qualcomm
        </div>
      </div>

      <div>
        <div>Computer Vision</div>
        <div>Multi Stream IP Camera Application</div>
        <p>A real-time object detection pipeline built with QIM SDK using a YOLOv8 TensorFlow Lite model, supporting USB cameras, ISP cameras, RTSP streams, and video files — with RTSP/WebRTC output.</p>

        <div>
          <span>QIM SDK Team · Apr 10, 2026</span>
          <span>Read more →</span>
        </div>
      </div>
    </div>
  </a>

  <a href="security-video-wall">
    <div>
      <div>
        <video />

        <div>
          QIM SDK · Qualcomm
        </div>
      </div>

      <div>
        <div>Multi-Stream AI</div>
        <div>Security Video Wall — 31-Stream Parallel Object Detection</div>
        <p>A scalable AI pipeline processing up to 31 input video streams in parallel with YOLOv8 object detection, compositing all streams into a single output and streaming over RTSP/WebRTC.</p>

        <div>
          <span>QIM SDK Team · Apr 5, 2026</span>
          <span>Read more →</span>
        </div>
      </div>
    </div>
  </a>

  <a href="restricted-zone-alarm">
    <div>
      <div>
        <video />

        <div>
          QIM SDK · Qualcomm
        </div>
      </div>

      <div>
        <div>Safety & Security</div>
        <div>Restricted Zone Alarm — Real-Time Zone Intrusion Detection</div>
        <p>Real-time pipeline that identifies when a person steps into a predefined restricted zone using QIM SDK foot detection — with visual alerts, bounding box status indicators, and RTSP output.</p>

        <div>
          <span>QIM SDK Team · Mar 28, 2026</span>
          <span>Read more →</span>
        </div>
      </div>
    </div>
  </a>

  <a href="touchless-wrist-steering-car-game">
    <div>
      <div>
        <video alt="Smart Board Application" />

        <div>
          QIM SDK · Qualcomm
        </div>
      </div>

      <div>
        <div>Gesture Recognition</div>
        <div>Building a Wrist-Controlled Car Game Using QIM SDK</div>
        <p>A split-screen touchless driving game controlled entirely by wrist movement — built on a two-stage palm detection and hand landmark pipeline running on Qualcomm QIM SDK with QNN HTP acceleration.</p>

        <div>
          <span>QIM SDK Team · Aug 3, 2026</span>
          <span>Read more →</span>
        </div>
      </div>
    </div>
  </a>

  <a href="smart-board-application">
    <div>
      <div>
        <video alt="Smart Board Application" />

        <div>
          QIM SDK · Qualcomm
        </div>
      </div>

      <div>
        <div>Gesture Recognition</div>
        <div>Touchless Smart Board — Transforming Digital Learning with QIM SDK</div>
        <p>A touchless virtual whiteboard controlled entirely by hand gestures — built on a two-stage palm detection and hand landmark pipeline running on Qualcomm QIM SDK with QNN HTP acceleration.</p>

        <div>
          <span>QIM SDK Team · Aug 3, 2026</span>
          <span>Read more →</span>
        </div>
      </div>
    </div>
  </a>

  <a href="product-counting-application">
    <div>
      <div>
        <video />

        <div>
          QIM SDK · Qualcomm
        </div>
      </div>

      <div>
        <div>Computer Vision</div>
        <div>Edge AI for Smart Manufacturing — Real-Time Bottle Counting Using Qualcomm QIM SDK</div>
        <p>A YOLOv8-based object detection pipeline that counts products crossing a center region of interest in real time — built with Qualcomm QIM SDK and QNN HTP acceleration.</p>

        <div>
          <span>QIM SDK Team · Aug 3, 2026</span>
          <span>Read more →</span>
        </div>
      </div>
    </div>
  </a>
</div>


# Monitoring PPE Compliance with AI-Powered Computer Vision
Source: https://imsdkdocs.qualcomm.com/blogs/montioring-ppe-compliance-with-ai-powered

Build a real-time PPE detection pipeline using Qualcomm QIM SDK with daisy-chained ML models for person detection and protective equipment recognition.

<div>
  <div>
    QIM SDK · Qualcomm
  </div>
</div>

<div>
  <div>
    Computer Vision
  </div>

  <p>
    Build a real-time PPE detection pipeline using Qualcomm QIM SDK with daisy-chained ML models
    for person detection and protective equipment recognition — running entirely on-device with
    hardware-accelerated inference via Qualcomm HTP.
  </p>

  <div>
    <span>QIM SDK Team</span>
    <span>·</span>
    <span>May 12, 2026</span>
    <span>·</span>
    <a href="/blogs">← All posts</a>
  </div>
</div>

<hr />

## Introduction

Ensuring worker safety in industrial and construction environments demands continuous, real-time monitoring at scale — a challenge that traditional manual approaches cannot meet efficiently. The QIM SDK addresses this directly by delivering a hardware-accelerated, end-to-end AI pipeline that automates PPE compliance monitoring at the edge, with minimal operational overhead.

Leveraging Qualcomm's dedicated hardware accelerators through the SDK's GStreamer plugin architecture, compute-intensive tasks — including video decoding, frame preparation (resizing, color format conversion, and pixel normalization), multi-stage AI inference, and encoding — are offloaded entirely from the CPU to purpose-built hardware blocks. This enables low-latency, power-efficient AI execution directly on Qualcomm edge devices, making continuous, real-world safety monitoring both practical and scalable.

At the core of this use case is a **multi-stage daisy-chain AI pipeline**, where models operate sequentially and build upon each other's outputs. A person detection model first identifies individuals within the full frame; a second model then performs per-person PPE compliance analysis — detecting helmets, vests, gloves, and masks — using dynamically cropped regions derived from the initial detections. This approach enables fine-grained, per-person analysis with high accuracy, while keeping compute focused on regions of interest rather than the full frame.

The metadata produced by this pipeline is **hierarchically structured**: base detections (persons) from the first model serve as parent entries, with PPE detection results from the second model attached as child metadata linked to each individual. This context-aware structure ensures that every detected safety item is explicitly associated with a specific person — enabling precise visualization, tracking, and downstream analytics.

The QIM SDK further accelerates visualization through **hardware-accelerated overlay rendering and blitting**, where bounding boxes, labels, and compliance indicators are composited directly onto video frames using optimized hardware operations — delivering smooth, real-time visualization without additional CPU load or pipeline latency.

Beyond visualization, the SDK provides native support for **AI metadata streaming**, synchronizing structured inference results with the video stream and transmitting them alongside the media pipeline. This transforms raw video into actionable, structured data — enabling external monitoring and alerting systems to consume real-time PPE compliance insights without re-running inference.

Integration with **Qualcomm AI Hub** further accelerates development by providing access to optimized, production-ready models for both person detection and PPE analysis, significantly reducing the effort required to move from prototype to production deployment.

The pipeline supports multiple input sources — USB, RTSP, ISP camera, and file-based video — and delivers results through real-time on-screen visualization, RTSP streaming, or WebRTC, with inference metadata transmitted in parallel. The result is a scalable, efficient edge AI system that transforms raw video into actionable safety intelligence — empowering organizations to proactively enforce compliance and mitigate risk in real time.

<Info>
  Check application source code on GitHub: [`gst-ppe-detection`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ppe-detection)
</Info>

## Use Case Overview

<Steps>
  <Step title="Video Input">
    The pipeline accepts continuous video input from multiple source types — RTSP streams, ISP camera feeds, USB cameras, and file-based video.
  </Step>

  <Step title="Person Detection">
    Each frame is submitted to a person detection model that identifies individuals and their locations within the scene.
  </Step>

  <Step title="PPE Detection">
    For each detected person, a dedicated PPE detection model analyzes the dynamically cropped region to identify the presence or absence of safety equipment — including helmets, vests, gloves, and masks.
  </Step>

  <Step title="Metadata Generation">
    Detection results are attached to the video stream as hierarchically structured metadata, explicitly linking each PPE detection to its corresponding individual.
  </Step>

  <Step title="Visualization">
    Bounding boxes and labels are rendered directly onto video frames in real time using hardware-accelerated overlay, providing intuitive interpretation of detection results.
  </Step>

  <Step title="Metadata Synchronization">
    [`qtimetamux`](../plugin-reference/qtimetamux) synchronizes all inference results with the original video frames, maintaining per-frame consistency throughout the pipeline.
  </Step>

  <Step title="Output">
    The annotated stream is delivered via RTSP or WebRTC. Structured PPE compliance data is transmitted in parallel as a JSON metadata stream — enabling seamless integration with external monitoring, alerting, and analytics systems.
  </Step>
</Steps>

## Pipeline diagram

<img alt="" />

## Elements used in pipeline

| Element | Description |
| - | - |
| `source` | Accepts input from an RTSP camera, ISP camera, USB camera, or a local file. |
| `tee` | Splits the incoming stream into multiple parallel branches for simultaneous downstream processing. |
| [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | Prepares video frames for inference by performing resizing, YUV-to-RGB color space conversion, and pixel normalization to match the model's input requirements. |
| [`qtimltflite`](../plugin-reference/qtimltflite) | Executes the TFLite inference model for person/feet detection on each incoming frame. |
| [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | Decodes raw output tensors into structured bounding boxes and labels. Post-processing logic is implemented as a dynamically loaded module, enabling model-specific strategies to be swapped without pipeline changes. |
| [`qtimetamux`](../plugin-reference/qtimetamux) | Synchronizes inference results with the original video stream and attaches them as per-frame structured metadata. |
| [`qtivoverlay`](../plugin-reference/qtivoverlay) | Renders bounding boxes, labels, and the restricted zone polygon directly onto video frames for real-time visual feedback. |
| [`qtimetaparser`](../plugin-reference/qtimetaparser) | Serializes per-frame ML metadata into JSON format for integration with external monitoring and analytics systems. |
| [`v4l2h264enc`](../plugin-reference/v4l2h264enc) / `h264parse` | Encodes the processed video stream into H.264 format for downstream transmission or storage. |
| `sink` | Streams the encoded video and associated metadata over RTSP or WebRTC via the `rtspbin` or `webrtcbin` plugins respectively, enabling remote clients to consume results in real time. |
| [`waylandsink`](../plugin-reference/waylandsink) | Renders the annotated video stream to a local Wayland display. |

## How it works

The PPE detection pipeline implements a **two-stage daisy-chain architecture**, where two sequential AI models operate in tandem — the output of the first model directly driving the execution of the second.

* **Stage 1 — Person Detection:** The first AI model processes the full video frame and produces bounding boxes identifying the location of each individual in the scene.
* **Crop Generation:** Since the second PPE detection model requires image crops — not bounding boxes — as input, a second instance of [`qtimlvconverter`](../plugin-reference/qtimlvconverter) operates in crop generation mode, receiving the bounding boxes produced by the first model and dynamically generating a cropped image region from the original frame for each detected person.
* **Stage 2 — PPE Detection:** The PPE detection model is invoked once per detected person, analyzing each cropped region independently to identify the presence or absence of safety equipment — including helmets, vests, gloves, and masks.
* **Metadata Re-attachment:** A second [`qtimetamux`](../plugin-reference/qtimetamux) instance re-attaches the PPE detection results to the original video stream, ensuring all detections are synchronized with the corresponding frame and person.
* **Hierarchical Metadata:** To preserve logical relationships across both model stages, the pipeline employs a **hierarchical metadata model** based on unique IDs and parent IDs — explicitly linking each PPE detection to its corresponding individual, enabling accurate per-person visualization, tracking, and downstream analytics.

<img alt="" />

## Run application on device

### Setup Requirements

#### Hardware

<img alt="" />

| Component | Description |
| - | - |
| **Edge Device** | RB3 Gen 2, IQ8, or IQ9 — Primary processing unit for AI inference and video composition. |
| **Camera Source** | IP/RTSP camera, ISP (on-device) camera, or USB camera. A local file source may be substituted if no physical camera is available. |
| **HDMI Display Monitor** | Connected to the edge device for rendering and visualizing pipeline output. |
| **PoE Switch** | Powers IP/RTSP cameras and provides network connectivity over a single Ethernet cable per camera. (Required for IP/RTSP camera setups only.) |
| **Local Network** | Ensures the edge device, RTSP camera, and host machine are reachable on the same network. (Required when using RTSP camera input or streaming results via RTSP or WebRTC.) |

#### Software

**Flash your Qualcomm Edge device** by following the device setup and flashing instructions [here](../installation)

**Once your device is ready**, follow the instructions below to set up the PPE AI Pipeline:

##### AI Model and config files

| File | Download | Save as |
| - | - | - |
| Person Foot Detection model | [Qualcomm AI Hub — FootTrackNet](https://aihub.qualcomm.com/models/foot_track_net) | `foot_track_net_quantized.tflite` |
| PPE Detection model | [Qualcomm AI Hub — GearGuardNet](https://aihub.qualcomm.com/models/gear_guard_net) | `gear_guard_net.tflite` |
| Foot track labels | <a href="../labels/foot_track_net.json">foot\_track\_net.json</a> | `foot_track_net.json` |
| Foot track net settings | <a href="../labels/foot_track_net_settings.json">foot\_track\_net\_settings.json</a> | `foot_track_net_settings.json` |
| Gear guard labels | <a href="../labels/gear_guard_net.json">gear\_guard\_net.json</a> | `gear_guard_net.json` |
| PPE sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/ppe_sample.mp4">Input video</a> | `ppe_sample.mp4` |

**Copy files to device**

<CodeGroup>
  ```bash SCP (SSH) theme={null}
  # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

  ssh <user>@<device-ip> "mkdir -p ~/{models,labels,media,media/output}"
  scp foot_track_net_quantized.tflite   <user>@<device-ip>:~/models/
  scp gear_guard_net.tflite             <user>@<device-ip>:~/models/
  scp foot_track_net.json               <user>@<device-ip>:~/labels/
  scp foot_track_net_settings.json      <user>@<device-ip>:~/labels/
  scp gear_guard_net.json               <user>@<device-ip>:~/labels/
  scp ppe_sample.mp4                     <user>@<device-ip>:~/media/
  ```
</CodeGroup>

**Connect to device**

```bash theme={null}
ssh <user>@<device-ip>
```

**Run the PPE Application**

<Note>
  A display must be connected to the device. If no display is available, use the `--no-display` flag to run in headless mode.
</Note>

Use the following base path for model and label files based on your OS:

<AccordionGroup>
  <Accordion title="Display only">
    ```bash theme={null}
    gst-ppe-detection \
      --input-type=file \
      --input-config=$HOME/media/ppe_sample.mp4
    ```
  </Accordion>

  <Accordion title="WebRTC output">
    ```bash theme={null}
    gst-ppe-detection \
      --input-type=file \
      --input-config=$HOME/media/ppe_sample.mp4 \
      --output-type=webrtc \
      --output-config=wss://webrtc.nirbheek.in:8443 \
      --webrtc-id=1010
    ```
  </Accordion>

  <Accordion title="RTSP output">
    ```bash theme={null}
    gst-ppe-detection \
      --input-type=file \
      --input-config=$HOME/media/ppe_sample.mp4 \
      --output-type=rtsp \
      --output-config=8900
    ```
  </Accordion>
</AccordionGroup>

> **Note:** This example uses an offline video file as input. To use an IP/RTSP camera or USB camera instead, update the `--input-type` argument accordingly — refer to the **Command-Line Options** section below for details.

It produces two key output results: an AI-annotated video stream and a JSON metadata stream. To visualize these results, refer to the **Host-Side Visualization** section below. To configure alternative input sources or output destinations, refer to the Command-Line Options section.

## Visualize the Results - Host-Side Visualization (Windows + WSL)

This section describes how to run the visualization client on a Windows host machine using **WSL (Windows Subsystem for Linux)**. The client renders the live video stream alongside a real-time AI metadata panel.

📥 The visualization client script can be downloaded here: <a href="../labels/rtsp_webrtc_client.zip">rtsp\_webrtc\_client.zip</a>

It displays:

* **Left panel** — Live video stream. (Output Video stream with AI overlays)
* **Right panel** — Real-time AI metadata (JSON): object detections, bounding boxes, and confidence scores.

**Step 1 — Install WSL and Ubuntu**

If WSL is not already installed, run the following from a Windows terminal:

```bash theme={null}
wsl --install Ubuntu-24.04
```

Once installed, open the Ubuntu terminal and update the system:

```bash theme={null}
sudo apt update && sudo apt upgrade -y
```

**Step 2 — Install System Dependencies**

The visualization script requires GStreamer and Python GObject Introspection (GI) bindings. Install all required packages with:

```bash theme={null}
sudo apt install -y \
  python3 python3-pip python3-gi python3-gi-cairo \
  gir1.2-gstreamer-1.0 \
  gir1.2-gst-plugins-base-1.0 \
  gir1.2-gst-plugins-bad-1.0 \
  gstreamer1.0-tools \
  gstreamer1.0-plugins-base \
  gstreamer1.0-plugins-good \
  gstreamer1.0-plugins-bad \
  gstreamer1.0-plugins-ugly \
  gstreamer1.0-libav \
  python3-websocket \
  libnice10 \
  libnice-dev \
  gstreamer1.0-nice
```

**Step 3 — Run the Visualization Client Script**

Navigate to the directory containing the script and run:

<AccordionGroup>
  <Accordion title="RTSP">
    ```bash theme={null}
    python3 rtsp_webrtc_client.py rtsp://<DEVICE_IP>:8900/live
    ```
  </Accordion>

  <Accordion title="WebRTC">
    ```bash theme={null}
    python3 rtsp_webrtc_client.py --source webrtc --signalling-server wss://webrtc.nirbheek.in:8443 --peer-id 1010
    ```
  </Accordion>
</AccordionGroup>

**Step 4 — Expected Output**

Once the client connects, the UI will display:

| Panel Content | Description |
| - | - |
| Left | Real-time decoded video stream |
| Right | Live AI metadata panel — object detections, bounding boxes, and confidence scores |

After following the steps, the video and metadata streams should be up and running.

<img alt="" />

The pipeline generates structured JSON metadata in the following format:

```json theme={null}
{
  "object_detection": [
    {
      "label": "person",
      "confidence": 76.62,
      "color": 16711935,
      "rectangle": {
        "x": 0.58,
        "y": 0.05,
        "width": 0.28,
        "height": 0.90
      },
      "landmarks": {
        "nose": {
          "x": 0.71,
          "y": 0.21
        }
      },
      "object_detection": [
        {
          "label": "helmet",
          "confidence": 97.79,
          "color": 65535,
          "rectangle": {
            "x": 0.64,
            "y": 0.05,
            "width": 0.13,
            "height": 0.29
          }
        }
      ]
    }
  ],
  "parameters": {
    "timestamp": "11341424121"
  }
}
```

## Command-Line Options

<AccordionGroup>
  <Accordion title="--input-type">
    Selects the video input source for the pipeline.

    | Value | Description |
    | - | - |
    | `usb` | USB camera. Requires `--input-config=/dev/video0`. |
    | `isp` | Built-in ISP (on-device) camera. Optionally specify a camera ID via `--input-config=0`. |
    | `rtsp` | External IP/RTSP camera or stream. Requires `--input-config=rtsp://...`. |
    | `file` | Local H.264-encoded video file. Requires `--input-config=/path/to/ppe_sample.mp4`. |
  </Accordion>

  <Accordion title="--input-config">
    Specifies the input source configuration corresponding to the selected `--input-type`.

    | Input Type | Value |
    | - | - |
    | USB | `/dev/videoX` |
    | ISP | `<camera ID>` |
    | RTSP | `rtsp://<ip-or-url>` |
    | File | `/path/to/ppe_sample.mp4` |
  </Accordion>

  <Accordion title="--output-type">
    Defines how the processed output video stream is delivered.

    | Value | Description |
    | - | - |
    | `none` | No video output (headless mode). Display output is controlled separately via `--no-display`. |
    | `file` | Saves the encoded output video stream to a file. Requires `--output-config`. |
    | `rtsp` | Streams the output video over RTSP. Requires `--output-config=<port>`. Access at `rtsp://<device-ip>:<port>/live`. |
    | `webrtc` | Streams the output video over WebRTC. Requires `--output-config=ws://...`. |
  </Accordion>

  <Accordion title="--output-config">
    Specifies the output destination configuration corresponding to the selected `--output-type`.

    | Output Type | Value |
    | - | - |
    | File | `/path/to/output.mp4` |
    | RTSP | `<port>` |
    | WebRTC | `ws://<signalling-server>:<port>` |
  </Accordion>

  <Accordion title="--model-base-path">
    Root directory where the application looks for AI model, label, and configuration files. Assets are resolved automatically:

    | Asset Type | Resolved Path |
    | - | - |
    | Model files (`*.tflite`) | `<base-path>/models/<model_file>` |
    | Label/settings files (`*.json`) | `<base-path>/labels/<labels_file>` |

    ```bash theme={null}
    --model-base-path=~      # QLI: /root, Ubuntu: /home/ubuntu
    ```
  </Accordion>

  <Accordion title="--no-display">
    Disables local on-screen rendering of the output video stream. Recommended for:

    * Headless deployments
    * Remote streaming setups (RTSP/WebRTC)
    * Performance optimization where display overhead is undesirable
  </Accordion>

  <Accordion title="--width / --height / --framerate">
    Sets the raw input video resolution and frame rate. Applicable only to ISP and USB inputs.

    ```bash theme={null}
    --width=1920 --height=1080 --framerate=30
    ```
  </Accordion>

  <Accordion title="--webrtc-id">
    Specifies the local WebRTC signaling client ID used for peer connection setup with the signaling server.

    ```bash theme={null}
    --webrtc-id=1010
    ```
  </Accordion>
</AccordionGroup>

## Implementation Deep-Dive

<AccordionGroup>
  <Accordion title="1. Application Configuration and Runtime Context">
    The application cleanly separates user configuration from runtime state — organizing command-line parameters, GStreamer objects, dynamic pad tracking, WebRTC signaling, and shutdown handling into predictable, well-defined locations.

    ```c theme={null}
    typedef struct GstAppConfig {
      gchar *input_type;
      gchar *input_location;
      gchar *input_format;
      gchar *output_type;
      gchar *output_location;
      gboolean no_display;
      gint width, height, framerate, rtsp_latency_ms, webrtc_id;
    } GstAppConfig;

    typedef struct GstAppContext {
      GstAppConfig config;
      GstElement *pipeline;
      GMainLoop  *mloop;
      GstElement *webrtc;
      gboolean    is_shutting_down;
    } GstAppContext;
    ```
  </Accordion>

  <Accordion title="2. Pipeline Assembly">
    The pipeline is composed of three branches: common input, common output, and application-specific processing. Construction order is deliberate — input first, output second, processing last.

    ```c theme={null}
    static gboolean gst_app_create_pipe (GstAppContext *appctx) {
      GstElement *input_tail = NULL, *output_head = NULL, *meta_head = NULL;
      appctx->pipeline = gst_pipeline_new ("gst-ppe-detection");

      if (!gst_app_create_input_pipe  (appctx, &input_tail))  return FALSE;
      if (!gst_app_create_output_pipe (appctx, &output_head, &meta_head)) return FALSE;
      if (!gst_app_create_user_pipe   (appctx, input_tail, output_head, meta_head)) return FALSE;
      return TRUE;
    }
    ```
  </Accordion>

  <Accordion title="3. Two-Stage Inference Element Creation">
    Dedicated `qtimlvconverter`, `qtimltflite`, `qtimlpostprocess`, and `qtimetamux` elements are allocated for each inference stage.

    ```c theme={null}
    qtimlvconverter_stage1 = gst_app_make_element ("qtimlvconverter", "qtimlvconverter_stage1");
    qtimlvconverter_stage2 = gst_app_make_element ("qtimlvconverter", "qtimlvconverter_stage2");
    qtimltflite_stage1     = gst_app_make_element ("qtimltflite",     "qtimltflite_stage1");
    qtimltflite_stage2     = gst_app_make_element ("qtimltflite",     "qtimltflite_stage2");
    qtimlpostprocess_stage1 = gst_app_make_element ("qtimlpostprocess", "qtimlpostprocess_stage1");
    qtimlpostprocess_stage2 = gst_app_make_element ("qtimlpostprocess", "qtimlpostprocess_stage2");
    qtimetamux_stage1      = gst_app_make_element ("qtimetamux",      "qtimetamux_stage1");
    qtimetamux_stage2      = gst_app_make_element ("qtimetamux",      "qtimetamux_stage2");
    qtivoverlay            = gst_app_make_element ("qtivoverlay",     "qtivoverlay");
    qtimlmetaparser        = gst_app_make_element ("qtimlmetaparser", "qtimlmetaparser");
    ```
  </Accordion>

  <Accordion title="4. ROI Batching and Accelerated Inference Configuration">
    Stage 2 uses cumulative ROI batching. Both models execute via the QNN external delegate. Each post-processing stage is configured with dedicated labels and settings.

    ```c theme={null}
    gst_element_set_enum_property (qtimlvconverter_stage2, "mode", "roi-batch-cumulative");

    g_object_set (G_OBJECT (qtimlpostprocess_stage1),
      "results", 10, "labels", STAGE1_LABELS_PATH,
      "bbox-stabilization", TRUE, "settings", STAGE1_SETTINGS_PATH, NULL);
    gst_element_set_enum_property (qtimlpostprocess_stage1, "module", "qpd");

    g_object_set (G_OBJECT (qtimlpostprocess_stage2),
      "results", 10, "labels", STAGE2_LABELS_PATH,
      "bbox-stabilization", TRUE, NULL);
    gst_element_set_enum_property (qtimlpostprocess_stage2, "module", "yolov8");

    delegate_options = gst_structure_from_string ("QNNExternalDelegate,backend_type=htp;", NULL);
    g_object_set (G_OBJECT (qtimltflite_stage1),
      "external-delegate-path", "libQnnTFLiteDelegate.so",
      "external-delegate-options", delegate_options,
      "model", STAGE1_MODEL_PATH, NULL);
    gst_element_set_enum_property (qtimltflite_stage1, "delegate", "external");
    g_object_set (G_OBJECT (qtimltflite_stage2),
      "external-delegate-path", "libQnnTFLiteDelegate.so",
      "external-delegate-options", delegate_options,
      "model", STAGE2_MODEL_PATH, NULL);
    gst_element_set_enum_property (qtimltflite_stage2, "delegate", "external");
    gst_structure_free (delegate_options);
    ```
  </Accordion>

  <Accordion title="5. Linking Stage 1 into Stage 2">
    Stage 1 attaches person detection metadata to the stream; Stage 2 consumes the resulting ROIs, runs PPE inference, overlays results, and forwards the annotated stream to the output branch.

    ```c theme={null}
    gst_element_link_many (input_tail, tee[0], queue[0], qtimetamux_stage1, NULL);
    gst_element_link_many (tee[0], queue[1], qtimlvconverter_stage1, queue[2],
      qtimltflite_stage1, queue[3], qtimlpostprocess_stage1, postprocess_caps_stage1,
      queue[4], qtimetamux_stage1, NULL);
    gst_element_link_many (qtimetamux_stage1, queue[5], tee[1], NULL);
    gst_element_link_many (tee[1], queue[6], qtimetamux_stage2, NULL);
    gst_element_link_many (tee[1], queue[7], qtimlvconverter_stage2, queue[8],
      qtimltflite_stage2, queue[9], qtimlpostprocess_stage2, postprocess_caps_stage2,
      queue[10], qtimetamux_stage2, queue[11], qtivoverlay, queue[12], tee[2], NULL);
    if (output_head != NULL)
      gst_element_link_many (tee[2], queue[13], output_head, NULL);
    if (meta_head != NULL)
      gst_element_link_many (tee[2], queue[14], qtimlmetaparser, meta_head, NULL);
    ```
  </Accordion>

  <Accordion title="6. Metadata over RTSP / WebRTC">
    Metadata is handled through a dedicated branch and optionally exported.

    * **RTSP** — metadata linked via sink pad on [`qtirtspbin`](../plugin-reference/qtirtspbin)
    * **WebRTC** — metadata sent via a dedicated data channel

    ```c theme={null}
    if (meta_head != NULL)
      gst_element_link_many (tee[1], queue[7], qtimlmetaparser, meta_head, NULL);
    ```

    **WebRTC metadata callback:**

    ```c theme={null}
    static GstFlowReturn
    gst_app_webrtc_meta_new_sample_cb (GstElement *appsink, gpointer userdata) {
      GstSample *sample = NULL;
      GstBuffer *buffer = NULL;
      g_signal_emit_by_name (appsink, "pull-sample", &sample);
      buffer = gst_sample_get_buffer (sample);
      gst_buffer_map (buffer, &mapinfo, GST_MAP_READ);
      metadata = g_strndup ((const gchar *) mapinfo.data, mapinfo.size);
      // send via WebRTC data channel
    }
    ```

    **Key WebRTC signaling callbacks:**

    | Callback | Responsibility |
    | - | - |
    | `on_offer_created` | Constructs and sends the SDP offer to the remote peer |
    | `on_ice_candidate` | Transmits ICE candidates to the signaling server |
    | `on_ws_message` | Handles incoming signaling messages from the WebSocket |
  </Accordion>
</AccordionGroup>

## Build the Application

* **Source code:** [gst-ppe-detection](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ppe-detection)
* **Build instructions:** [Steps to build custom application](../advanced/yocto-build#steps-to-build-custom-application)

## Conclusion

The QIM SDK's modular, plugin-based architecture enables developers to rapidly build scalable multi-stage AI video analytics pipelines without sacrificing flexibility. By attaching each model's output directly to the corresponding video frame as structured metadata, the SDK preserves inference context across pipeline stages and enables accurate downstream processing. Results can be delivered through on-screen overlays, network streams, or as independently transmitted metadata — giving developers full control over how and where AI-driven insights are consumed.


# Multi Stream IP Camera application
Source: https://imsdkdocs.qualcomm.com/blogs/multi-stream-ip-camera

A real-time object detection pipeline built with QIM SDK using a YOLOv8 TensorFlow Lite model, supporting multiple input sources and RTSP/WebRTC output.

<div>
  <video />

  <div>
    QIM SDK · Qualcomm
  </div>
</div>

<div>
  <div>
    Computer Vision
  </div>

  <p>
    A real-time object detection pipeline built with QIM SDK using a YOLOv8 TensorFlow Lite model,
    supporting USB cameras, ISP cameras, RTSP streams, and video files — with RTSP/WebRTC output.
  </p>

  <div>
    <span>QIM SDK Team</span>
    <span>·</span>
    <span>Apr 10, 2026</span>
    <span>·</span>
    <a href="/blogs">← All posts</a>
  </div>
</div>

<hr />

## Introduction

Managing video feeds from multiple IP cameras simultaneously is a core requirement in modern surveillance, retail analytics, smart city, and industrial monitoring deployments. Processing each stream independently on a CPU is costly and difficult to scale — but with the QIM SDK, multiple concurrent RTSP streams can be decoded, processed through AI inference, and rendered or streamed out in real time, all using Qualcomm's dedicated hardware accelerators.

The QIM SDK's GStreamer-based plugin architecture offloads compute-intensive tasks — including multi-stream hardware H.264/H.265 decoding, frame preparation, AI inference via the Neural Processing Unit (NPU), and re-encoding — entirely from the CPU to purpose-built hardware blocks. This enables low-latency, power-efficient processing of multiple simultaneous IP camera feeds directly on Qualcomm edge devices.

At the core of this use case is a **parallel multi-stream pipeline**, where each incoming RTSP stream is independently decoded and processed through its own AI inference chain. Results from all streams are then composited into a unified output — either for display, recording, or network streaming. The pipeline supports flexible input configurations, from a single RTSP source up to multiple concurrent streams, making it suitable for scalable, real-world deployments.

<Info>
  Check application source code on GitHub: [`gst-ip-camera`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ip-camera)
</Info>

## Use Case Overview

<Steps>
  <Step title="Video Input">
    Accepts multiple concurrent RTSP streams from IP cameras, each decoded independently using hardware-accelerated H.264/H.265 decoding.
  </Step>

  <Step title="AI Inference">
    Each decoded stream is passed through a dedicated AI inference chain — including preprocessing (`qtimlvconverter`), model execution (`qtimltflite`), and post-processing (`qtimlpostprocess`) — for per-stream object detection.
  </Step>

  <Step title="Metadata Attachment">
    Inference results are attached to each stream as structured per-frame metadata via `qtimetamux`, maintaining stream–detection correspondence.
  </Step>

  <Step title="Visualization">
    Bounding boxes and labels are rendered onto each stream's frames via `qtivoverlay` for real-time visual feedback.
  </Step>

  <Step title="Composition">
    Annotated frames from all streams are composited into a unified output layout using `qtivcomposer`.
  </Step>

  <Step title="Output">
    The composited stream is delivered to a local Wayland display, encoded and saved to file, or streamed over RTSP/WebRTC.
  </Step>
</Steps>

## Pipeline diagram

<img alt="Pipeline Diagram" />

## Elements used in pipeline

| Element | Description |
| - | - |
| `rtspsrc` | Receives an RTSP stream from an IP camera over the network. |
| `rtph264depay / h264parse` | Depayloads and parses the incoming RTP/H.264 bitstream. |
| [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | Hardware-accelerated H.264 video decoder. |
| `tee` | Splits each decoded stream into parallel branches for simultaneous AI inference and output. |
| [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | Prepares video frames for inference — handles resizing, YUV-to-RGB color space conversion, and pixel normalization. |
| [`qtimltflite`](../plugin-reference/qtimltflite) | Executes the TFLite inference model on each frame using the Qualcomm NPU. |
| [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | Decodes raw output tensors into structured bounding boxes and class labels via a dynamically loaded module. |
| [`qtimetamux`](../plugin-reference/qtimetamux) | Synchronizes inference results with the original video stream and attaches them as per-frame structured metadata. |
| [`qtivoverlay`](../plugin-reference/qtivoverlay) | Renders bounding boxes and labels directly onto video frames for real-time visualization. |
| [`qtivcomposer`](../plugin-reference/qtivcomposer) | Composes multiple annotated video streams into a single unified output frame. |
| [`waylandsink`](../plugin-reference/waylandsink) | Renders the composited output to a local Wayland display. |
| [`v4l2h264enc`](../plugin-reference/v4l2h264enc) / `h264parse` / `mp4mux` / `filesink` | Encodes and saves the output to a local file. |
| [`qtirtspbin`](../plugin-reference/qtirtspbin) | Streams the output over RTSP for remote viewing. |

## How it works

<Steps>
  <Step title="Stream Ingestion">
    Each `rtspsrc` element receives an RTSP stream from an IP camera. The RTP/H.264 payload is depayloaded and parsed, then passed to a hardware-accelerated decoder (`v4l2h264dec`) to produce raw NV12 frames.
  </Step>

  <Step title="Parallel AI Processing">
    A `tee` splits each decoded stream into two branches. The first branch feeds `qtimlvconverter` → `qtimltflite` → `qtimlpostprocess` for AI inference. The second branch feeds `qtimetamux` as the reference video.
  </Step>

  <Step title="Metadata Synchronization">
    `qtimetamux` attaches inference results from the AI branch to the corresponding reference video frames, maintaining per-stream temporal alignment.
  </Step>

  <Step title="Visualization">
    `qtivoverlay` draws bounding boxes and labels onto each annotated stream.
  </Step>

  <Step title="Composition">
    `qtivcomposer` tiles the annotated streams side-by-side (or in a configurable grid layout) into a single output frame.
  </Step>

  <Step title="Output Delivery">
    The composited frame is delivered to a Wayland display, saved to file, or streamed over RTSP.
  </Step>
</Steps>

## Run application on device

### Setup Requirements

#### Hardware

<img alt="HW Setup" />

| Component | Description |
| - | - |
| **Edge Device** | RB3 Gen 2, IQ8, or IQ9 — Primary processing unit for AI inference and video composition. |
| **IP/RTSP Cameras** | One or more IP cameras accessible over the local network via RTSP. |
| **HDMI Display Monitor** | Connected to the edge device for rendering the composited output. |
| **PoE Switch** | Powers IP cameras and provides network connectivity over Ethernet. (Required for IP/RTSP camera setups.) |
| **Local Network** | Ensures the edge device, IP cameras, and host machine are reachable on the same network. |

#### Software

**Flash your Qualcomm Edge device** by following the device setup and flashing instructions [here](../installation).

**Once your device is ready**, follow the instructions below to set up the Multi-Stream IP Camera pipeline.

##### AI Model and config files

| File | Download | Save as |
| - | - | - |
| YOLOv8 W8A8 model | [Qualcomm AI Hub — YOLOv8 Detection](https://aihub.qualcomm.com/iot/models/yolov8_det) | `yolov8_det_w8a8.tflite` |
| Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
| Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4">Input video</a> | `ai_demo_sample.mp4` |

**Copy files to device**

<CodeGroup>
  ```bash SCP (SSH) theme={null}
  # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
  # Run from your host machine — replace <user> and <device-ip>

  ssh <user>@<device-ip> "mkdir -p ~/{models,labels,media,media/output}"
  scp yolov8_det_w8a8.tflite   <user>@<device-ip>:~/models/
  scp yolov8.json          <user>@<device-ip>:~/labels/
  scp ai_demo_sample.mp4            <user>@<device-ip>:~/media/
  ```
</CodeGroup>

**Connect to device**

```bash theme={null}
ssh <user>@<device-ip>
```

**Run the Multi-Stream IP Camera Application**

<Note>
  A display must be connected to the device. If no display is available, use the `--no-display` flag to run in headless mode.
</Note>

Use the following base path for model and label files based on your OS:

<AccordionGroup>
  <Accordion title="Display only">
    ```bash theme={null}
    gst-ip-camera \
      --input-type=file \
      --input-config=$HOME/media/ai_demo_sample.mp4
    ```
  </Accordion>

  <Accordion title="WebRTC output">
    ```bash theme={null}
    gst-ip-camera \
      --input-type=file \
      --input-config=$HOME/media/ai_demo_sample.mp4 \
      --output-type=webrtc \
      --output-config=wss://webrtc.nirbheek.in:8443 \
      --webrtc-id=1010
    ```
  </Accordion>

  <Accordion title="RTSP output">
    ```bash theme={null}
    gst-ip-camera \
      --input-type=rtsp \
      --input-config=rtsp://<camera-ip>:<port>/stream \
      --output-type=rtsp \
      --output-config=8900
    ```
  </Accordion>

  <Accordion title="Multi-camera display (RTSP inputs)">
    ```bash theme={null}
    gst-ip-camera \
      --input-type=rtsp \
      --input-config=rtsp://<camera1-ip>:<port>/stream,rtsp://<camera2-ip>:<port>/stream
    ```
  </Accordion>
</AccordionGroup>

> **Note:** This example uses an offline video file as input. To use an IP/RTSP camera or USB camera instead, update the `--input-type` argument accordingly — refer to the **Command-Line Options** section below for details.

## Visualize the Results - Host-Side Visualization (Windows + WSL)

This section describes how to run the visualization client on a Windows host machine using **WSL (Windows Subsystem for Linux)**. The client renders the live composited video stream alongside a real-time AI metadata panel.

📥 The visualization client script can be downloaded here: <a href="../labels/rtsp_webrtc_client.zip">rtsp\_webrtc\_client.zip</a>

It displays:

* **Left panel** — Live composited video stream with AI overlays from all camera inputs.
* **Right panel** — Real-time AI metadata (JSON): object detections, bounding boxes, and confidence scores per stream.

**Step 1 — Install WSL and Ubuntu**

If WSL is not already installed, run the following from a Windows terminal:

```bash theme={null}
wsl --install Ubuntu-24.04
```

Once installed, open the Ubuntu terminal and update the system:

```bash theme={null}
sudo apt update && sudo apt upgrade -y
```

**Step 2 — Install System Dependencies**

```bash theme={null}
sudo apt install -y \
  python3 python3-pip python3-gi python3-gi-cairo \
  gir1.2-gstreamer-1.0 \
  gir1.2-gst-plugins-base-1.0 \
  gir1.2-gst-plugins-bad-1.0 \
  gstreamer1.0-tools \
  gstreamer1.0-plugins-base \
  gstreamer1.0-plugins-good \
  gstreamer1.0-plugins-bad \
  gstreamer1.0-plugins-ugly \
  gstreamer1.0-libav \
  python3-websocket \
  libnice10 \
  libnice-dev \
  gstreamer1.0-nice
```

**Step 3 — Run the Visualization Client Script**

<AccordionGroup>
  <Accordion title="RTSP">
    ```bash theme={null}
    python3 rtsp_webrtc_client.py rtsp://<DEVICE_IP>:8900/live
    ```
  </Accordion>

  <Accordion title="WebRTC">
    ```bash theme={null}
    python3 rtsp_webrtc_client.py --source webrtc --signalling-server wss://webrtc.nirbheek.in:8443 --peer-id 1010
    ```
  </Accordion>
</AccordionGroup>

**Step 4 — Expected Output**

| Panel Content | Description |
| - | - |
| Left | Real-time composited video — all streams tiled into a single view with bounding boxes and labels |
| Right | Live AI metadata panel — per-stream object detections, bounding boxes, and confidence scores |

<img alt="Expected Output" />

## Command-Line Options

<AccordionGroup>
  <Accordion title="--input-type">
    Selects the video input source for the pipeline.

    | Value | Description |
    | - | - |
    | `rtsp` | IP/RTSP camera stream. Requires `--input-config=rtsp://...`. |
    | `file` | Local H.264-encoded video file. Requires `--input-config=/path/to/ai_demo_sample.mp4`. |
    | `usb` | USB camera. Requires `--input-config=/dev/video0`. |
    | `isp` | Built-in ISP (on-device) camera. Optionally specify a camera ID via `--input-config=0`. |
  </Accordion>

  <Accordion title="--input-config">
    Specifies the input source configuration corresponding to the selected `--input-type`.

    | Input Type | Value |
    | - | - |
    | RTSP | `rtsp://<ip-or-url>` (comma-separated for multiple streams) |
    | File | `/path/to/ai_demo_sample.mp4` |
    | USB | `/dev/videoX` |
    | ISP | `<camera ID>` |
  </Accordion>

  <Accordion title="--output-type">
    Defines how the processed output video stream is delivered.

    | Value | Description |
    | - | - |
    | `display` | Renders composited output on a local Wayland display. |
    | `file` | Saves the encoded composited output to a file. Requires `--output-config`. |
    | `rtsp` | Streams composited output over RTSP. Requires `--output-config=<port>`. |
    | `webrtc` | Streams composited output over WebRTC. Requires `--output-config=ws://...`. |
    | `none` | No video output (headless mode). |
  </Accordion>

  <Accordion title="--output-config">
    Specifies the output destination configuration corresponding to the selected `--output-type`.

    | Output Type | Value |
    | - | - |
    | File | `/path/to/output.mp4` |
    | RTSP | `<port>` |
    | WebRTC | `ws://<signalling-server>:<port>` |
  </Accordion>

  <Accordion title="--no-display">
    Disables local on-screen rendering. Recommended for headless deployments and remote streaming setups.
  </Accordion>

  <Accordion title="--model-base-path">
    Specifies the root directory for AI model, label, and configuration files.

    | Asset Type | Resolved Path |
    | - | - |
    | Model files (`*.tflite`) | `<base-path>/models/<model_file>` |
    | Label/settings files (`*.json`) | `<base-path>/labels/<labels_file>` |
  </Accordion>

  <Accordion title="--width / --height / --framerate">
    Sets the raw input video resolution and frame rate. Applicable only to ISP and USB inputs.

    ```bash theme={null}
    --width=1920 --height=1080 --framerate=30
    ```
  </Accordion>
</AccordionGroup>

## Build the Application

* **Source code:** [gst-ip-camera](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ip-camera)
* **Build instructions:** [Steps to build custom application](../advanced/yocto-build#steps-to-build-custom-application)

## Conclusion

The QIM SDK enables developers to build scalable multi-stream IP camera applications without sacrificing performance or flexibility. By leveraging Qualcomm's dedicated hardware accelerators for decoding, inference, composition, and encoding — and by attaching structured AI metadata directly to each video frame — the SDK delivers a complete, production-ready foundation for multi-camera AI analytics at the edge. Whether the target use case is surveillance, retail analytics, or industrial monitoring, the pipeline can be configured to scale from a single stream to many concurrent feeds with minimal code changes.


# Edge AI for Smart Manufacturing — Real-Time Bottle Counting Using Qualcomm QIM SDK
Source: https://imsdkdocs.qualcomm.com/blogs/product-counting-application

A YOLOv8-based object detection pipeline that counts products crossing a center region of interest in real time, built with Qualcomm QIM SDK and QNN HTP acceleration.

<div>
  <video />

  <div>
    QIM SDK · Qualcomm
  </div>
</div>

<div>
  <div>
    Computer Vision
  </div>

  <p>
    A YOLOv8-based object detection pipeline that counts products crossing a center region of
    interest in real time, built with Qualcomm QIM SDK and QNN HTP acceleration.
  </p>

  <div>
    <span>QIM SDK Team</span>
    <span>·</span>
    <span>Aug 3, 2026</span>
    <span>·</span>
    <a href="/blogs">← All posts</a>
  </div>
</div>

<hr />

## Introduction

Counting products as they move along a conveyor belt or production line is a common but essential task in manufacturing, packaging, and warehouse environments. Manual counting is difficult to scale, while basic frame-by-frame object detection can easily over-count the same item as it appears across multiple consecutive frames.

The Product Counting Application addresses this challenge with a Qualcomm QIM SDK pipeline that combines a YOLOv8 object detector with a center region-of-interest (ROI) counting strategy. Instead of counting every detection in every frame, the application defines a narrow vertical band in the center of the camera view. The count increases only when a tracked object's centroid passes through this band, similar to how a physical counting gate would operate on a production line.

Inference runs fully on-device through the QNN HTP delegate, keeping the YOLOv8 TFLite model off the CPU and allowing the counting logic to keep pace with the live camera stream. Between detection and counting, a lightweight object tracker assigns each item a stable identity across frames, ensuring that each object is counted only once—even if it briefly leaves and re-enters the ROI.

## Use Case Overview

<Steps>
  <Step title="Video Capture">
    A USB camera feed is captured and normalized to a fixed resolution, format, and frame rate before entering the inference branch.
  </Step>

  <Step title="YOLOv8 Detection">
    Frames are preprocessed and passed through a quantized YOLOv8 model on the QNN HTP delegate, producing per-frame bounding box detections.
  </Step>

  <Step title="Metadata Delivery">
    Detection metadata is synchronized with the video stream and delivered to the application through an `appsink` callback as JSON.
  </Step>

  <Step title="Center-ROI Evaluation">
    Each detection's centroid is checked against a vertical counting band spanning the full frame height and the middle third of the frame width.
  </Step>

  <Step title="Cumulative Tracking">
    A tracker matches detections to previously seen objects across frames and increments the count only once per confirmed object, supporting re-entry if an object leaves and comes back.
  </Step>

  <Step title="Overlay Rendering">
    The ROI band and running count are drawn onto the display branch alongside the YOLOv8 detection boxes, and the composited frame is shown on the connected display.
  </Step>
</Steps>

## Pipeline diagram

<img alt="Product Counting Pipeline" />

## Elements used in pipeline

| Element | Description |
| - | - |
| `v4l2src` | Captures raw frames from the USB camera (`/dev/video2` by default). |
| [`qtivtransform`](../plugin-reference/qtivtransform) | Formats the raw camera frames for the pipeline; a second instance later reformats frames for the display/overlay branch. |
| Caps filter (`NV12`, `1920x1080`, `30/1`) | Fixes camera output to a known resolution, format, and frame rate for the rest of the pipeline. |
| `tee` (`split`) | Splits the raw video into an inference branch and a pass-through branch carried into the metadata muxer. |
| `queue` | Decouples and buffers data between branches; used multiple times throughout the pipeline. |
| [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | Resizes, converts color space, and normalizes frames ahead of inference. |
| [`qtimltflite`](../plugin-reference/qtimltflite) | Runs the quantized YOLOv8 detection model through the QNN HTP external delegate (`libQnnTFLiteDelegate.so`). |
| [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | Decodes raw model output into structured detections using the `yolov8` module. |
| `tee` (`post_split`) | Splits the detection-carrying stream again, feeding both the display path and the counting path. |
| [`qtimetamux`](../plugin-reference/qtimetamux) | Synchronizes detection metadata with the corresponding video frame. |
| [`qtivoverlay`](../plugin-reference/qtivoverlay) | Draws YOLOv8 bounding boxes and labels onto the video stream. |
| [`qtimlmetaparse`](../plugin-reference/qtimetaparser) / [`qtimlmetaparser`](../plugin-reference/qtimetaparser) | Parses metadata into JSON. The application selects whichever element name is available at runtime. |
| `appsink` (`count_sink`) | Delivers per-frame detection JSON to the counting logic (`emit-signals=true`, `sync=false`, `max-buffers=1`, `drop=true`). |
| Caps filter (`BGRA`) | Reformats the display branch into a format Cairo can draw onto directly. |
| `cairooverlay` (`roi_overlay`) | Renders the green ROI rectangle and the running "ROI Objects Counted" text onto the display branch. |
| [`waylandsink`](../plugin-reference/waylandsink) | Displays the final composited output (`sync=true`, `async=true`). |

## How it works

<Steps>
  <Step title="Capture and Format">
    `v4l2src` captures the raw camera feed, which is normalized to a fixed NV12 resolution and frame rate before a `tee` splits it into an inference branch and a reference branch.
  </Step>

  <Step title="Detection">
    The inference branch runs `qtimlvconverter` to prepare frames, then `qtimltflite` executes the quantized YOLOv8 model (`/etc/models/yolov8_det_w8a8.tflite`) via the QNN HTP delegate. `qtimlpostprocess`, using the `yolov8` module, decodes the raw output into bounding boxes, and `qtimetamux` merges the detections back onto the reference video frame.
  </Step>

  <Step title="Metadata Delivery">
    A second `tee` sends the metadata-carrying stream down both a display branch and a metadata branch. The metadata branch parses detections into JSON and hands them to the `count_sink` `appsink` callback for every frame.
  </Step>

  <Step title="Flexible Metadata Parsing">
    Because the detection-list and bounding-box key names can vary depending on the post-processing module version, the callback normalizes incoming JSON, checking a set of known key aliases for the detection list and for the bounding box, before extracting label, confidence, and rectangle values.
  </Step>

  <Step title="Center-ROI Evaluation">
    The counting logic defines a vertical band covering the full frame height and the middle third of the frame width, then tests whether each detection's centroid falls inside that band.
  </Step>

  <Step title="Cumulative Tracking">
    A tracker matches each frame's detections against previously tracked objects using centroid distance, bounding-box overlap, label agreement, and simple motion prediction. It smooths bounding boxes, tolerates brief detection dropouts, and increments the count only once an object is confirmed inside the ROI, while still allowing the same physical slot to be counted again if an object exits and re-enters.
  </Step>

  <Step title="Overlay and Display">
    The display branch is reformatted to BGRA, and `cairooverlay` draws the ROI rectangle and the current count on top of the `qtivoverlay`-annotated frame before `waylandsink` renders the result.
  </Step>
</Steps>

## Setup Requirements

### Hardware

<img alt="HW Setup" />

| Component | Description |
| - | - |
| **Edge Device** | RB3 Gen 2, IQ8, or IQ9. Runs the YOLOv8 detection inference and counting logic. |
| **Camera Source** | USB camera by default; an IP/RTSP camera, ISP camera, or local video file can be substituted. |
| **HDMI Display Monitor** | Connected to the edge device to show the live detection and counting overlay. |
| **PoE Switch / Local Network** | Required only when using an IP/RTSP camera or streaming results over the network. |

### Software

Flash your Qualcomm Edge device by following the device setup and flashing instructions [here](../installation), then install the Python and GStreamer prerequisites (`python3`, `gstreamer1.0`, `python3-gi`) needed to run the pipeline.

The YOLOv8 detection model and its labels are expected at `/etc/models/yolov8_det_w8a8.tflite` and `/etc/labels/yolov8.json` respectively.

<Accordion title="Try me">
  <Tabs>
    <Tab title="C++">
      <Info>
        Check application source code on GitHub: [`demo_product_counting/main.cc`](https://github.com/qualcomm/qimsdk/blob/main/cpp/examples/demo-apps/demo_product_counting/main.cc)

        Pre-built application on device: `/usr/bin/qimsdk_demo_product_counting`
      </Info>

      #### Download Required Files

      | File | Save as |
      | - | - |
      | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |

      <Note>
        If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
      </Note>

      <Steps>
        <Step title="Copy Files to Device">
          <CodeGroup>
            ```bash SCP (SSH) theme={null}
            # Replace <user> and <device-ip> with your device credentials.
            ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
            scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            ```
          </CodeGroup>
        </Step>

        <Step title="Connect a USB camera">
          Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the `--input-config` value below if your camera is exposed on a different node).
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          /usr/bin/qimsdk_demo_product_counting --input-config /dev/video2
          ```

          The application overlays YOLOv8 detection bounding boxes on the live USB camera feed and displays a running "ROI Objects Counted" total whenever a detected object's centroid crosses into the defined region of interest.

          To stop the application, press **CTRL + C**.
        </Step>
      </Steps>
    </Tab>

    <Tab title="Python">
      <Info>
        Check application source code on GitHub: [`qimsdk_demo_product_counting.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/demo-apps/qimsdk_demo_product_counting.py)

        Pre-built application on device: `/usr/bin/qimsdk_demo_product_counting.py`
      </Info>

      #### Download Required Files

      | File | Save as |
      | - | - |
      | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |

      <Note>
        If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
      </Note>

      <Steps>
        <Step title="Copy Files to Device">
          <CodeGroup>
            ```bash SCP (SSH) theme={null}
            # Replace <user> and <device-ip> with your device credentials.
            ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
            scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            ```
          </CodeGroup>
        </Step>

        <Step title="Connect a USB camera">
          Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the `--input-config` value below if your camera is exposed on a different node).
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          python3 /usr/bin/qimsdk_demo_product_counting.py --input-config /dev/video2
          ```

          The application overlays YOLOv8 detection bounding boxes on the live USB camera feed and displays a running "ROI Objects Counted" total whenever a detected object's centroid crosses into the defined region of interest.

          To stop the application, press **CTRL + C**.
        </Step>
      </Steps>
    </Tab>
  </Tabs>
</Accordion>

## Application Function Breakdown

The Product Counting application is organized into a handful of function groups, each responsible for one stage of the detection-to-count flow:

* **Metadata parsing**: normalizes incoming JSON keys and extracts label, confidence, and bounding-box values regardless of which detection-list or bbox key names the post-processing module emits.
* **ROI and counting**: computes the center ROI band from the frame dimensions and tests whether a detection's centroid falls inside it.
* **Object tracking**: a cumulative tracker that assigns stable IDs, matches detections across frames using centroid distance, IoU, label, and motion prediction, smooths bounding boxes, tolerates brief dropouts, and prevents duplicate counts while still supporting re-entry counting.
* **Rendering**: draws the ROI rectangle and the current count text onto the display branch each frame.
* **Callback and pipeline utility**: the `appsink` callback that receives each frame's metadata, a helper that selects the correct metadata-parser element name at runtime, and setup logic that wires the runtime hooks together before the pipeline starts.

## Expected Output

<video />

Running the application opens a live view on the connected monitor:

* The camera feed is shown with YOLOv8 detection boxes and labels overlaid.
* A green vertical rectangle marks the center counting ROI, spanning the full frame height.
* A running "ROI Objects Counted" total is drawn on screen and increments as tracked objects pass through the ROI.
* Objects that leave and re-enter the ROI are counted again, while objects lingering inside it are not double-counted.

Press `Ctrl+C` to stop the application; the pipeline transitions to the `NULL` state and releases the camera and display cleanly.

## Conclusion

The Product Counting Application demonstrates how an on-device YOLOv8 detector, a simple spatial rule based on a central ROI band, and a lightweight cumulative tracker can transform raw AI metadata into accurate, real-time counts. QNN HTP acceleration enables detection to keep pace with fast-moving objects, making this approach applicable beyond conveyor belts to any use case that requires counting objects as they cross a defined boundary—from retail shelf monitoring to people-flow and footfall analytics.


# Restricted Zone Violation Detection with Foot-Based Spatial Intelligence
Source: https://imsdkdocs.qualcomm.com/blogs/restricted-zone-alarm

Real-time pipeline that detects when a person steps into a predefined restricted zone using QIM SDK foot detection — with color-coded visual alerts and RTSP/WebRTC output.

<div>
  <video />

  <div>
    QIM SDK · Qualcomm
  </div>
</div>

<div>
  <div>
    Safety & Security
  </div>

  <p>
    Real-time pipeline that detects when a person steps into a predefined restricted zone using
    QIM SDK foot detection — with color-coded visual alerts, bounding box status indicators, and RTSP/WebRTC output.
  </p>

  <div>
    <span>QIM SDK Team</span>
    <span>·</span>
    <span>Jun 14, 2026</span>
    <span>·</span>
    <a href="/blogs">← All posts</a>
  </div>
</div>

<hr />

## Introduction

Monitoring access to restricted areas is a common requirement in industrial sites, warehouses, and other controlled environments. While these zones are often clearly marked, enforcing compliance in real time remains difficult. Manual supervision and post-event video review do not scale well and cannot provide the immediate response required in safety-critical scenarios.

The QIM SDK enables a shift from reactive monitoring to real-time, automated situational awareness. By using hardware-accelerated GStreamer plugins, the SDK offloads compute-intensive tasks — including video decoding, frame preparation, multi-stage AI inference, and encoding — to dedicated hardware blocks. Frame preparation includes resizing, color conversion (YUV to RGB), and pixel normalization for neural network input.

At the core of this use case is a streamlined pipeline that combines machine learning with spatial reasoning. A foot detection model identifies the location of a person's feet in each frame, providing a reliable indicator of physical position within the scene. These detections are then evaluated against a predefined restricted zone to determine, in real time, whether a boundary has been crossed.

The SDK also provides hardware-accelerated visualization through overlay rendering. Detection results and zone boundaries are composited directly onto video frames, with dynamic visual cues: **green** bounding boxes indicate the foot is outside the restricted zone, **red** indicates a violation.

Beyond visualization, the SDK supports AI metadata streaming, synchronizing detection and zone-evaluation results with the video stream and transmitting them in parallel for alerting, logging, or dashboard integration.

<Info>
  Check application source code on GitHub: [`gst-restricted-zone`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-restricted-zone)
</Info>

## Use Case Overview

<Steps>
  <Step title="Video Input">
    A camera monitors the restricted area and its surroundings. The pipeline accepts input from RTSP, ISP, USB, or file-based sources.
  </Step>

  <Step title="Foot Detection">
    Each frame is submitted to a foot detection model that identifies the location of each person's feet within the scene.
  </Step>

  <Step title="Zone Evaluation">
    `qtirestrictedzonedbg` evaluates each detected foot position against the predefined restricted zone polygon to determine whether a boundary has been crossed.
  </Step>

  <Step title="Metadata Generation">
    Detection results are attached to the video stream as structured per-frame metadata for downstream consumption.
  </Step>

  <Step title="Visualization">
    [`qtivoverlay`](../plugin-reference/qtivoverlay) renders bounding boxes and the restricted zone polygon onto video frames. Bounding boxes are **green** (outside zone) or **red** (violation).
  </Step>

  <Step title="Metadata Synchronization">
    [`qtimetamux`](../plugin-reference/qtimetamux) synchronizes all inference and zone-evaluation results with the original video frames, maintaining per-frame consistency.
  </Step>

  <Step title="Output">
    The annotated stream is delivered via RTSP or WebRTC. Structured detection data is transmitted in parallel as JSON metadata for alerting, logging, and external integrations.
  </Step>
</Steps>

## Pipeline diagram

<img alt="Restricted Zone Pipeline" />

## Elements used in pipeline

| Element | Description |
| - | - |
| `source` | Accepts video input from an RTSP camera, ISP camera, USB camera, or local file source. |
| `tee` | Splits the stream into parallel branches for simultaneous display and AI inference. |
| [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | Prepares frames for inference — performs resizing, YUV-to-RGB conversion, and normalization to match model input requirements. |
| [`qtimltflite`](../plugin-reference/qtimltflite) | Executes the TFLite foot detection model on each frame using the Qualcomm HTP via the QNN external delegate. |
| [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | Converts raw model tensors into structured bounding boxes and labels via a dynamically loaded module. |
| [`qtimetamux`](../plugin-reference/qtimetamux) | Synchronizes inference results with the original video stream as per-frame structured metadata. |
| [`qtiobjtracker`](../plugin-reference/qtiobjtracker) | Tracks detected feet across consecutive frames and assigns a consistent ID to each object. |
| `qtirestrictedzonedbg` | Defines the restricted zone polygon and evaluates detections against it. Updates bounding box color to **red** upon violation. |
| [`qtivoverlay`](../plugin-reference/qtivoverlay) | Renders bounding boxes, labels, and the restricted zone polygon directly onto video frames. |
| [`qtimlmetaparser`](../plugin-reference/qtimetaparser) | Serializes per-frame metadata into JSON format for integration with external systems. |
| [`v4l2h264enc`](../plugin-reference/v4l2h264enc) / `h264parse` | Encodes the processed video stream into H.264 format. |
| [`waylandsink`](../plugin-reference/waylandsink) | Displays the video locally on the device. |
| `sink` | Streams the encoded video and metadata over RTSP or WebRTC via `rtspbin` or `webrtcbin`. |

## How it works

<Steps>
  <Step title="Stage 1 — Foot Detection">
    The full video frame is preprocessed by [`qtimlvconverter`](../plugin-reference/qtimlvconverter) to align with model input requirements. Preprocessed tensors are passed to [`qtimltflite`](../plugin-reference/qtimltflite), which runs the foot detection model and produces raw output tensors.
  </Step>

  <Step title="Post-Processing">
    [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) converts the tensors into bounding box detections. [`qtimetamux`](../plugin-reference/qtimetamux) attaches these detections to the original video frame as metadata.
  </Step>

  <Step title="Stage 2 — Zone Evaluation">
    `qtirestrictedzonedbg` evaluates each detected foot position against the configured restricted zone polygon. If a foot lies inside the polygon, the pipeline flags it as a violation in real time.
  </Step>

  <Step title="Tracking">
    `qtiobjtracker` maintains object identity across frames, assigning a consistent tracking ID to each detected foot for reliable zone evaluation over time.
  </Step>

  <Step title="Visualization and Output">
    [`qtivoverlay`](../plugin-reference/qtivoverlay) renders bounding boxes and the restricted zone on the video stream. Visual cues update dynamically based on zone evaluation. The processed stream can be displayed locally, saved to file, or streamed via RTSP or WebRTC. `qtimlmetaparser` serializes metadata into JSON for external system integration.
  </Step>
</Steps>

## Run application on device

### Setup Requirements

#### Hardware

<img alt="Hardware Setup" />

| Component | Description |
| - | - |
| **Edge Device** | RB3 Gen 2, IQ8, or IQ9 — Primary processing unit for AI inference and video composition. |
| **Camera Source** | IP/RTSP camera, ISP (on-device) camera, or USB camera. A local file source may be substituted if no physical camera is available. |
| **HDMI Display Monitor** | Connected to the edge device for rendering and visualizing pipeline output. |
| **PoE Switch** | Powers IP/RTSP cameras and provides network connectivity over a single Ethernet cable per camera. (Required for IP/RTSP camera setups only.) |
| **Local Network** | Ensures the edge device, RTSP camera, and host machine are reachable on the same network. (Required when using RTSP camera input or streaming results via RTSP or WebRTC.) |

#### Software

**Flash your Qualcomm Edge device** by following the device setup and flashing instructions [here](../installation).

**Once your device is ready**, follow the instructions below to set up the Restricted Zone pipeline.

##### AI Model and config files

| File | Download | Save as |
| - | - | - |
| Foot detection model | [Qualcomm AI Hub — Foot Track Net](https://aihub.qualcomm.com/iot/models/foot_track_net) | `foot_track_net_quantized.tflite` |
| Detection labels | <a href="../labels/foot_track_net.json">foot\_track\_net.json</a> | `foot_track_net.json` |
| Detection settings | <a href="../labels/foot_track_net_settings.json">foot\_track\_net\_settings.json</a> | `foot_track_net_settings.json` |
| Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/rz_sample.mp4">Input video</a> | `rz_sample.mp4` |

> **Note:** The restricted zone polygon is defined in `polygon_config.json`, which is installed with the application at `/etc/configs/polygon_config.json`. Edit this file (or pass a different one via `--polygon-config-path`) to customize the zone shape and color — see **Update Restricted Zone Area** below.

**Copy files to device**

<CodeGroup>
  ```bash SCP (SSH) theme={null}
  # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

  ssh <user>@<device-ip> "mkdir -p ~/{models,labels,media,media/output}"
  scp foot_track_net_quantized.tflite   <user>@<device-ip>:~/models/
  scp foot_track_net.json               <user>@<device-ip>:~/labels/
  scp foot_track_net_settings.json      <user>@<device-ip>:~/labels/
  scp rz_sample.mp4                         <user>@<device-ip>:~/media/
  ```
</CodeGroup>

**Connect to device**

```bash theme={null}
ssh <user>@<device-ip>
```

**Run the Restricted Zone Application**

> **Note:** A display must be connected to the device. If no display is available, use the `--no-display` flag.

<AccordionGroup>
  <Accordion title="Display only">
    ```bash theme={null}
    gst-restricted-zone \
      --input-type=file \
      --input-config=$HOME/media/rz_sample.mp4
    ```
  </Accordion>

  <Accordion title="WebRTC output">
    ```bash theme={null}
    gst-restricted-zone \
      --input-type=file \
      --input-config=$HOME/media/rz_sample.mp4 \
      --output-type=webrtc \
      --output-config=wss://webrtc.nirbheek.in:8443 \
      --webrtc-id=1010
    ```
  </Accordion>

  <Accordion title="RTSP output">
    ```bash theme={null}
    gst-restricted-zone \
      --input-type=file \
      --input-config=$HOME/media/rz_sample.mp4 \
      --output-type=rtsp \
      --output-config=8900
    ```
  </Accordion>
</AccordionGroup>

> **Note:** This example uses an offline video file as input. To use an IP/RTSP camera or USB camera instead, update the `--input-type` argument accordingly — refer to the **Command-Line Options** section below.

It produces two key output results: an AI-annotated video stream and a JSON metadata stream. To visualize these results, refer to the **Host-Side Visualization** section below.

## Visualize the Results - Host-Side Visualization (Windows + WSL)

This section describes how to run the visualization client on a Windows host machine using **WSL (Windows Subsystem for Linux)**. The client renders the live video stream alongside a real-time AI metadata panel.

📥 The visualization client script can be downloaded here: <a href="../labels/rtsp_webrtc_client.zip">rtsp\_webrtc\_client.zip</a>

It displays:

* **Left panel** — Live video stream with AI overlays (bounding boxes and restricted zone polygon).
* **Right panel** — Real-time AI metadata (JSON): object detections, tracking IDs, bounding boxes, and confidence scores.

**Step 1 — Install WSL and Ubuntu**

If WSL is not already installed, run the following from a Windows terminal:

```bash theme={null}
wsl --install Ubuntu-24.04
```

Once installed, update the system:

```bash theme={null}
sudo apt update && sudo apt upgrade -y
```

**Step 2 — Install System Dependencies**

```bash theme={null}
sudo apt install -y \
  python3 python3-pip python3-gi python3-gi-cairo \
  gir1.2-gstreamer-1.0 \
  gir1.2-gst-plugins-base-1.0 \
  gir1.2-gst-plugins-bad-1.0 \
  gstreamer1.0-tools \
  gstreamer1.0-plugins-base \
  gstreamer1.0-plugins-good \
  gstreamer1.0-plugins-bad \
  gstreamer1.0-plugins-ugly \
  gstreamer1.0-libav \
  python3-websocket \
  libnice10 \
  libnice-dev \
  gstreamer1.0-nice
```

**Step 3 — Run the Visualization Client Script**

<AccordionGroup>
  <Accordion title="RTSP">
    ```bash theme={null}
    python3 rtsp_webrtc_client.py rtsp://<DEVICE_IP>:8900/live
    ```
  </Accordion>

  <Accordion title="WebRTC">
    ```bash theme={null}
    python3 rtsp_webrtc_client.py --source webrtc --signalling-server wss://webrtc.nirbheek.in:8443 --peer-id 1010
    ```
  </Accordion>
</AccordionGroup>

**Step 4 — Expected Output**

| Panel | Content |
| - | - |
| Left | Real-time decoded video stream with foot detection bounding boxes and restricted zone polygon |
| Right | Live AI metadata — tracking IDs, bounding boxes, foot coordinates, and confidence scores |

After following the steps, the video and metadata streams should be up and running. When a person steps into the restricted zone, the corresponding bounding box changes from **green** to **red**.

The pipeline generates structured JSON metadata in the following format:

```json theme={null}
{
  "object_detection": [
    {
      "tracking_id": 7,
      "label": "person",
      "confidence": 71.63,
      "color": 16711935,
      "rectangle": {
        "x": 0.16, "y": 0.05,
        "width": 0.16, "height": 0.16
      },
      "landmarks": {
        "left_ankle":  { "x": 0.21, "y": 0.10 },
        "right_ankle": { "x": 0.20, "y": 0.07 }
      }
    }
  ],
  "parameters": { "timestamp": "49058322222" }
}
```

## Command-Line Options

<AccordionGroup>
  <Accordion title="--input-type">
    Selects the video input source for the pipeline.

    | Value | Description |
    | - | - |
    | `usb` | USB camera. Requires `--input-config=/dev/video0`. |
    | `isp` | Built-in ISP (on-device) camera. Optionally specify a camera ID via `--input-config=0`. |
    | `rtsp` | External IP/RTSP camera or stream. Requires `--input-config=rtsp://...`. |
    | `file` | Local H.264-encoded video file. Requires `--input-config=/path/to/ai_demo_sample.mp4`. |
  </Accordion>

  <Accordion title="--input-config">
    Specifies the input source configuration corresponding to the selected `--input-type`.

    | Input Type | Value |
    | - | - |
    | USB | `/dev/videoX` |
    | ISP | `<camera ID>` |
    | RTSP | `rtsp://<ip-or-url>` |
    | File | `/path/to/rz_sample.mp4` |
  </Accordion>

  <Accordion title="--output-type">
    Defines how the processed output video stream is delivered.

    | Value | Description |
    | - | - |
    | `none` | No video output (headless mode). |
    | `file` | Save encoded output to a file. Requires `--output-config`. |
    | `rtsp` | Stream over RTSP. Requires `--output-config=<port>`. Access at `rtsp://<device-ip>:<port>/live`. |
    | `webrtc` | Stream over WebRTC. Requires `--output-config=ws://...`. |
  </Accordion>

  <Accordion title="--output-config">
    Specifies the output destination configuration.

    | Output Type | Value |
    | - | - |
    | File | `/path/to/output.mp4` |
    | RTSP | `<port>` |
    | WebRTC | `ws://<signalling-server>:<port>` |
  </Accordion>

  <Accordion title="--model-base-path">
    Root directory for AI model, label, and configuration files.

    | Asset Type | Resolved Path |
    | - | - |
    | Model files (`*.tflite`) | `<base-path>/models/<model_file>` |
    | Label/settings files (`*.json`) | `<base-path>/labels/<labels_file>` |

    ```bash theme={null}
    --model-base-path=/root        # QLI
    --model-base-path=/home/ubuntu # Ubuntu
    ```
  </Accordion>

  <Accordion title="--no-display">
    Disables local on-screen rendering. Recommended for headless deployments, remote streaming (RTSP/WebRTC), or performance optimization.
  </Accordion>

  <Accordion title="--width / --height / --framerate">
    Sets the raw input video resolution and frame rate. Applicable only to ISP and USB inputs.

    ```bash theme={null}
    --width=1920 --height=1080 --framerate=30
    ```
  </Accordion>

  <Accordion title="--webrtc-id">
    Specifies the local WebRTC signaling client ID.

    ```bash theme={null}
    --webrtc-id=1010
    ```
  </Accordion>

  <Accordion title="--polygon-config-path">
    Path to the JSON file describing the restricted zone polygon and its overlay color.

    ```bash theme={null}
    --polygon-config-path=/etc/configs/polygon_config.json  # default
    ```
  </Accordion>
</AccordionGroup>

## Update Restricted Zone Area

The restricted zone is defined in an external JSON file rather than hardcoded in the application. By default, the app reads `/etc/configs/polygon_config.json`, or any path passed via `--polygon-config-path`.

```json theme={null}
{
  "polygon_edges": [
    [640, 420],
    [687, 408],
    [843, 476],
    [848, 528],
    [937, 583],
    [641, 729]
  ],
  "color": "0x7F00007F"
}
```

| Field | Description |
| - | - |
| `polygon_edges` | An ordered list of `[x, y]` pixel coordinates describing the zone polygon. Points can be specified in clockwise or counterclockwise order. A polygon can contain any number of vertices. |
| `color` | Overlay fill color for the zone, expressed as an `0xAARRGGBB` hex string. |

At startup, the application loads this file, converts the vertex list into the `<<x1,y1>,<x2,y2>,...,<xn,yn>>` format expected by `qtirestrictedzonedbg`, and applies it — together with the overlay color — to the element's `zone-config` and `qtivoverlay` properties. If a detected foot falls inside the polygon, a violation is triggered.

To change the zone shape or color, edit `polygon_config.json` (or point `--polygon-config-path` at a different file) and restart the application — **no rebuild is required**.

## Implementation Deep-Dive

<AccordionGroup>
  <Accordion title="1. Application Configuration and Runtime Context">
    The application separates user-configurable settings from runtime state.

    ```c theme={null}
    typedef struct GstAppConfig {
      gchar *input_type, *input_config;
      gchar *output_type, *output_config;
      gchar *model_base_path;
      gboolean no_display;
      gint width, height, framerate, webrtc_id;
    } GstAppConfig;

    typedef struct GstAppContext {
      GstAppConfig config;
      GstElement  *pipeline;
      GMainLoop   *mloop;
      GstElement  *webrtc;
      gboolean     is_shutting_down;
    } GstAppContext;
    ```
  </Accordion>

  <Accordion title="2. Reusable Pipeline Skeleton">
    The pipeline is assembled from three logical sections: input branch, output branch, and application-specific user branch.

    ```c theme={null}
    static gboolean gst_app_create_pipe (GstAppContext *appctx) {
      GstElement *input_tail = NULL, *output_head = NULL, *meta_head = NULL;

      appctx->pipeline = gst_pipeline_new ("gst-ip-camera");

      if (!gst_app_create_input_pipe  (appctx, &input_tail))  return FALSE;
      if (!gst_app_create_output_pipe (appctx, &output_head, &meta_head)) return FALSE;
      if (!gst_app_create_user_pipe   (appctx, input_tail, output_head, meta_head)) return FALSE;
      return TRUE;
    }
    ```
  </Accordion>

  <Accordion title="3. Inference, Zone, and Overlay Configuration">
    The pipeline uses the QNN external delegate, QPD post-processing, JSON metadata serialization, and overlay masks. The restricted zone polygon and its overlay color are not hardcoded — they are loaded at startup from the JSON file passed via `--polygon-config-path` (default `/etc/configs/polygon_config.json`):

    ```json theme={null}
    {
      "polygon_edges": [
        [640, 420], [687, 408], [843, 476],
        [848, 528], [937, 583], [641, 729]
      ],
      "color": "0x7F00007F"
    }
    ```

    `load_polygon_config()` parses this file and converts `polygon_edges` into the `<<x1,y1>,...,<xn,yn>>` string expected by `qtirestrictedzonedbg`:

    ```c theme={null}
    gchar *polygon = NULL, *color = NULL;
    load_polygon_config (appctx->config.polygon_config_path, &polygon, &color);

    gchar *overlay_rz = g_strdup_printf (
        "{(structure)\"Zone1,polygon=%s,color=%s;\"}", polygon, color);
    ```

    ```c theme={null}
    gst_element_set_enum_property (qtimlpostprocess, "module", "qpd");

    delegate_options = gst_structure_from_string (
        "QNNExternalDelegate,backend_type=htp;", NULL);
    g_object_set (G_OBJECT (qtimltflite),
        "external-delegate-path", "libQnnTFLiteDelegate.so",
        "external-delegate-options", delegate_options,
        "model", model_path, NULL);
    gst_element_set_enum_property (qtimltflite, "delegate", "external");

    g_object_set (G_OBJECT (qtirestrictedzonedbg),
        "zone-config", g_strdup_printf ("Zones,Zone1=%s;", polygon),
        NULL);

    g_object_set (G_OBJECT (qtivoverlay), "masks", overlay_rz, NULL);
    ```
  </Accordion>

  <Accordion title="4. Pipeline Linking">
    The metadata-enriched stream is routed through tracking, zone evaluation, and overlay before reaching the output branch.

    ```c theme={null}
    gst_element_link_many (input_tail, tee[0], queue[0], qtimetamux, NULL);

    gst_element_link_many (tee[0], queue[1], qtimlvconverter, queue[2],
        qtimltflite, queue[3], qtimlpostprocess, postprocess_caps,
        queue[4], qtimetamux, NULL);

    gst_element_link_many (qtimetamux, queue[5], qtiobjtracker,
        qtirestrictedzonedbg, qtivoverlay, tee[1], NULL);

    if (output_head != NULL)
        gst_element_link_many (tee[1], queue[6], output_head, NULL);
    if (meta_head != NULL)
        gst_element_link_many (tee[1], queue[7], qtimlmetaparser, meta_head, NULL);
    ```
  </Accordion>

  <Accordion title="5. WebRTC Signaling">
    WebRTC signaling uses explicit SDP offer/answer and ICE candidate exchange via WebSocket with `libsoup`.

    ```c theme={null}
    g_signal_emit_by_name (webrtcbin, "create-data-channel", name, NULL, &ch);

    GstPromise *promise = gst_promise_new_with_change_func (on_offer_created, appctx, NULL);
    g_signal_emit_by_name (webrtcbin, "create-offer", NULL, promise);

    g_signal_connect (appctx->webrtc, "on-ice-candidate",
        G_CALLBACK (on_webrtc_ice_candidate), appctx);
    ```

    | Callback | Responsibility |
    | - | - |
    | `on_offer_created` | Constructs and sends the SDP offer |
    | `on_ice_candidate` | Transmits ICE candidates to the signaling server |
    | `on_ws_message` | Handles incoming WebSocket signaling messages |
  </Accordion>
</AccordionGroup>

## Build the Application

* **Source code:** [gst-restricted-zone](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-restricted-zone)
* **Build instructions:** [Steps to build custom application](../advanced/yocto-build#steps-to-build-custom-application)

## Conclusion

The Restricted Zone Alarm demonstrates how the QIM SDK converts live video into actionable spatial intelligence. By combining AI-based foot detection with in-pipeline zone evaluation, it detects boundary violations in real time with low integration overhead.

The pipeline is designed for interoperability — results can be rendered on-screen, streamed over the network, or exported as structured metadata for downstream systems. This modular architecture makes the solution easy to extend for visualization, alerting, and external processing, making it a practical foundation for location-aware video analytics.


# Building a Scalable Multi-Stream AI Video Wall with QIM SDK
Source: https://imsdkdocs.qualcomm.com/blogs/security-video-wall

Process up to 31 concurrent IP camera streams in parallel with YOLOv8 object detection, compositing all streams into a single output and streaming over RTSP/WebRTC.

<div>
  <video />

  <div>
    QIM SDK · Qualcomm
  </div>
</div>

<div>
  <div>
    Multi-Stream AI
  </div>

  <p>
    Process up to 31 concurrent IP camera streams in parallel with YOLOv8 object detection,
    compositing all streams into a unified video wall output and streaming over RTSP or WebRTC.
  </p>

  <div>
    <span>QIM SDK Team</span>
    <span>·</span>
    <span>Jun 14, 2026</span>
    <span>·</span>
    <a href="/blogs">← All posts</a>
  </div>
</div>

<hr />

## Introduction

Modern security systems rarely depend on a single camera. In real deployments, operators need to monitor dozens of streams simultaneously across warehouses, campuses, retail spaces, and large environments. As camera counts grow, manual monitoring becomes impractical, and traditional CPU-only processing struggles to keep up — increasing latency, power usage, and system load.

The QIM SDK addresses this directly. It enables scalable, real-time multi-stream video analytics at the edge using hardware-accelerated GStreamer plugins that shift demanding work — decoding, frame preparation, AI inference, and encoding — entirely onto dedicated hardware blocks. Frame preparation includes resizing frames to match model input, converting YUV to RGB, and normalizing pixel values for neural network input.

The pipeline processes each stream independently, running object detection on every input source while preserving a consistent visual experience. To improve efficiency, inference runs at a reduced frame rate without affecting the continuity of the displayed output. Detection results are rendered as RGBA overlay masks and composited with the corresponding video feeds, allowing object visualization without modifying source frames directly.

Integration with Qualcomm AI Hub gives developers access to optimized, ready-to-use models such as YOLO-based detectors for multi-stream analytics.

<Info>
  Check application source code on GitHub: [`gst-video-wall`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-video-wall)
</Info>

## Use Case Overview

<Steps>
  <Step title="Video Input">
    Each RTSP camera provides an H.264/H.265-encoded stream decoded into raw YUV frames for inference and visualization.
  </Step>

  <Step title="Frame Rate Optimization">
    `videorate` controls the inference processing rate to improve performance while preserving temporal accuracy in the displayed output.
  </Step>

  <Step title="Object Detection">
    Each decoded frame is analyzed independently by an object detection model running on the Qualcomm NPU.
  </Step>

  <Step title="Detection Output">
    The model produces an RGBA overlay mask containing bounding boxes and class labels over a transparent background.
  </Step>

  <Step title="Composition">
    [`qtivcomposer`](../plugin-reference/qtivcomposer) composites all stream masks onto their corresponding video frames and tiles them into a configurable M×N grid (up to 8×4).
  </Step>

  <Step title="Metadata Synchronization">
    [`qtimetamux`](../plugin-reference/qtimetamux) attaches detection results as structured per-frame metadata synchronized with the video stream.
  </Step>

  <Step title="Output">
    The final composited stream is displayed locally on an HDMI monitor or streamed over RTSP/WebRTC with structured metadata transmitted in parallel.
  </Step>
</Steps>

## Pipeline diagram

<img alt="Security Video Wall Pipeline" />

## Elements used in pipeline

| Element | Description |
| - | - |
| `source` | Accepts input from an RTSP camera, USB camera, or local file source. |
| `tee` | Splits each decoded stream into parallel branches for simultaneous display and AI inference. |
| `videorate` | Adjusts the video frame rate — reduces rate by half to lower compute load while maintaining display continuity. |
| [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | Prepares frames for inference — resizes, converts YUV to RGB, and normalizes input to match model requirements. |
| [`qtimltflite`](../plugin-reference/qtimltflite) | Runs the TFLite object detection model on each frame using the Qualcomm NPU via the QNN external delegate. |
| [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | Converts raw output tensors into structured bounding boxes and labels via a dynamically loaded module. |
| [`qtimetamux`](../plugin-reference/qtimetamux) | Synchronizes inference results with the original video stream as per-frame structured metadata. |
| [`qtivcomposer`](../plugin-reference/qtivcomposer) | Composites video from all streams into a single 8×4 grid output and overlays RGBA masks onto corresponding YUV frames. |
| [`v4l2h264enc`](../plugin-reference/v4l2h264enc) / `h264parse` | Encodes the composited stream into H.264 format for transmission. |
| [`waylandsink`](../plugin-reference/waylandsink) | Displays the composited video locally on the device. |
| `sink` | Streams the encoded video and metadata over RTSP or WebRTC via `rtspbin` or `webrtcbin`. |

## How it works

<Steps>
  <Step title="Stream Ingestion">
    Each `rtspsrc` element receives an RTSP stream. The RTP/H.264 payload is depayloaded, parsed, and decoded into raw NV12 frames by the hardware decoder.
  </Step>

  <Step title="Parallel Processing">
    A `tee` splits each decoded stream into two branches: one forwards frames directly to the compositor, the other runs AI inference at a reduced frame rate.
  </Step>

  <Step title="ML Inference">
    The inference branch runs through [`qtimlvconverter`](../plugin-reference/qtimlvconverter) → [`qtimltflite`](../plugin-reference/qtimltflite) → [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) producing an RGBA overlay mask.
  </Step>

  <Step title="Composition">
    [`qtivcomposer`](../plugin-reference/qtivcomposer) tiles all annotated streams into an 8×4 grid. If detection runs at a lower rate than the input, the most recent mask is reused to maintain a stable overlay.
  </Step>

  <Step title="Output Delivery">
    The composited frame is delivered to a Wayland display, saved to file, or streamed over RTSP/WebRTC.
  </Step>
</Steps>

## Run application on device

### Setup Requirements

#### Hardware

<img alt="Hardware Setup" />

| Component | Description |
| - | - |
| **Edge Device** | IQ9 — Primary processing unit for AI inference and video composition. |
| **Camera Source** | IP/RTSP cameras. A local file source may be substituted if no physical camera is available. |
| **HDMI Display Monitor** | Connected to the edge device for rendering and visualizing pipeline output. |
| **PoE Switch** | Powers IP/RTSP cameras and provides network connectivity over a single Ethernet cable per camera. (Required for IP/RTSP setups only.) |
| **Local Network** | Ensures the edge device, cameras, and host machine are reachable on the same network. (Required for RTSP input or RTSP/WebRTC output.) |

#### Software

**Flash your Qualcomm Edge device** by following the device setup and flashing instructions [here](../installation).

**Once your device is ready**, follow the instructions below to set up the Security Video Wall pipeline.

##### AI Model and config files

| File | Download | Save as |
| - | - | - |
| YOLOv8 W8A8 model | [Qualcomm AI Hub — YOLOv8 Detection](https://aihub.qualcomm.com/iot/models/yolov8_det) | `yolov8_det_w8a8.tflite` |
| Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
| Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

**Copy files to device**

<CodeGroup>
  ```bash SCP (SSH) theme={null}
  # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

  ssh <user>@<device-ip> "mkdir -p ~/{models,labels,media,media/output}"
  scp yolov8_det_w8a8.tflite   <user>@<device-ip>:~/models/
  scp yolov8.json                   <user>@<device-ip>:~/labels/
  scp ai_demo_sample.mp4                     <user>@<device-ip>:~/media/
  ```
</CodeGroup>

**Connect to device**

```bash theme={null}
ssh <user>@<device-ip>
```

**Run the Security Video Wall**

> **Note:** A display must be connected to the device. If no display is available, use the `--no-display` flag.

<AccordionGroup>
  <Accordion title="Display only">
    ```bash theme={null}
    ulimit -n 16192 && \
    gst-video-wall \
      --input-count=4 \
      $(for i in $(seq 1 4); do echo "--input-type=file --input-config=$HOME/media/ai_demo_sample.mp4"; done)
    ```
  </Accordion>

  <Accordion title="WebRTC output">
    ```bash theme={null}
    ulimit -n 16192 && \
    gst-video-wall \
      --input-count=31 \
      $(for i in $(seq 1 31); do echo "--input-type=file --input-config=$HOME/media/ai_demo_sample.mp4"; done) \
      --output-type=webrtc \
      --output-config=wss://webrtc.nirbheek.in:8443 \
      --webrtc-id=1010
    ```
  </Accordion>

  <Accordion title="RTSP output">
    ```bash theme={null}
    ulimit -n 16192 && \
    gst-video-wall \
      --input-count=31 \
      $(for i in $(seq 1 31); do echo "--input-type=file --input-config=$HOME/media/ai_demo_sample.mp4"; done) \
      --output-type=rtsp \
      --output-config=8900
    ```
  </Accordion>
</AccordionGroup>

> **Note:** This example uses an offline video file as input. To use IP/RTSP cameras, update `--input-type=rtsp` and `--input-config=rtsp://...` accordingly.

It produces an AI-annotated video stream. To visualize the results, refer to the **Host-Side Visualization** section below.

## Visualize the Results - Host-Side Visualization (Windows + WSL)

This section describes how to run the visualization client on a Windows host machine using **WSL (Windows Subsystem for Linux)**. The client renders the live composited video stream alongside a real-time AI metadata panel.

📥 The visualization client script can be downloaded here: <a href="../labels/rtsp_webrtc_client.zip">rtsp\_webrtc\_client.zip</a>

It displays:

* **Left panel** — Live composited video stream with AI overlays from all camera inputs.
* **Right panel** — Real-time AI metadata (JSON): object detections, bounding boxes, and confidence scores per stream.

**Step 1 — Install WSL and Ubuntu**

If WSL is not already installed, run the following from a Windows terminal:

```bash theme={null}
wsl --install Ubuntu-24.04
```

Once installed, update the system:

```bash theme={null}
sudo apt update && sudo apt upgrade -y
```

**Step 2 — Install System Dependencies**

```bash theme={null}
sudo apt install -y \
  python3 python3-pip python3-gi python3-gi-cairo \
  gir1.2-gstreamer-1.0 \
  gir1.2-gst-plugins-base-1.0 \
  gir1.2-gst-plugins-bad-1.0 \
  gstreamer1.0-tools \
  gstreamer1.0-plugins-base \
  gstreamer1.0-plugins-good \
  gstreamer1.0-plugins-bad \
  gstreamer1.0-plugins-ugly \
  gstreamer1.0-libav \
  python3-websocket \
  libnice10 \
  libnice-dev \
  gstreamer1.0-nice
```

**Step 3 — Run the Visualization Client Script**

<AccordionGroup>
  <Accordion title="RTSP">
    ```bash theme={null}
    python3 rtsp_webrtc_client.py rtsp://<DEVICE_IP>:8900/live
    ```
  </Accordion>

  <Accordion title="WebRTC">
    ```bash theme={null}
    python3 rtsp_webrtc_client.py --source webrtc --signalling-server wss://webrtc.nirbheek.in:8443 --peer-id 1010
    ```
  </Accordion>
</AccordionGroup>

**Step 4 — Expected Output**

| Panel | Content |
| - | - |
| Left | Real-time composited video — all streams tiled in a grid with bounding boxes and labels |
| Right | Live AI metadata — per-stream object detections, bounding boxes, and confidence scores |

<img alt="Expected Output" />

Beyond the default setup, the application offers flexible input and output configurations that can be tailored via command-line options, as described below:

## Command-Line Options

<AccordionGroup>
  <Accordion title="--input-count">
    Specifies the total number of input video streams. Must match the number of `--input-type` and `--input-config` entries. Valid range: 1 to 31.

    ```bash theme={null}
    --input-count=4
    ```
  </Accordion>

  <Accordion title="--input-type">
    Selects the video input source for each stream.

    | Value | Description |
    | - | - |
    | `rtsp` | External IP/RTSP camera. Requires `--input-config=rtsp://...`. |
    | `file` | Local H.264-encoded video file. Requires `--input-config=/path/to/ai_demo_sample.mp4`. |
  </Accordion>

  <Accordion title="--input-config">
    Specifies the input source configuration for the selected `--input-type`.

    | Input Type | Value |
    | - | - |
    | RTSP | `rtsp://<ip-or-url>` |
    | File | `/path/to/ai_demo_sample.mp4` |
  </Accordion>

  <Accordion title="--output-type">
    Defines how the processed output is delivered.

    | Value | Description |
    | - | - |
    | `none` | No video output (headless mode). |
    | `file` | Save encoded output to a file. Requires `--output-config`. |
    | `rtsp` | Stream over RTSP. Requires `--output-config=<port>`. Access at `rtsp://<device-ip>:<port>/live`. |
    | `webrtc` | Stream over WebRTC. Requires `--output-config=ws://...`. |
  </Accordion>

  <Accordion title="--output-config">
    Specifies the output destination configuration.

    | Output Type | Value |
    | - | - |
    | File | `/path/to/output.mp4` |
    | RTSP | `<port>` |
    | WebRTC | `ws://<signalling-server>:<port>` |
  </Accordion>

  <Accordion title="--model-base-path">
    Root directory for AI model, label, and configuration files.

    | Asset Type | Resolved Path |
    | - | - |
    | Model files (`*.tflite`) | `<base-path>/models/<model_file>` |
    | Label/settings files (`*.json`) | `<base-path>/labels/<labels_file>` |

    ```bash theme={null}
    --model-base-path=/root        # QLI
    --model-base-path=/home/ubuntu # Ubuntu
    ```
  </Accordion>

  <Accordion title="--no-display">
    Disables local on-screen rendering. Recommended for headless deployments, remote streaming (RTSP/WebRTC), or performance optimization.
  </Accordion>

  <Accordion title="--num-npus">
    Specifies the video frame rate for the input stream.

    ```bash theme={null}
    --num-npus=N
    ```
  </Accordion>

  <Accordion title="--webrtc-id">
    Specifies the local WebRTC signaling client ID.

    ```bash theme={null}
    --webrtc-id=1010
    ```
  </Accordion>
</AccordionGroup>

## Implementation Deep-Dive

<AccordionGroup>
  <Accordion title="1. Application Configuration and Runtime Context">
    The application separates user configuration from runtime state.

    ```c theme={null}
    typedef struct GstAppConfig {
      gint    input_count;
      gchar **input_types;
      gchar **input_configs;
      gchar  *output_type;
      gchar  *output_config;
      gchar  *model_base_path;
      gboolean no_display;
      gint    width, height, framerate, webrtc_id;
    } GstAppConfig;

    typedef struct GstAppContext {
      GstAppConfig config;
      GstElement  *pipeline;
      GMainLoop   *mloop;
      GstAppPadLinkData qtdemux_links[GST_APP_MAX_INPUTS];
      GstAppPadLinkData rtspsrc_links[GST_APP_MAX_INPUTS];
      GstElement  *webrtc;
      gboolean     is_shutting_down;
    } GstAppContext;
    ```
  </Accordion>

  <Accordion title="2. Reusable Pipeline Skeleton">
    The pipeline is assembled from three logical sections: input branch, output branch, and application-specific user branch.

    ```c theme={null}
    static gboolean gst_app_create_pipe (GstAppContext *appctx) {
      GstElement *input_tails[GST_APP_MAX_INPUTS] = { NULL };
      GstElement *output_head = NULL;

      appctx->pipeline = gst_pipeline_new ("gst-video-wall");

      if (!gst_app_create_input_pipe  (appctx, input_tails))          return FALSE;
      if (!gst_app_create_output_pipe (appctx, &output_head))          return FALSE;
      if (!gst_app_create_user_pipe   (appctx, input_tails, output_head)) return FALSE;
      return TRUE;
    }
    ```
  </Accordion>

  <Accordion title="3. Multi-Input Configuration and Composer Geometry">
    Constants define the maximum input count and compositor layout.

    ```c theme={null}
    #define GST_APP_MAX_INPUTS          31
    #define GST_APP_COMPOSER_COLUMNS     8
    #define GST_APP_COMPOSER_ROWS        4
    #define GST_APP_COMPOSER_CELL_WIDTH  240
    #define GST_APP_COMPOSER_CELL_HEIGHT 135
    #define MODEL_PATH  "yolov8_det_w8a8.tflite"
    #define LABELS_PATH "yolov8.json"
    ```

    Each input stream contributes one direct video layer and one RGBA overlay layer to the same grid position in the compositor.
  </Accordion>

  <Accordion title="4. WebRTC Signaling">
    WebRTC signaling uses explicit SDP offer/answer and ICE candidate exchange via WebSocket with `libsoup`.

    ```c theme={null}
    g_signal_emit_by_name (webrtcbin, "create-data-channel", name, NULL, &ch);

    GstPromise *promise = gst_promise_new_with_change_func (on_offer_created, appctx, NULL);
    g_signal_emit_by_name (webrtcbin, "create-offer", NULL, promise);

    g_signal_connect (appctx->webrtc, "on-ice-candidate",
        G_CALLBACK (on_webrtc_ice_candidate), appctx);
    ```

    | Callback | Responsibility |
    | - | - |
    | `on_offer_created` | Constructs and sends the SDP offer |
    | `on_ice_candidate` | Transmits ICE candidates to the signaling server |
    | `on_ws_message` | Handles incoming WebSocket signaling messages |
  </Accordion>
</AccordionGroup>

## Build the Application

* **Source code:** [gst-video-wall](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-video-wall)
* **Build instructions:** [Steps to build custom application](../advanced/yocto-build#steps-to-build-custom-application)

## Conclusion

The QIM SDK modular architecture gives developers flexibility and control when building real-time multi-stream video analytics pipelines. By separating inference from post-processing, the post-processing stage can be customized without changing the model execution path. This keeps AI results and video frames decoupled while enabling efficient visualization — post-processing generates an RGBA overlay mask composited onto the original frame without duplicating video data, delivering lower latency, reduced memory overhead, and better scalability for real-time AI video applications.


# Touchless Smart Board — Transforming Digital Learning with QIM SDK
Source: https://imsdkdocs.qualcomm.com/blogs/smart-board-application

A two-stage palm detection and hand landmark pipeline turns hand gestures into a real-time, touchless virtual whiteboard, built with QIM SDK.

<div>
  <video alt="Smart Board Application" />

  <div>
    QIM SDK · Qualcomm
  </div>
</div>

<div>
  <div>
    Gesture Recognition
  </div>

  <p>
    A touchless virtual whiteboard controlled entirely by hand gestures, built on a two-stage
    palm detection and hand landmark pipeline running on Qualcomm QIM SDK with QNN HTP acceleration.
  </p>

  <div>
    <span>QIM SDK Team</span>
    <span>·</span>
    <span>Aug 3, 2026</span>
    <span>·</span>
    <a href="/blogs">← All posts</a>
  </div>
</div>

<hr />

## Introduction

Touchless interaction is becoming increasingly important in edge AI applications. Smart classrooms, conference rooms, digital collaboration platforms, and accessibility tools can all benefit from interfaces that eliminate the need for physical controls. The Smartboard Application demonstrates this concept by transforming a live USB camera feed into a real-time virtual whiteboard that users can write on, erase, and manipulate entirely through hand gestures.

The application uses the same two-stage hand-tracking architecture found in the QIM SDK's gesture pipelines. A palm detection model first identifies the hand, after which the detected region is processed by a hand-landmark model that extracts 21 keypoints. Both models run through the QNN HTP delegate, keeping AI inference off the CPU and maintaining a responsive, real-time whiteboard experience.

Unlike a conventional detection demo focused primarily on visual overlays, Smartboard uses landmark metadata to support a broad range of interactive features. Finger positions are analyzed frame by frame to enable mode switching, freehand drawing, shape recognition, erasing, and even grabbing and repositioning existing content on the board.

## Use Case Overview

<Steps>
  <Step title="Two-Stage Hand Tracking">
    Palm detection locates the hand in the frame; the detected region is converted into a region of interest and passed to the hand-landmark model, which extracts 21 keypoints including the fingertips and wrist.
  </Step>

  <Step title="Gesture Interpretation">
    Finger positions determine the active mode and action. An open palm enters Letter mode, a closed fist enters Shape mode, the index finger draws, the pinky erases, and a thumb-to-index pinch grabs existing content.
  </Step>

  <Step title="Persistent Canvas">
    A Cairo-rendered canvas accumulates freehand strokes and recognized shapes across frames, so drawn content persists on screen rather than being redrawn from scratch each frame.
  </Step>

  <Step title="Shape Recognition">
    Rough hand-drawn strokes made in Shape mode are analyzed and, when they match a supported pattern, converted into cleaner circles, rectangles, lines, or arrows.
  </Step>

  <Step title="Object Movement">
    A pinch gesture near an existing word or shape grabs that object, letting the user drag it to a new position on the board before releasing the pinch.
  </Step>
</Steps>

## Pipeline diagram

<img alt="Smart Board Pipeline" />

## Elements used in pipeline

| Element | Description |
| - | - |
| `v4l2src` | Captures raw frames from the USB camera. |
| [`qtivtransform`](../plugin-reference/qtivtransform) | Applies a horizontal flip so the camera feed behaves like a mirror, matching the player's natural hand motion. |
| [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | Hardware-accelerated resize, color-space conversion, and normalization ahead of each inference stage, used in image-batch mode for palm detection and ROI-batch mode for hand landmarks. |
| [`qtimltflite`](../plugin-reference/qtimltflite) | Runs the palm detection and hand landmark TFLite models on the QNN HTP delegate. |
| [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | Decodes raw model output into structured detections, using the `palmd` module for palm detection and `hlandmark` for hand landmarks. |
| [`qtimetamux`](../plugin-reference/qtimetamux) | Synchronizes inference metadata from each stage with the corresponding video frame, used once after palm detection and again after hand landmark detection. |
| `qtimetatransform` | Converts palm-detection metadata into a hand region of interest via the `roi-palmd` module, so the landmark model focuses only on the detected hand. |
| [`qtivoverlay`](../plugin-reference/qtivoverlay) | Draws the QIM SDK ML metadata overlay onto the video stream. |
| [`qtimlmetaparser`](../plugin-reference/qtimetaparser) | Parses the final hand landmark metadata into JSON and exposes it to the application through an `appsink` callback. |
| `cairooverlay` | Renders the persistent Smartboard canvas, including freehand strokes, recognized shapes, live previews, eraser cursor, drag indicators, and board-mode status. |
| [`waylandsink`](../plugin-reference/waylandsink) | Displays the final composited output. |

## How it works

<Steps>
  <Step title="Capture and Mirror">
    The USB camera feed is flipped horizontally and normalized to a fixed resolution and frame rate, so on-screen motion matches the user's real-world hand movement.
  </Step>

  <Step title="Palm Detection and ROI Transform">
    A `tee` splits the stream so one branch runs palm detection while the other carries the raw video into a metadata muxer. Once merged, `qtimetatransform` uses the `roi-palmd` module to turn the detected palm region into a focused crop for the next stage.
  </Step>

  <Step title="Hand Landmark Detection">
    The stream splits again after the ROI transform. The landmark branch preprocesses the cropped region in ROI-batch mode and runs the hand-landmark model through the QNN HTP delegate, producing 21 keypoints per detected hand.
  </Step>

  <Step title="Metadata Parsing and Gesture Handling">
    After a second metadata mux merges the landmark data with the video stream, the pipeline splits into a display branch and a metadata branch. The metadata branch parses the landmarks to JSON and delivers them to an application-side gesture handler through an `appsink` callback.
  </Step>

  <Step title="Gesture-to-Action Mapping">
    The gesture handler tracks which fingers are extended to decide the active mode and action: an open palm switches to Letter mode, a closed fist switches to Shape mode, the index finger draws, the pinky erases nearby strokes, and a thumb-to-index pinch grabs the nearest word or shape for dragging.
  </Step>

  <Step title="Canvas Rendering">
    The display branch draws the QIM SDK metadata overlay first, then hands off to a Cairo drawing callback that renders the persistent whiteboard canvas, including committed strokes, recognized shapes, live drawing previews, the eraser cursor, and drag indicators, before the composited frame reaches the display.
  </Step>
</Steps>

## Setup Requirements

### Hardware

<img alt="HW Setup" />

| Component | Description |
| - | - |
| **Edge Device** | RB3 Gen 2, IQ8, or IQ9. Runs the hand-tracking inference and gesture logic. |
| **Camera Source** | USB camera positioned to capture the user's hand and drawing surface area. |
| **HDMI Display Monitor** | Connected to the edge device to show the live whiteboard output. |

### Software

Flash your Qualcomm Edge device by following the device setup and flashing instructions [here](../installation), then install the Python and GStreamer prerequisites (`python3`, `gstreamer1.0`, `python3-gi`) needed to run the pipeline.

The palm detection and hand landmark TFLite models, along with their label and settings files, are the same assets used by the [hand gesture recognition pipeline](/blogs/smart-hand-gesture-recognition). See that post's model download steps for the palm and hand landmark model files. The application performs a runtime validation check for all required model and config files before starting the pipeline, and prints a clear error if any path is missing.

<Accordion title="Try me">
  <Tabs>
    <Tab title="C++">
      <Info>
        Check application source code on GitHub: [`demo_smartboard/main.cc`](https://github.com/qualcomm/qimsdk/blob/main/cpp/examples/demo-apps/demo_smartboard/main.cc)

        Pre-built application on device: `/usr/bin/qimsdk_demo_smartboard`
      </Info>

      #### Download Required Files

      Download the gesture recognizer task bundle from Google MediaPipe to obtain the palm detection and hand landmark models:

      ```bash theme={null}
      # Download the gesture recognizer task bundle
      wget https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/latest/gesture_recognizer.task

      # Extract the top-level task
      unzip gesture_recognizer.task

      # Extract hand landmarker models
      unzip hand_landmarker.task
      # save hand_detector.tflite as palm_detection_full.tflite
      # save hand_landmarks_detector.tflite as hand_landmark_full.tflite
      ```

      | File | Save as |
      | - | - |
      | Palm detection model (see steps above) | palm\_detection\_full.tflite |
      | Hand landmark model (see steps above) | hand\_landmark\_full.tflite |
      | <a href="../labels/palmd_labels.json">palmd\_labels.json</a> | palmd\_labels.json |
      | <a href="../labels/palmd_settings.json">palmd\_settings.json</a> | palmd\_settings.json |
      | <a href="../labels/hlandmarks.json">hlandmarks.json</a> | hlandmarks.json |
      | <a href="../labels/hlandmark_settings.json">hlandmark\_settings.json</a> | hlandmark\_settings.json |

      <Note>
        If a downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
      </Note>

      <Steps>
        <Step title="Copy Files to Device">
          <CodeGroup>
            ```bash SCP (SSH) theme={null}
            # Replace <user> and <device-ip> with your device credentials.
            ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
            scp palm_detection_full.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp hand_landmark_full.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp palmd_labels.json          <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            scp palmd_settings.json        <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            scp hlandmarks.json            <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            scp hlandmark_settings.json    <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            ```
          </CodeGroup>
        </Step>

        <Step title="Connect a USB camera">
          Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the `--input-config` value below if your camera is exposed on a different node).
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          /usr/bin/qimsdk_demo_smartboard --input-config /dev/video2
          ```

          The application tracks hand and fingertip position from the live USB camera feed, letting the user draw, recognize shapes, erase, and drag/move strokes on a virtual whiteboard overlaid on the camera feed.

          To stop the application, press **CTRL + C**.
        </Step>
      </Steps>
    </Tab>

    <Tab title="Python">
      <Info>
        Check application source code on GitHub: [`qimsdk_demo_smartboard.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/demo-apps/qimsdk_demo_smartboard.py)

        Pre-built application on device: `/usr/bin/qimsdk_demo_smartboard.py`
      </Info>

      #### Download Required Files

      Download the gesture recognizer task bundle from Google MediaPipe to obtain the palm detection and hand landmark models:

      ```bash theme={null}
      # Download the gesture recognizer task bundle
      wget https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/latest/gesture_recognizer.task

      # Extract the top-level task
      unzip gesture_recognizer.task

      # Extract hand landmarker models
      unzip hand_landmarker.task
      # save hand_detector.tflite as palm_detection_full.tflite
      # save hand_landmarks_detector.tflite as hand_landmark_full.tflite
      ```

      | File | Save as |
      | - | - |
      | Palm detection model (see steps above) | palm\_detection\_full.tflite |
      | Hand landmark model (see steps above) | hand\_landmark\_full.tflite |
      | <a href="../labels/palmd_labels.json">palmd\_labels.json</a> | palmd\_labels.json |
      | <a href="../labels/palmd_settings.json">palmd\_settings.json</a> | palmd\_settings.json |
      | <a href="../labels/hlandmarks.json">hlandmarks.json</a> | hlandmarks.json |
      | <a href="../labels/hlandmark_settings.json">hlandmark\_settings.json</a> | hlandmark\_settings.json |

      <Note>
        If a downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
      </Note>

      <Steps>
        <Step title="Copy Files to Device">
          <CodeGroup>
            ```bash SCP (SSH) theme={null}
            # Replace <user> and <device-ip> with your device credentials.
            ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
            scp palm_detection_full.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp hand_landmark_full.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp palmd_labels.json          <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            scp palmd_settings.json        <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            scp hlandmarks.json            <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            scp hlandmark_settings.json    <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            ```
          </CodeGroup>
        </Step>

        <Step title="Connect a USB camera">
          Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the `--input-config` value below if your camera is exposed on a different node).
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          python3 /usr/bin/qimsdk_demo_smartboard.py --input-config /dev/video2
          ```

          The application tracks hand and fingertip position from the live USB camera feed, letting the user draw, recognize shapes, erase, and drag/move strokes on a virtual whiteboard overlaid on the camera feed.

          To stop the application, press **CTRL + C**.
        </Step>
      </Steps>
    </Tab>
  </Tabs>
</Accordion>

## Application Function Breakdown

The Smartboard application is organized into a handful of function groups, each responsible for one stage of the gesture-to-canvas flow:

* **Runtime and state management**: validates that the palm and hand-landmark models, labels, and settings files are present before the pipeline starts, and maintains a state object tracking freehand strokes, recognized shapes, board mode, and active drag/erase state.
* **Metadata parsing**: extracts the 21 hand landmark points from the raw metadata stream and converts them into a simple keypoint map that the gesture logic can consume without caring about the underlying JSON structure.
* **Gesture detection**: inspects which fingers are extended to classify the current hand pose as an open palm, closed fist, or pinch, and applies stability checks so the board doesn't flicker between modes from noisy frame-to-frame landmark jitter.
* **Drawing and shape recognition**: turns index-finger movement into smoothed freehand strokes, and analyzes completed strokes in Shape mode to snap them into clean circles, rectangles, lines, or arrows when they match a recognized pattern.
* **Erasing**: measures the pinky tip's distance to existing strokes and shapes, and removes anything that falls within the eraser radius.
* **Drag and move**: checks whether a pinch point lands near an existing shape or word, groups nearby stroke segments so a whole word moves together, and updates the object's position as the hand moves until the pinch releases.
* **Rendering**: draws the committed strokes, clean shapes, live previews, eraser cursor, drag indicator, and board-mode status onto the canvas each frame.
* **Pipeline setup**: assembles the full two-stage QIM SDK pipeline (camera capture, palm detection, ROI transform, hand landmark detection, metadata parsing, and the display/canvas branches) and verifies that all required GStreamer and QIM SDK plugins are available before launch.

## Expected Output

<video alt="Smart Board Application" />

Running the application opens a live view on the connected monitor:

* The camera feed is shown with hand tracking overlaid, including the palm detection box and hand landmark points.
* On top of that, the Smartboard canvas renders drawing strokes, shape previews, finalized clean shapes, the eraser cursor, drag indicators, and the current board mode.

The Smartboard supports the following real-time interactions:

* **Letter mode**: an open palm switches the board to freehand writing with the index finger.
* **Shape mode**: a closed fist switches the board to shape drawing, converting rough strokes into circles, rectangles, lines, or arrows.
* **Erasing**: the pinky finger removes nearby strokes or shapes.
* **Object movement**: a pinch gesture near existing content grabs and repositions it.

Press `Ctrl+C` to stop the application; the pipeline transitions to the `NULL` state and releases the camera and display cleanly.

## Conclusion

The Smartboard Application demonstrates how metadata-driven application logic can extend well beyond traditional overlay rendering. Instead of using hand-landmark keypoints solely to draw bounding boxes, it transforms them into the primary control mechanism for a real-time, interactive whiteboard. QNN HTP acceleration keeps both inference stages efficient enough to support smooth freehand writing and responsive interaction.

This approach can be adapted to a wide range of touchless interfaces, including smart classrooms, meeting-room collaboration tools, accessibility solutions, and applications where natural hand gestures replace a mouse, stylus, or touchscreen.


# Building a Real-Time Hand Gesture Recognition AI Pipeline Using the QIM SDK
Source: https://imsdkdocs.qualcomm.com/blogs/smart-hand-gesture-recognition

A four-stage sequential AI pipeline — palm detection, hand landmark estimation, gesture embedding, and gesture classification — built with Qualcomm QIM SDK for real-time edge deployment.

<div>
  <video />
</div>

<div>
  QIM SDK · Qualcomm
</div>

<div>
  <div>
    Computer Vision
  </div>

  <p>
    Build a four-stage real-time hand gesture recognition pipeline using Qualcomm QIM SDK — covering
    palm detection, hand landmark estimation, gesture embedding, and gesture classification, all
    running on-device with hardware-accelerated inference.
  </p>

  <div>
    <span>QIM SDK Team</span>
    <span>·</span>
    <span>Jun 10, 2026</span>
    <span>·</span>
    <a href="/blogs">← All posts</a>
  </div>
</div>

<hr />

## Introduction

Real-time gesture recognition is redefining how humans interact with machines — enabling touchless, natural interfaces across augmented reality, gaming, accessibility, and industrial control. But accurately interpreting hand gestures demands more than simple object detection; it requires a multi-stage AI pipeline capable of progressively refining raw visual input into high-level, actionable intent.

The QIM SDK brings this capability directly to the edge. By routing compute-intensive tasks through hardware-accelerated GStreamer plugins, the SDK offloads video decoding, frame preparation, multi-stage inference, and encoding entirely to dedicated hardware blocks — delivering low-latency, power-efficient execution even for complex multi-model workloads.

At the core of this use case is a **four-stage sequential inference pipeline**:

<CardGroup>
  <Card title="Stage 1" icon="hand">
    Palm Detection
  </Card>

  <Card title="Stage 2" icon="circle-nodes">
    Hand Landmark Estimation
  </Card>

  <Card title="Stage 3" icon="diagram-project">
    Gesture Embedding
  </Card>

  <Card title="Stage 4" icon="tag">
    Gesture Classification
  </Card>
</CardGroup>

Each stage builds directly on the output of the previous, progressively transforming raw video frames into structured, high-level gesture data. Between stages, intermediate transformations — such as rotating and cropping detected hand regions for correct spatial alignment — ensure consistent, accurate results across varying hand positions and orientations.

Metadata is **hierarchically structured** throughout the pipeline. Each detected hand establishes a root metadata entry, with landmark detections, embeddings, and gesture classifications attached as child metadata linked to the originating detection. Visualization is fully hardware-accelerated: bounding boxes, key-points, and gesture labels are composited directly onto video frames via optimized overlay rendering.

The pipeline accepts input from USB cameras, RTSP streams, ISP (on-device) cameras, and local video files, and delivers results through real-time on-screen visualization or remote streaming over RTSP and WebRTC — with structured inference metadata transmitted in parallel.

<Info>
  Check application source code on GitHub: [`gst-gesture-recognition`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-gesture-recognition)
</Info>

## Use Case Overview

<Steps>
  <Step title="Source">
    The pipeline accepts continuous video input from a USB camera, RTSP stream, ISP (on-device) camera, or local video file.
  </Step>

  <Step title="Palm Detection">
    Each incoming frame is processed by the palm detection model, which identifies the presence, location, and rotation angle of any hands in the scene.
  </Step>

  <Step title="Hand Landmark Detection">
    For each detected hand, the landmark model identifies 21 key points — fingertips, joints, and wrist — capturing the full structural pose. The hand region is first rotated and cropped via affine transformation for correct spatial alignment.
  </Step>

  <Step title="Gesture Embedding">
    The detected landmarks are encoded into a compact numerical representation that summarizes the hand pose in a form optimized for classification.
  </Step>

  <Step title="Gesture Classification">
    The embedding is passed to the gesture classification model, which maps the hand pose to a predefined gesture label — such as open hand, fist, or thumbs up.
  </Step>

  <Step title="Metadata Synchronization">
    Results from all inference stages are structured hierarchically. [`qtimetamux`](../plugin-reference/qtimetamux) synchronizes this structured metadata with the original video frames, maintaining per-frame consistency throughout the pipeline.
  </Step>

  <Step title="Output">
    Annotated frames are H.264-encoded and delivered over RTSP or WebRTC, with structured inference metadata transmitted in parallel as a JSON stream.
  </Step>
</Steps>

## Pipeline diagram

<img alt="Hand Gesture Recognition Pipeline" />

## Elements used in pipeline

| Element | Description |
| - | - |
| `source` | Accepts video input from a USB camera, ISP camera, RTSP stream, or local video file. |
| `tee` | Splits the stream into multiple parallel branches for simultaneous processing. |
| [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | Hardware-accelerated resize, YUV→RGB conversion, and pixel normalization to meet each model's input requirements. |
| [`qtimltflite`](../plugin-reference/qtimltflite) | Executes TFLite inference models, producing raw output tensors. |
| [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | Decodes raw tensors into structured bounding boxes, keypoints, labels, and confidence scores via dynamically loaded modules. |
| [`qtimetamux`](../plugin-reference/qtimetamux) | Synchronizes inference results with the original video stream as structured per-frame metadata. |
| `qtimetatransform` | Transforms metadata as it flows through the pipeline — modifying coordinate systems to ensure compatibility with downstream elements. |
| [`qtivoverlay`](../plugin-reference/qtivoverlay) | Composites bounding boxes, keypoints, and labels onto video frames using hardware-accelerated overlay rendering. |
| [`qtimlmetaparser`](../plugin-reference/qtimetaparser) | Serializes per-frame inference metadata into JSON for integration with external systems. |
| [`v4l2h264enc`](../plugin-reference/v4l2h264enc) / `h264parse` | Hardware-accelerated H.264 encoding of the processed video stream. |
| [`waylandsink`](../plugin-reference/waylandsink) | Renders the output to the local display via the Wayland compositor. |

## How it works

<Steps>
  <Step title="Stage 1 — Palm Detection">
    The first model processes the full video frame, identifying the location, orientation, and bounding box of each detected hand. This stage also produces key points used to estimate the hand's rotation angle. A `qtimetatransform` element uses this information to compute an **affine transformation matrix**, which is attached as metadata and carried forward to the next stage.
  </Step>

  <Step title="Affine Crop Generation">
    The hand landmark model requires a normalized, upright view of the hand — not raw bounding box coordinates. A dedicated [`qtimlvconverter`](../plugin-reference/qtimlvconverter) instance consumes the affine transformation matrix and applies it to crop and rotate the detected hand region from the original frame, producing a correctly aligned input for the next stage.
  </Step>

  <Step title="Stage 2 — Hand Landmark Detection">
    The landmark model is invoked once per detected hand, processing the cropped and aligned region to produce **21 keypoints** — fingertips, joints, and wrist — along with handedness and confidence scores. A post-processing step then applies a reverse transformation to remap the keypoints accurately back onto the original frame coordinate space.
  </Step>

  <Step title="Stage 3 — Gesture Embedding">
    The landmark keypoints are passed to the gesture embedder, which encodes the hand pose into a compact numerical embedding vector. This stage operates directly on tensor data — no image cropping or geometric transformation is required.
  </Step>

  <Step title="Stage 4 — Gesture Classification">
    The embedding vector is passed to the gesture classifier, which maps the hand pose representation to a predefined gesture label — such as open hand, fist, or thumbs up. Like the embedder, this stage operates purely on tensors with no additional preprocessing.
  </Step>

  <Step title="Hierarchical Metadata">
    To preserve logical relationships across all four stages, the pipeline employs a **hierarchical metadata model** based on unique IDs and parent IDs. The palm detection stage creates a **root metadata entry** for each detected hand; each subsequent stage — landmark detection, embedding, and classification — attaches its results as **child metadata** referencing the parent. This ensures every classified gesture is explicitly traceable to its originating hand detection.
  </Step>

  <Step title="Output">
    Annotated frames with bounding boxes, keypoints, and gesture labels are rendered via hardware-accelerated overlay. The H.264-encoded stream is delivered over RTSP or WebRTC, with structured inference metadata transmitted in parallel as a JSON stream.
  </Step>
</Steps>

## Run application on device

### Setup Requirements

#### Hardware

<img alt="HW Setup" />

| Component | Description |
| - | - |
| **Edge Device** | RB3 Gen 2, IQ8, or IQ9 — Primary processing unit for AI inference and video composition. |
| **Camera Source** | IP/RTSP camera, ISP (on-device) camera, or USB camera. A local file source may be substituted if no physical camera is available. |
| **HDMI Display Monitor** | Connected to the edge device for rendering and visualizing pipeline output. |
| **PoE Switch** | Powers IP/RTSP cameras and provides network connectivity over a single Ethernet cable per camera. (Required for IP/RTSP camera setups only.) |
| **Local Network** | Ensures the edge device, RTSP camera, and host machine are reachable on the same network. (Required when using RTSP camera input or streaming results via RTSP or WebRTC.) |

#### Software

**Flash your Qualcomm Edge device** by following the device setup and flashing instructions [here](../installation)

**Once your device is ready**, follow the instructions below to set up the Gesture Recognition AI Pipeline:

##### AI Model and config files

Download the gesture recognizer models from Google MediaPipe:

```bash theme={null}
# Download the gesture recognizer task bundle
wget https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/latest/gesture_recognizer.task

# Extract the top-level task
unzip gesture_recognizer.task

# Extract hand landmarker models
unzip hand_landmarker.task
# save hand_detector.tflite as palm_detection.tflite
# save hand_landmarks_detector.tflite as hand_landmark.tflite

# Extract gesture recognizer models
unzip hand_gesture_recognizer.task
# → gesture_embedder.tflite, canned_gesture_classifier.tflite
```

<Note>
  These are FLOAT precision models.
</Note>

| File | Download | Save as |
| - | - | - |
| Palm detection model | See download steps above | `palm_detection.tflite` |
| Hand landmark model | See download steps above | `hand_landmark.tflite` |
| Gesture embedder model | See download steps above | `gesture_embedder.tflite` |
| Gesture classifier model | See download steps above | `canned_gesture_classifier.tflite` |
| Palm detection labels | <a href="../labels/palmd_labels.json">palmd\_labels.json</a> | `palmd_labels.json` |
| Palm detection settings | <a href="../labels/palmd_settings.json">palmd\_settings.json</a> | `palmd_settings.json` |
| Hand landmark labels | <a href="../labels/hlandmark_labels.json">hlandmark\_labels.json</a> | `hlandmark_labels.json` |
| Hand landmark settings | <a href="../labels/hlandmark_settings.json">hlandmark\_settings.json</a> | `hlandmark_settings.json` |
| Gesture labels | <a href="../labels/gesture_labels.json">gesture\_labels.json</a> | `gesture_labels.json` |
| Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/gesture_sample.mp4">Input video</a> | `gesture_sample.mp4` |

**Copy files to device**

<CodeGroup>
  ```bash SCP (SSH) theme={null}
  # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

  ssh <user>@<device-ip> "mkdir -p ~/{models,labels,media}"
  scp palm_detection.tflite              <user>@<device-ip>:~/models/
  scp hand_landmark.tflite               <user>@<device-ip>:~/models/
  scp gesture_embedder.tflite            <user>@<device-ip>:~/models/
  scp canned_gesture_classifier.tflite   <user>@<device-ip>:~/models/
  scp palmd_labels.json                  <user>@<device-ip>:~/labels/
  scp palmd_settings.json                <user>@<device-ip>:~/labels/
  scp hlandmark_labels.json              <user>@<device-ip>:~/labels/
  scp hlandmark_settings.json            <user>@<device-ip>:~/labels/
  scp gesture_labels.json                <user>@<device-ip>:~/labels/
  scp gesture_sample.mp4                  <user>@<device-ip>:~/media/
  ```
</CodeGroup>

**Connect to device**

```bash theme={null}
ssh <user>@<device-ip>
```

**Run the Gesture Recognition Application**

<Note>
  A display must be connected to the device. If no display is available, use the `--no-display` flag to run in headless mode.
</Note>

Use the following base path for model and label files based on your OS:

<AccordionGroup>
  <Accordion title="Display only">
    ```bash theme={null}
    gst-gesture-recognition \
      --input-type=file \
      --input-config=$HOME/media/gesture_sample.mp4
    ```
  </Accordion>

  <Accordion title="WebRTC output">
    ```bash theme={null}
    gst-gesture-recognition \
      --input-type=file \
      --input-config=$HOME/media/gesture_sample.mp4 \
      --output-type=webrtc \
      --output-config=wss://webrtc.nirbheek.in:8443 \
      --webrtc-id=1010
    ```
  </Accordion>

  <Accordion title="RTSP output">
    ```bash theme={null}
    gst-gesture-recognition \
      --input-type=file \
      --input-config=$HOME/media/gesture_sample.mp4 \
      --output-type=rtsp \
      --output-config=8900
    ```
  </Accordion>
</AccordionGroup>

> **Note:** This example uses an offline video file as input. To use an IP/RTSP camera or USB camera instead, update the `--input-type` argument accordingly — refer to the **Command-Line Options** section below for details.

It produces two key output results: an AI-annotated video stream and a JSON metadata stream. To visualize these results, refer to the **Host-Side Visualization** section below.

## Visualize the Results - Host-Side Visualization (Windows + WSL)

This section describes how to run the visualization client on a Windows host machine using **WSL (Windows Subsystem for Linux)**. The client renders the live video stream alongside a real-time AI metadata panel.

📥 The visualization client script can be downloaded here: <a href="../labels/rtsp_webrtc_client.zip">rtsp\_webrtc\_client.zip</a>

It displays:

* **Left panel** — Live video stream with AI overlays (bounding boxes, keypoints, gesture labels)
* **Right panel** — Real-time AI metadata (JSON): object detections, bounding boxes, and confidence scores.

**Step 1 — Install WSL and Ubuntu**

If WSL is not already installed, run the following from a Windows terminal:

```bash theme={null}
wsl --install Ubuntu-24.04
```

Once installed, open the Ubuntu terminal and update the system:

```bash theme={null}
sudo apt update && sudo apt upgrade -y
```

**Step 2 — Install System Dependencies**

The visualization script requires GStreamer and Python GObject Introspection (GI) bindings. Install all required packages with:

```bash theme={null}
sudo apt install -y \
  python3 python3-pip python3-gi python3-gi-cairo \
  gir1.2-gstreamer-1.0 \
  gir1.2-gst-plugins-base-1.0 \
  gir1.2-gst-plugins-bad-1.0 \
  gstreamer1.0-tools \
  gstreamer1.0-plugins-base \
  gstreamer1.0-plugins-good \
  gstreamer1.0-plugins-bad \
  gstreamer1.0-plugins-ugly \
  gstreamer1.0-libav \
  python3-websocket \
  libnice10 \
  libnice-dev \
  gstreamer1.0-nice
```

**Step 3 — Run the Visualization Client Script**

Navigate to the directory containing the script and run:

<AccordionGroup>
  <Accordion title="RTSP">
    ```bash theme={null}
    python3 rtsp_webrtc_client.py rtsp://<DEVICE_IP>:8900/live
    ```
  </Accordion>

  <Accordion title="WebRTC">
    ```bash theme={null}
    python3 rtsp_webrtc_client.py --source webrtc --signalling-server wss://webrtc.nirbheek.in:8443 --peer-id 1010
    ```
  </Accordion>
</AccordionGroup>

**Step 4 — Expected Output**

Once the client connects, the UI will display:

| Panel | Content |
| - | - |
| Left | Real-time decoded video stream with gesture overlays |
| Right | Live AI metadata — detected hands, keypoints, and gesture labels |

After following the steps, the video and metadata streams should be up and running.

Gesture Recognition AI Pipeline is configured for single-hand gesture recognition — the most common interaction scenario, where one dominant gesture is performed at a time. This keeps computational overhead low while maintaining reliable detection and classification. When multiple hands are present, the model prioritizes the detection with the highest confidence score, ensuring stable and accurate gesture recognition even in dynamic conditions.

<img alt="Expected Output" />

The pipeline generates structured JSON metadata in the following format:

```bash theme={null}
{
  "object_detection": [
    {
      "label": "palm",
      "confidence": 85.94279479980469,
      "color": 16711935,
      "rectangle": {
        "x": 0.3484375,
        "y": 0.15555555555555556,
        "width": 0.22708333333333333,
        "height": 0.40370370370370373
      },
      "xtraparams": {
        "affine-matrix": [
          0.42060701741537004,
          1.2300771264034409,
          38.53241177825402,
          -1.2300771264034409,
          0.42060701741537004,
          1313.4940252711156,
          0.0,
          0.0,
          1.0
        ]
      },
      "video_landmarks": [
        {
          "keypoints": [
            { "keypoint": "wrist", "x": 0.5411458333333333, "y": 0.42407407407407405, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "thumb cmc", "x": 0.5432291666666667, "y": 0.3648148148148148, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "thumb mcp", "x": 0.5359375, "y": 0.3037037037037037, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "thumb ip", "x": 0.5348958333333333, "y": 0.24537037037037038, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "thumb tip", "x": 0.5395833333333333, "y": 0.1925925925925926, "confidence": 99.70703125, "color": 16711935 },

            { "keypoint": "index finger mcp", "x": 0.47760416666666666, "y": 0.32962962962962963, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "index finger pip", "x": 0.43854166666666666, "y": 0.29907407407407405, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "index finger dip", "x": 0.4109375, "y": 0.2814814814814815, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "index finger tip", "x": 0.3848958333333333, "y": 0.26666666666666666, "confidence": 99.70703125, "color": 16711935 },

            { "keypoint": "middle finger mcp", "x": 0.4635416666666667, "y": 0.37407407407407406, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "middle finger pip", "x": 0.41822916666666665, "y": 0.36203703703703705, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "middle finger dip", "x": 0.3880208333333333, "y": 0.35648148148148145, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "middle finger tip", "x": 0.36041666666666666, "y": 0.3537037037037037, "confidence": 99.70703125, "color": 16711935 },

            { "keypoint": "ring finger mcp", "x": 0.46197916666666666, "y": 0.41759259259259257, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "ring finger pip", "x": 0.4192708333333333, "y": 0.4212962962962963, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "ring finger dip", "x": 0.39114583333333336, "y": 0.425, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "ring finger tip", "x": 0.36614583333333334, "y": 0.42777777777777776, "confidence": 99.70703125, "color": 16711935 },

            { "keypoint": "pinky mcp", "x": 0.46979166666666666, "y": 0.45462962962962963, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "pinky pip", "x": 0.43802083333333336, "y": 0.475, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "pinky dip", "x": 0.41822916666666665, "y": 0.49074074074074076, "confidence": 99.70703125, "color": 16711935 },
            { "keypoint": "pinky tip", "x": 0.39947916666666666, "y": 0.5027777777777778, "confidence": 99.70703125, "color": 16711935 }
          ],
          "links": [
            { "start": 0, "end": 17 },
            { "start": 1, "end": 0 },
            { "start": 2, "end": 1 },
            { "start": 3, "end": 2 },
            { "start": 4, "end": 3 },
            { "start": 5, "end": 0 },
            { "start": 6, "end": 5 },
            { "start": 7, "end": 6 },
            { "start": 8, "end": 7 },
            { "start": 9, "end": 5 },
            { "start": 10, "end": 9 },
            { "start": 11, "end": 10 },
            { "start": 12, "end": 11 },
            { "start": 13, "end": 9 },
            { "start": 14, "end": 13 },
            { "start": 15, "end": 14 },
            { "start": 16, "end": 15 },
            { "start": 17, "end": 13 },
            { "start": 18, "end": 17 },
            { "start": 19, "end": 18 },
            { "start": 20, "end": 19 }
          ]
        }
      ],
      "image_classification": [
        {
          "label": "Open Palm",
          "confidence": 0.7060546875,
          "color": 4294902015
        }
      ]
    }
  ],
  "parameters": {
    "timestamp": "28356149027"
  }
}

```

Beyond the default setup, the application offers flexible input and output configurations that can be tailored via command-line options, as described below:

## Command-Line Options

<AccordionGroup>
  <Accordion title="--input-type">
    Selects the video input source for the pipeline.

    | Value | Description |
    | - | - |
    | `usb` | USB camera. Requires `--input-config=/dev/video0`. |
    | `isp` | Built-in ISP (on-device) camera. Optionally specify a camera ID via `--input-config=0`. |
    | `rtsp` | External IP/RTSP camera or stream. Requires `--input-config=rtsp://...`. |
    | `file` | Local H.264-encoded video file. Requires `--input-config=/path/to/gesture_sample.mp4`. |
  </Accordion>

  <Accordion title="--input-config">
    Specifies the input source configuration corresponding to the selected `--input-type`.

    | Input Type | Value |
    | - | - |
    | USB | `/dev/videoX` |
    | ISP | `<camera ID>` |
    | RTSP | `rtsp://<ip-or-url>` |
    | File | `/path/to/gesture_sample.mp4` |
  </Accordion>

  <Accordion title="--output-type">
    Defines how the processed output stream is delivered.

    | Value | Description |
    | - | - |
    | `none` | No video output (headless mode). |
    | `file` | Save encoded output to a file. Requires `--output-config=/path/to/output.mp4`. |
    | `rtsp` | Stream over RTSP. Requires `--output-config=<port>`. Access at `rtsp://<device-ip>:<port>/live`. |
    | `webrtc` | Stream over WebRTC. Requires `--output-config=ws://<signalling-server>:<port>`. |
  </Accordion>

  <Accordion title="--output-config">
    Specifies the output destination configuration corresponding to the selected `--output-type`.

    | Output Type | Value |
    | - | - |
    | File | `/path/to/output.mp4` |
    | RTSP | `<port>` |
    | WebRTC | `ws://<signalling-server>:<port>` |
  </Accordion>

  <Accordion title="--model-base-path">
    Root directory for model, label, and config files. The application resolves assets automatically:

    | Asset type | Resolved path |
    | - | - |
    | Model files (`*.tflite`) | `<base-path>/models/<file>` |
    | Label / settings files (`*.json`) | `<base-path>/labels/<file>` |

    ```bash theme={null}
    --model-base-path=/root        # QLI
    --model-base-path=/home/ubuntu # Ubuntu
    ```
  </Accordion>

  <Accordion title="--no-display">
    Disables local on-screen rendering. Recommended for headless deployments, remote streaming setups (RTSP/WebRTC), or performance optimization.
  </Accordion>

  <Accordion title="--width / --height / --framerate">
    Sets the raw input video resolution and frame rate. Applicable only to ISP and USB inputs.

    ```bash theme={null}
    --width=1920 --height=1080 --framerate=30
    ```
  </Accordion>

  <Accordion title="--webrtc-id">
    Specifies the local WebRTC signaling client ID used for peer connection setup with the signaling server.

    ```bash theme={null}
    --webrtc-id=1010
    ```
  </Accordion>
</AccordionGroup>

## JSON Metadata Output

The pipeline generates structured per-frame metadata. Each detected hand produces a root entry with child landmark, embedding, and classification results:

<Accordion title="Sample JSON output">
  ```json theme={null}
  {
    "object_detection": [
      {
        "label": "palm",
        "confidence": 85.94,
        "rectangle": {
          "x": 0.348, "y": 0.155,
          "width": 0.227, "height": 0.403
        },
        "xtraparams": {
          "affine-matrix": [
            0.4206, 1.2300, 38.53,
            -1.2300, 0.4206, 1313.49,
            0.0, 0.0, 1.0
          ]
        },
        "video_landmarks": [
          {
            "keypoints": [
              { "keypoint": "wrist",             "x": 0.541, "y": 0.424, "confidence": 99.7 },
              { "keypoint": "thumb tip",         "x": 0.539, "y": 0.192, "confidence": 99.7 },
              { "keypoint": "index finger tip",  "x": 0.384, "y": 0.266, "confidence": 99.7 },
              { "keypoint": "middle finger tip", "x": 0.360, "y": 0.353, "confidence": 99.7 },
              { "keypoint": "ring finger tip",   "x": 0.366, "y": 0.427, "confidence": 99.7 },
              { "keypoint": "pinky tip",         "x": 0.399, "y": 0.502, "confidence": 99.7 }
            ]
          }
        ],
        "image_classification": [
          { "label": "Open Palm", "confidence": 0.706 }
        ]
      }
    ],
    "parameters": { "timestamp": "28356149027" }
  }
  ```
</Accordion>

## Implementation Deep-Dive

<AccordionGroup>
  <Accordion title="1. Application Configuration and Runtime Context">
    The application separates user configuration from runtime state using two structs:

    ```c theme={null}
    typedef struct GstAppConfig {
      gchar *input_type;
      gchar *input_config;
      gchar *output_type;
      gchar *output_config;
      gchar *model_base_path;
      gboolean no_display;
      gint width, height, framerate, webrtc_id;
    } GstAppConfig;

    typedef struct GstAppContext {
      GstAppConfig config;
      GstElement *pipeline;
      GMainLoop  *mloop;
      GstElement *webrtc;
      gboolean    is_shutting_down;
    } GstAppContext;
    ```
  </Accordion>

  <Accordion title="2. Pipeline Assembly">
    The pipeline is composed of three independent branches: input, processing, and output. Construction order is deliberate — input first, output second, processing branch last.

    ```c theme={null}
    static gboolean gst_app_create_pipe (GstAppContext *appctx) {
      GstElement *input_tail = NULL, *output_head = NULL, *meta_head = NULL;
      appctx->pipeline = gst_pipeline_new ("gst-gesture-recognition");

      if (!gst_app_create_input_pipe  (appctx, &input_tail))  return FALSE;
      if (!gst_app_create_output_pipe (appctx, &output_head, &meta_head)) return FALSE;
      if (!gst_app_create_user_pipe   (appctx, input_tail, output_head, meta_head)) return FALSE;
      return TRUE;
    }
    ```
  </Accordion>

  <Accordion title="3. Multi-Stage Inference Model Configuration">
    Each stage is configured with a GPU-delegated `qtimltflite` instance and a task-specific `qtimlpostprocess` module:

    ```c theme={null}
    /* Stage 1 — Palm Detection */
    palm_inf = gst_app_make_element ("qtimltflite", "palm_inf");
    gst_element_set_enum_property (palm_inf, "delegate", "gpu");
    g_object_set (palm_inf, "model", "<base>/models/palm_detection.tflite", NULL);

    palm_post = gst_app_make_element ("qtimlpostprocess", "palm_post");
    gst_element_set_enum_property (palm_post, "module", "palmd");
    g_object_set (palm_post, "results", 1, NULL);

    /* Stage 2 — Hand Landmark */
    hand_pre = gst_app_make_element ("qtimlvconverter", "hand_pre");
    gst_element_set_enum_property (hand_pre, "mode", "roi-batch-non-cumulative");
    hand_inf = gst_app_make_element ("qtimltflite", "hand_inf");
    gst_element_set_enum_property (hand_inf, "delegate", "gpu");

    hand_post = gst_app_make_element ("qtimlpostprocess", "hand_post");
    gst_element_set_enum_property (hand_post, "module", "hlandmark");
    g_object_set (hand_post, "results", 6, NULL);

    /* Stage 3 — Gesture Embedding */
    gesture_pre = gst_app_make_element ("qtimlpostprocess", "gesture_pre");
    gst_element_set_enum_property (gesture_pre, "module", "tensor");
    gesture_embed = gst_app_make_element ("qtimltflite", "gesture_embed");
    gst_element_set_enum_property (gesture_embed, "delegate", "gpu");

    /* Stage 4 — Gesture Classification */
    gesture_class = gst_app_make_element ("qtimltflite", "gesture_class");
    gst_element_set_enum_property (gesture_class, "delegate", "gpu");
    gesture_post = gst_app_make_element ("qtimlpostprocess", "gesture_post");
    gst_element_set_enum_property (gesture_post, "module", "mobilenet");
    g_object_set (gesture_post, "results", 8, NULL);
    ```
  </Accordion>

  <Accordion title="4. Linking Palm and Hand Landmark Branches">
    ```c theme={null}
    /* input → tee1 → metamux1 (passthrough) + palm detection */
    gst_element_link (input_tail, tee1);
    gst_element_link_many (tee1, q, metamux1, metatransform, tee2, NULL);
    gst_element_link_many (tee1, q, palm_pre, palm_inf, palm_post, caps1, metamux1, NULL);

    /* tee2 → overlay branch + hand landmark branch */
    gst_element_link_many (tee2, q, metamux2, overlay, tee3, NULL);
    gst_element_link_many (tee2, q, hand_pre, hand_inf, tee4, NULL);
    gst_element_link_many (tee4, q, hand_post, caps2, metamux2, NULL);
    ```
  </Accordion>

  <Accordion title="5. Gesture Branch and Output">
    ```c theme={null}
    /* tee4 → embedding → classification → metamux2 */
    gst_element_link_many (tee4, q,
        gesture_pre, gesture_embed, gesture_class, gesture_post,
        caps3, metamux2, NULL);

    /* tee3 → video output */
    gst_element_link_many (tee3, q, output_head, NULL);

    /* tee3 → metadata output (optional) */
    if (meta_head != NULL)
        gst_element_link_many (tee3, q, parser, meta_head, NULL);
    ```
  </Accordion>

  <Accordion title="6. WebRTC Signaling">
    WebRTC communication uses explicit SDP offer/answer exchange and ICE candidate negotiation via WebSocket using `libsoup`:

    ```c theme={null}
    /* Create data channel for metadata */
    g_signal_emit_by_name (webrtcbin, "create-data-channel", name, NULL, &ch);

    /* Send SDP offer */
    GstPromise *promise = gst_promise_new_with_change_func (on_offer_created, appctx, NULL);
    g_signal_emit_by_name (webrtcbin, "create-offer", NULL, promise);

    /* ICE candidates */
    g_signal_connect (appctx->webrtc, "on-ice-candidate",
        G_CALLBACK (on_webrtc_ice_candidate), appctx);
    ```

    | Callback | Responsibility |
    | - | - |
    | `on_offer_created` | Constructs and sends the SDP offer to the remote peer |
    | `on_ice_candidate` | Transmits ICE candidates to the signaling server |
    | `on_ws_message` | Handles incoming signaling messages from the WebSocket |
  </Accordion>
</AccordionGroup>

## Build the Application

* **Source code:** [gst-gesture-recognition](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-gesture-recognition)
* **Build instructions:** [Steps to build custom application](../advanced/yocto-build#steps-to-build-custom-application)

## Conclusion

The QIM SDK's modular architecture enables developers to compose intelligent video analytics pipelines with speed and flexibility. Models run in parallel or sequentially, with each stage's output automatically attached to the corresponding video frame as structured metadata. Decoupled video and inference processing allows multiple models to execute concurrently — maximizing throughput without sacrificing accuracy — with all results unified through a single GStreamer element for clean, scalable integration.

Whether building touchless interfaces or gesture-driven applications, the QIM SDK provides a solid, production-ready foundation for advanced edge AI.


# Building a Wrist-Controlled Car Game Using QIM SDK
Source: https://imsdkdocs.qualcomm.com/blogs/touchless-wrist-steering-car-game

A two-stage palm detection and hand landmark pipeline turns wrist movement into a virtual steering wheel for a real-time, touchless split-screen driving game, built with QIM SDK

<div>
  <video alt="Smart Board Application" />

  <div>
    QIM SDK · Qualcomm
  </div>
</div>

<div>
  <div>
    Gesture Recognition
  </div>

  <p>
    A split-screen touchless driving game controlled entirely by wrist movement, built on a two-stage
    palm detection and hand landmark pipeline running on Qualcomm QIM SDK with QNN HTP acceleration.
  </p>

  <div>
    <span>QIM SDK Team</span>
    <span>·</span>
    <span>Aug 3, 2026</span>
    <span>·</span>
    <a href="/blogs">← All posts</a>
  </div>
</div>

<hr />

## Introduction

"TouchlessDrive" is a touchless car game that turns a player's raised hands into a virtual steering wheel. Using a single USB camera, the system tracks both wrists and interprets the line between them as the steering input. When the player tilts their hands left or right, the on-screen car responds in real time. The experience is displayed in a split-screen layout, with the live camera feed shown alongside the game interface.

Under the hood, TouchlessDrive runs on a two-stage hand-tracking pipeline built with the Qualcomm QIM SDK. First, a palm detection model identifies each hand in the frame. Then, a hand-landmark model extracts precise keypoints, including the wrist locations used for steering. Both models run through the QNN TFLite delegate, offloading inference to the on-device HTP instead of the CPU. This helps keep the game loop responsive and capable of real-time performance.

The project highlights how the QIM SDK's metadata pipeline can power interactive application logic beyond traditional video analytics. In TouchlessDrive, AI-generated hand keypoints are transformed directly into game-controller input, enabling a natural, controller-free gaming experience.

## Use Case Overview

<Steps>
  <Step title="Hand Tracking">
    Two-stage inference, palm detection followed by hand landmark estimation, locates both of the player's hands and extracts their wrist keypoints every frame.
  </Step>

  <Step title="Virtual Steering Wheel">
    The line connecting the two wrist points is treated as a steering wheel. Its tilt angle, measured relative to horizontal, becomes the steering input.
  </Step>

  <Step title="Steering-to-Motion Mapping">
    The tilt angle is converted into a clamped steering value that moves the on-screen car left or right, with smoothing applied to avoid jittery motion.
  </Step>

  <Step title="Split-Screen Rendering">
    The display is divided into two halves: the left shows the raw camera feed with hand-tracking overlays, and the right renders the game itself, including the road, car, obstacles, and score.
  </Step>

  <Step title="Game Loop">
    Obstacles spawn and scroll down the game lane; a collision ends the run and briefly shows a crash state before the game automatically resets.
  </Step>
</Steps>

## Pipeline diagram

<img alt="Touchless Wrist Steering Car Game Pipeline" />

## Elements used in pipeline

| Element | Description |
| - | - |
| `v4l2src` | Captures raw frames from the USB camera. |
| [`qtivtransform`](../plugin-reference/qtivtransform) | Applies a horizontal flip so the camera feed behaves like a mirror, matching the player's actual hand motion. |
| [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | Hardware-accelerated resize, color-space conversion, and normalization ahead of each inference stage. |
| [`qtimltflite`](../plugin-reference/qtimltflite) | Runs the palm detection and hand landmark TFLite models on the QNN HTP delegate. |
| [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | Decodes raw model output into structured detections, using the `palmd` module for palm detection and `hlandmark` for hand landmarks. |
| [`qtimetamux`](../plugin-reference/qtimetamux) | Synchronizes inference metadata from each stage with the corresponding video frame. |
| `qtimetatransform` | Remaps palm-detection metadata (via the `roi-palmd` module) into per-hand regions of interest for the landmark stage. |
| [`qtivoverlay`](../plugin-reference/qtivoverlay) | Draws bounding boxes and keypoints onto the camera-feed half of the split screen. |
| [`qtimlmetaparser`](../plugin-reference/qtimetaparser) | Exposes per-frame landmark metadata to the application through an `appsink` callback. |
| `cairooverlay` | Renders the game itself, including the road, car, obstacles, score, and the steering-wheel visualization, as a Cairo drawing surface. |
| [`waylandsink`](../plugin-reference/waylandsink) | Displays the final composited split-screen output. |

## How it works

<Steps>
  <Step title="Capture and Mirror">
    The USB camera feed is flipped horizontally so on-screen motion matches the player's real-world hand movement, then formatted to a fixed resolution and frame rate for the rest of the pipeline.
  </Step>

  <Step title="Two-Stage Hand Tracking">
    Palm detection runs first, locating each hand's bounding region. `qtimetatransform` uses the `roi-palmd` module to convert each detected region into an aligned crop, which the hand landmark model then processes to extract per-hand keypoints, most importantly the wrist.
  </Step>

  <Step title="Wrist Extraction">
    An `appsink` callback receives the landmark metadata for every frame and pulls out the wrist keypoint (keypoint ID 0) for each tracked hand. When two hands are present, the pair with the largest horizontal separation is selected as the active steering pair.
  </Step>

  <Step title="Steering Angle Calculation">
    The angle of the line between the two wrist points is computed and mapped to a steering value clamped to a fixed range, then smoothed across frames so small tracking jitter doesn't translate into an erratic car.
  </Step>

  <Step title="Game Update">
    Each frame, the steering value nudges the car's horizontal position, obstacles advance down the lane, and a collision check runs against the car's current position. A collision triggers a brief crash state and increments state that leads to an automatic restart.
  </Step>

  <Step title="Split-Screen Composition">
    A Cairo drawing callback renders both halves of the display in a single pass: the left half shows the camera feed with the tracked wrist line and a steering-wheel indicator overlaid, and the right half draws the road, car, obstacles, and score.
  </Step>
</Steps>

## Setup Requirements

### Hardware

<img alt="HW Setup" />

| Component | Description |
| - | - |
| **Edge Device** | RB3 Gen 2, IQ8, or IQ9. Runs the hand-tracking inference and the game loop. |
| **Camera Source** | USB camera positioned to capture the player's hands and upper body. |
| **HDMI Display Monitor** | Connected to the edge device to show the split-screen game output. |

### Software

Flash your Qualcomm Edge device by following the device setup and flashing instructions [here](../installation), then install the Python and GStreamer prerequisites (`python3`, `gstreamer1.0`, `python3-gi`) needed to run the pipeline.

The palm detection and hand landmark TFLite models, along with their label and settings files, are the same assets used by the [hand gesture recognition pipeline](/blogs/smart-hand-gesture-recognition). See that post's model download steps for the palm and hand landmark model files.

<Accordion title="Try me">
  <Tabs>
    <Tab title="C++">
      <Info>
        Check application source code on GitHub: [`demo_car_game/main.cc`](https://github.com/qualcomm/qimsdk/blob/main/cpp/examples/demo-apps/demo_car_game/main.cc)

        Pre-built application on device: `/usr/bin/qimsdk_demo_car_game`
      </Info>

      #### Download Required Files

      Download the gesture recognizer task bundle from Google MediaPipe to obtain the palm detection and hand landmark models:

      ```bash theme={null}
      # Download the gesture recognizer task bundle
      wget https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/latest/gesture_recognizer.task

      # Extract the top-level task
      unzip gesture_recognizer.task

      # Extract hand landmarker models
      unzip hand_landmarker.task
      # save hand_detector.tflite as palm_detection_full.tflite
      # save hand_landmarks_detector.tflite as hand_landmark_full.tflite
      ```

      | File | Save as |
      | - | - |
      | Palm detection model (see steps above) | palm\_detection\_full.tflite |
      | Hand landmark model (see steps above) | hand\_landmark\_full.tflite |
      | <a href="../labels/palmd_labels.json">palmd\_labels.json</a> | palmd\_labels.json |
      | <a href="../labels/palmd_settings.json">palmd\_settings.json</a> | palmd\_settings.json |
      | <a href="../labels/hlandmarks.json">hlandmarks.json</a> | hlandmarks.json |
      | <a href="../labels/hlandmark_settings.json">hlandmark\_settings.json</a> | hlandmark\_settings.json |

      <Note>
        If a downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
      </Note>

      <Steps>
        <Step title="Copy Files to Device">
          <CodeGroup>
            ```bash SCP (SSH) theme={null}
            # Replace <user> and <device-ip> with your device credentials.
            ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
            scp palm_detection_full.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp hand_landmark_full.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp palmd_labels.json          <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            scp palmd_settings.json        <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            scp hlandmarks.json            <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            scp hlandmark_settings.json    <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            ```
          </CodeGroup>
        </Step>

        <Step title="Connect a USB camera">
          Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the `--input-config` value below if your camera is exposed on a different node).
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          /usr/bin/qimsdk_demo_car_game --input-config /dev/video2
          ```

          The application tracks both wrists from the live USB camera feed, mapping their relative tilt to a virtual steering wheel that steers a split-screen driving game in real time.

          To stop the application, press **CTRL + C**.
        </Step>
      </Steps>
    </Tab>

    <Tab title="Python">
      <Info>
        Check application source code on GitHub: [`qimsdk_demo_car_game.py`](https://github.com/qualcomm/qimsdk/blob/main/python/examples/demo-apps/qimsdk_demo_car_game.py)

        Pre-built application on device: `/usr/bin/qimsdk_demo_car_game.py`
      </Info>

      #### Download Required Files

      Download the gesture recognizer task bundle from Google MediaPipe to obtain the palm detection and hand landmark models:

      ```bash theme={null}
      # Download the gesture recognizer task bundle
      wget https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/latest/gesture_recognizer.task

      # Extract the top-level task
      unzip gesture_recognizer.task

      # Extract hand landmarker models
      unzip hand_landmarker.task
      # save hand_detector.tflite as palm_detection_full.tflite
      # save hand_landmarks_detector.tflite as hand_landmark_full.tflite
      ```

      | File | Save as |
      | - | - |
      | Palm detection model (see steps above) | palm\_detection\_full.tflite |
      | Hand landmark model (see steps above) | hand\_landmark\_full.tflite |
      | <a href="../labels/palmd_labels.json">palmd\_labels.json</a> | palmd\_labels.json |
      | <a href="../labels/palmd_settings.json">palmd\_settings.json</a> | palmd\_settings.json |
      | <a href="../labels/hlandmarks.json">hlandmarks.json</a> | hlandmarks.json |
      | <a href="../labels/hlandmark_settings.json">hlandmark\_settings.json</a> | hlandmark\_settings.json |

      <Note>
        If a downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
      </Note>

      <Steps>
        <Step title="Copy Files to Device">
          <CodeGroup>
            ```bash SCP (SSH) theme={null}
            # Replace <user> and <device-ip> with your device credentials.
            ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels}"
            scp palm_detection_full.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp hand_landmark_full.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
            scp palmd_labels.json          <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            scp palmd_settings.json        <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            scp hlandmarks.json            <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            scp hlandmark_settings.json    <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
            ```
          </CodeGroup>
        </Step>

        <Step title="Connect a USB camera">
          Connect a USB (UVC) camera to the target device and verify it is exposed as `/dev/video2` (update the `--input-config` value below if your camera is exposed on a different node).
        </Step>

        <Step title="Run the application">
          ```bash theme={null}
          python3 /usr/bin/qimsdk_demo_car_game.py --input-config /dev/video2
          ```

          The application tracks both wrists from the live USB camera feed, mapping their relative tilt to a virtual steering wheel that steers a split-screen driving game in real time.

          To stop the application, press **CTRL + C**.
        </Step>
      </Steps>
    </Tab>
  </Tabs>
</Accordion>

## Application Function Breakdown

The TouchlessDrive application is organized into a handful of function groups, each responsible for one stage of the hand-to-steering flow:

* **Metadata parsing**: extracts the wrist keypoint (ID 0) for each tracked hand from the raw landmark metadata stream, handling frames where zero, one, or two hands are detected.
* **Steering pair selection**: when two hands are present, selects the pair with the largest horizontal separation as the active steering wheel, discarding stray or partial detections.
* **Steering angle calculation**: computes the tilt angle of the line between the two wrist points, maps it to a clamped steering value, and smooths it across frames to filter out landmark jitter.
* **Game state and physics**: advances the car's horizontal position based on the current steering value, scrolls obstacles down the lane, runs collision checks, and manages the crash/restart state machine.
* **Rendering**: a Cairo drawing callback composes the split-screen output each frame — the camera feed with the wrist line and steering-wheel indicator on the left, and the road, car, obstacles, and score on the right.
* **Pipeline setup**: assembles the two-stage palm detection and hand landmark QIM SDK pipeline and wires the `appsink` callback and Cairo overlay into the runtime before the pipeline starts.

## Expected Output

<video alt="Touchless Wrist Steering Car Game" />

Running the application opens a full-screen split view:

* **Left panel**: live camera feed with the tracked wrist-to-wrist line and a steering-wheel indicator overlaid.
* **Right panel**: the driving game, including a scrolling road, the player-controlled car, incoming obstacles, and a running score.

Tilting the virtual wheel formed by both wrists steers the car left or right; colliding with an obstacle shows a brief crash state before the game restarts automatically. Press `Ctrl+C` to stop the application.

## Conclusion

TouchlessDrive demonstrates how a two-stage hand-tracking pipeline, accelerated entirely on-device through the QNN HTP, can extend beyond traditional video analytics to enable real-time interactive experiences. The wrist-tracking technique used in the driving game can also be adapted for touchless kiosks, accessibility solutions, and other gesture-based human–computer interaction applications built with the Qualcomm QIM SDK.


# Introduction
Source: https://imsdkdocs.qualcomm.com/index

Qualcomm® Intelligent Multimedia SDK

The Qualcomm® Intelligent Multimedia SDK (QIM SDK) is a comprehensive, hardware-accelerated multimedia and AI framework that empowers developers to rapidly build, deploy, and scale intelligent vision, audio, and AI applications on Qualcomm® platforms. Designed for developers, solution integrators, and OEMs, QIM SDK simplifies the creation of advanced edge AI solutions across a wide range of IoT segments, including smart cameras, robotics, drones, home appliances, AI appliances, and connected devices.

<img alt="QIM SDK HW pipeline" />

QIM SDK provides the flexibility to develop applications using the programming environment that best fits your expertise and workflow. Developers can build AI-powered multimedia pipelines using native GStreamer, C++, or Python, enabling both low-level optimization and high-level application development. This flexibility significantly reduces the learning curve and accelerates adoption across diverse developer communities.

* **[Python App Builder](/app-builder/python-quickstart)** — Simple, high-level syntax for fast prototyping and rapid iteration on multi-stream, daisy-chained pipelines with fine-grained control.
* **[C++ App Builder](/app-builder/cpp-quickstart)** — Low-level control and minimal runtime overhead for building high-performance, production-grade multi-stream, daisy-chained pipelines.
* **[GStreamer App Builder](/quickstart)** based native SDK — HW acceleration across Camera/ISP, GPU, Video, Display, IoT protocols, streaming.
* **[Coding Agent](/app-builder-coding-agent)** — Agentic, skill-based coding agent that generates and deploys applications across all three App Builders (Python, C++, and GStreamer).

<img alt="QIM SDK HW pipeline" />

To further streamline development, QIM SDK includes AI-powered Coding Agent capabilities that assist developers in building, optimizing, and integrating AI and multimedia pipelines more efficiently. By reducing manual effort and simplifying complex workflows, the Coding Agent helps accelerate application development and improve developer productivity.

With QIM SDK, developers can rapidly compose and deploy single-stream or multi-stream AI applications while leveraging the full capabilities of Qualcomm® SoCs, including:

* Neural Processing Units (NPUs) and AI accelerators
* Video encode and decode engines
* Camera Image Signal Processors (ISPs)
* GPUs and display subsystems
* Audio DSPs and multimedia accelerators

The SDK's modular, plugin-based architecture abstracts hardware complexity and eliminates the need for extensive platform-specific expertise. Applications can be developed once and deployed seamlessly across multiple Qualcomm® chipset generations and product tiers, enabling portability, scalability, and faster time-to-market.

By combining hardware acceleration, multi-language development support, AI-assisted coding capabilities, and platform portability, QIM SDK enables developers to focus on innovation and deliver high-performance intelligent edge solutions faster and with greater efficiency.

## Recommended path for new developers

<Steps>
  <Step title="Setup the SDK">
    Get the QIM SDK installed on your Linux device<br />
    [Go to Installation](/installation)
  </Step>

  <Step title="Create your first AI pipeline">
    Create a simple Hello World style AI pipeline<br />
    [Go to Quickstart Guide](/app-builder-quickstart)
  </Step>

  <Step title="Understand the main components of an AI pipline">
    Learn to capture video and audio from a device, preprocess inputs, run AI inference, and interpret the results<br />
    [Go to Building AI Pipelines](/qimsdk-overview/sdkoverview)
  </Step>

  <Step title="Explore features of QIM SDK">
    Browse sample pipelines and code, dive into advanced features and containerization, and reference the full plugin documentation<br />
  </Step>
</Steps>

## What do you want to do?

<Columns>
  <Card title="Installation" icon="pen-to-square" href="/installation">
    Install QIM SDK on your Linux device.
  </Card>

  <Card title="Building AI pipelines" icon="palette" href="/qimsdk-overview/sdkoverview">
    Learn how to quickly build AI pipelines using the provided examples.
  </Card>

  <Card title="Plugin Documentation" icon="map" href="/plugin-reference/introduction">
    Reference documentation for all QIM SDK GStreamer plugins.
  </Card>

  <Card title="Release Notes" icon="note" href="/relnotes">
    View the latest release notes.
  </Card>
</Columns>

## Need inspiration?

<Card title="AI pipeline examples" icon="pipe-section" href="/sample-pipelines/aipipelines">
  View complete end-to-end pipeline examples you can leverage.
</Card>

<Card title="See code samples" icon="terminal" href="/sample-application/ai-sample-applications">
  Get started with Python, C++, and GStreamer code samples.
</Card>


# Installation
Source: https://imsdkdocs.qualcomm.com/installation



Qualcomm® Intelligent Multimedia SDK (QIM SDK) is an AI framework designed to build Vision AI pipelines. It runs on Qualcomm’s powerful and power-efficient IoT platforms, providing GStreamer-based plugins for multimedia, computer vision (CV), and artificial intelligence (AI) to facilitate seamless application development.

<Tabs>
  <Tab title="QIM SDK on Yocto">
    To set up your device with Qualcomm Linux (Yocto), download the prebuilt image for your platform and flash it by following the [Qualcomm Linux Flash Guide](https://dragonwingdocs.qualcomm.com/Key-Documents/Flash-Guide/obtain-prebuilts) on Dragonwing Docs.
  </Tab>

  <Tab title="QIM SDK on Ubuntu">
    ## Prerequisites

    Make sure your [Dragonwing](https://dragonwingdocs.qualcomm.com/index) device has been set up with Ubuntu and you have shell access.<br />
    [Download Ubuntu on Qualcomm IoT Platforms](https://dragonwingdocs.qualcomm.com/Ubuntu/ubuntu-overview) and `Select Your Device`.

    <Warning>
      QIM SDK 2.0 is not available on Ubuntu in this release. See the [release notes](/relnotes) to learn more.
    </Warning>

    ## Installation

    <Steps>
      <Step title="Add Qualcomm PPA">
        Update package metadata and add the Qualcomm Ubuntu PPA.

        ```bash theme={null}
        sudo apt update
        sudo add-apt-repository ppa:ubuntu-qcom-iot/qcom-ppa
        ```
      </Step>

      <Step title="Install GStreamer core components">
        Install the required GStreamer tools and base multimedia packages.

        ```bash theme={null}
        sudo apt install -y \
          gstreamer1.0-tools \
          gstreamer1.0-plugins-good \
              gstreamer1.0-plugins-base \
          gstreamer1.0-libav
        ```
      </Step>

      <Step title="Install Qualcomm hardware-accelerated GStreamer plugins">
        Install the Qualcomm GStreamer plugin package.

        ```bash theme={null}
        sudo apt install -y \
          gstreamer1.0-plugins-qcom
        ```
      </Step>

      <Step title="Install AI runtime and inference components">
        Install AI runtime packages and developer tools required for inference workflows.

        ```bash theme={null}
        sudo apt install -y \
          tensorflow-lite-qcom-apps \
          libqnn-dev \
          libsnpe-dev
        ```
      </Step>

      <Step title="Validate QIM SDK plugins on device">
        Verify that Qualcomm-specific GStreamer plugins are available after installation.

        ```bash theme={null}
        gst-inspect-1.0 | grep qti
        ```

        After installation, you should see Qualcomm-specific GStreamer plugins listed.

        <img alt="List of QIM SDK plugins" />
      </Step>

      <Step title="Install optional developer packages">
        Use the following optional packages based on your development or deployment needs.

        <AccordionGroup>
          <Accordion title="Multimedia acceleration and system integration">
            Install packages for FastCV, DMA buffer heap integration, and sensor test applications.

            ```bash theme={null}
            sudo apt install -y \
              qcom-fastcv-binaries-dev \
              qcom-libdmabufheap-dev \
              qcom-sensors-test-apps
            ```
          </Accordion>

          <Accordion title="Utilities and media tools">
            Install common utilities used during media development, debugging, and validation.

            ```bash theme={null}
            sudo apt install -y \
              ffmpeg \
              net-tools \
              pulseaudio-utils \
              selinux-utils \
              unzip \
              v4l-utils \
              qnn-tools \
              snpe-tools
            ```
          </Accordion>

          <Accordion title="Sample applications and Python examples">
            Install sample applications and Python examples for quick validation and development.

            ```bash theme={null}
            sudo apt install -y \
              gstreamer1.0-qcom-sample-apps \
              gstreamer1.0-qcom-python-examples
            ```
          </Accordion>

          <Accordion title="Wayland and display support (server only)">
            Install display and Wayland-related packages for systems that require GUI or compositor support.

            ```bash theme={null}
            sudo apt install -y \
              qcom-adreno1 \
              weston-autostart \
              xwayland
            ```
          </Accordion>
        </AccordionGroup>
      </Step>
    </Steps>
  </Tab>
</Tabs>


# Plugins Introduction
Source: https://imsdkdocs.qualcomm.com/plugin-reference/introduction

Documentation for QIM SDK GStreamer plugins

<Note>
  Please refer [GStreamer Plugin Documentation](https://gstreamer.freedesktop.org/documentation/plugins_doc.html) for standard GStreamer plugins.
</Note>

## Welcome

The QIM SDK extends the powerful **GStreamer multimedia framework** by introducing a comprehensive suite of plugins tailored for building **end-to-end AI, video, and streaming pipelines**. Designed to work seamlessly alongside existing **open-source GStreamer components**, these plugins enable developers to rapidly prototype, deploy, and scale real-world applications without sacrificing flexibility or performance.

With the increasing demand for intelligent multimedia applications—such as smart surveillance, video analytics, and edge AI—the QIM SDK provides optimized building blocks that simplify pipeline development while ensuring efficient execution. Developers can combine standard GStreamer elements with QIM-specific plugins to construct modular pipelines that handle everything from data ingestion and preprocessing to inference, post-processing, and streaming.

A key advantage of the QIM SDK is its ability to leverage heterogeneous compute resources available on Qualcomm platforms, including the **GPU, DSP, and camera subsystem**. This enables high-throughput, low-latency processing for both AI/ML workloads and multimedia operations. The plugins are designed to abstract hardware complexity while still allowing fine-grained control when needed.

Whether you are building simple media pipelines or complex AI-driven workflows, the QIM SDK provides a scalable and efficient foundation to accelerate development and deployment.

## Plugin Classification

### 1. AI / ML Plugins

| Plugin | Description |
| - | - |
| [qtimlvconverter](/plugin-reference/qtimlvconverter) | Converts input data into ML-friendly tensor formats |
| [qtimlonnx](/plugin-reference/qtimlonnx) | Executes ONNX-based machine learning models |
| [qtimlqnn](/plugin-reference/qtimlqnn) | Runs models using Qualcomm Neural Network backend |
| [qtimlsnpe](/plugin-reference/qtimlsnpe) | Runs models using Snapdragon Neural Processing Engine |
| [qtimltflite](/plugin-reference/qtimltflite) | Executes TensorFlow Lite models |
| [qtimldemux](/plugin-reference/qtimldemux) | Splits multiplexed ML data streams |
| [qtimlpostprocess](/plugin-reference/qtimlpostprocess) | Handles output parsing and post-processing |

### 2. Video Processing Plugins

| Plugin | Description |
| - | - |
| [qtivtransform](/plugin-reference/qtivtransform) | Performs scaling, rotation, and format conversion |
| [qtivcomposer](/plugin-reference/qtivcomposer) | Composes multiple video streams into one |
| [qtivoverlay](/plugin-reference/qtivoverlay) | Adds graphical overlays (bounding boxes, text) |
| [qtivsplit](/plugin-reference/qtivsplit) | Splits video streams into multiple outputs |

### 3. Camera Plugin

| Plugin | Description |
| - | - |
| [qticamsrc](/plugin-reference/qticamsrc) | Captures video directly from the camera stack |

### 4. Codec / V4L2 Plugins

| Plugin | Description |
| - | - |
| [v4l2h264dec](/plugin-reference/v4l2h264dec) | Hardware-accelerated H.264 decoder |
| [v4l2h264enc](/plugin-reference/v4l2h264enc) | Hardware-accelerated H.264 encoder |
| [v4l2h265dec](/plugin-reference/v4l2h265dec) | Hardware-accelerated H.265 (HEVC) decoder |
| [v4l2h265enc](/plugin-reference/v4l2h265enc) | Hardware-accelerated H.265 encoder |
| [v4l2vp9dec](/plugin-reference/v4l2vp9dec) | Hardware-accelerated VP9 decoder |
| [v4l2av1dec](/plugin-reference/v4l2av1dec) | Hardware-accelerated AV1 decoder |
| [qtijpegenc](/plugin-reference/qtijpegenc) | JPEG image encoder |

### 5. Streaming Plugins

| Plugin | Description |
| - | - |
| [qtirtspbin](/plugin-reference/qtirtspbin) | Enables RTSP streaming pipelines |
| [qtiredissink](/plugin-reference/qtiredissink) | Streams data to Redis endpoints |
| [qtisocketsink](/plugin-reference/qtisocketsink) | Sends data over network sockets |
| [qtisocketsrc](/plugin-reference/qtisocketsrc) | Receives data over network sockets |
| [waylandsink](/plugin-reference/waylandsink) | Displays output using Wayland |

### 6. Metadata & Utilities Plugins

| Plugin | Description |
| - | - |
| [qtimetamux](/plugin-reference/qtimetamux) | Combines metadata streams |
| [qtimetaparser](/plugin-reference/qtimetaparser) | Parses ML metadata outputs |
| [qtimsg](/plugin-reference/qtimsg) | Publishes messages to external systems |
| [qtimsg](/plugin-reference/qtimsg) | Subscribes to external message streams |
| [qtiobjtracker](/plugin-reference/qtiobjtracker) | Tracks detected objects across frames |
| [qtibatch](/plugin-reference/qtibatch) | Batches multiple frames or inputs for processing |

## Proprietary Plugins

<Note>
  The following plugins are only available in `qcom-multimedia-proprietary-image`. <br />
  For more information on QLI images refer to [Qualcomm Linux release](https://dragonwingdocs.qualcomm.com/Key-Documents/Yocto-Guide/add-image-recipes#qualcomm-reference-images)

  | Plugin |
  | - |
  | [qticamsrc](/plugin-reference/qticamsrc) |
  | [qtijpegenc](/plugin-reference/qtijpegenc) |
  | [qtimlqnn](/plugin-reference/qtimlqnn) |
  | [qtimlsnpe](/plugin-reference/qtimlsnpe) |
</Note>

## Summary

By combining these plugins with standard GStreamer elements, the QIM SDK empowers developers to construct **scalable, production-ready multimedia and AI pipelines**. The modular architecture ensures flexibility, while hardware acceleration across GPU, DSP, and camera subsystems ensures optimal performance.


# qtibatch
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtibatch

GStreamer buffer batching element

# Overview

`qtibatch` is a GStreamer element that aggregates multiple video or audio buffers into a single batched buffer for downstream processing. It is intended for multimedia and AI pipelines where downstream elements operate more efficiently on grouped input than on individual buffers.

The element supports two batching modes:

* **Multi-stream batching** — combines buffers arriving concurrently from multiple parallel streams
* **Sequential batching** — combines successive buffers from a single stream

This makes `qtibatch` suitable for both parallel multi-stream workloads and temporal workflows that require multiple consecutive frames or samples as a single input unit.

Batching is commonly used to:

* Improve hardware utilization
* Increase throughput for downstream inference or analytics stages
* Enable models that require batched input
* Construct temporal or depth-oriented input tensors from consecutive frames
* Create overlapping batches for sliding-window style processing

Typical use cases include:

* **Multi-stream video batching** — aggregating frames from multiple video sources into a single batch for parallel processing
* **Multi-stream audio batching** — aggregating audio buffers from multiple sources into a single batch for parallel processing
* **Depth batching** — grouping consecutive frames from the same stream to create tensors with an additional temporal or depth dimension
* **Overlapping batch construction** — building batches with shared frames between adjacent batches for temporal models or window-based analysis

<Note>
  `qtibatch` performs buffer aggregation only. It does not modify video pixels or audio samples, and it does not perform inference, preprocessing, or post-processing. Its sole responsibility is to package multiple input buffers into a single logical batch for efficient downstream consumption.
</Note>

<img alt="qtibatch pipeline" />

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />qtibatch

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { ANY }` |
| `audio/x-raw` | `format: { ANY }` |
| Availability: *On request* | |
| Direction: *sink* | |
| Pad Name: `sink_%u` | |

### src

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { ANY }` <br /> `note: batched buffers` |
| `audio/x-raw` | `format: { ANY }` <br /> `note: batched buffers` |
| Availability: *Always* | |
| Direction: *source* | |

# Element Properties

| Property | Description |
| - | - |
| `moving-window-size` | Number of new buffers that will be used for output frames.<br /><br />`Type: Unsigned Integer`<br />`Default: 1`<br />`Range: 1 - 16`<br />`Flags: readable/writable (changeable only in NULL, READY)` |

# Internal Architecture Details

`qtibatch` is organized around three functional components:

## Sink Pads

`qtibatch` supports either a single sink pad or multiple sink pads, depending on the batching mode:

* A **single sink pad** is used for sequential batching of successive buffers from the same stream
* **Multiple sink pads** are used for parallel batching of buffers from multiple input streams

Each sink pad receives individual video or audio buffers from upstream elements.

## Aggregation Logic

The aggregation logic collects incoming buffers, tracks their arrival status, and determines when a batch is ready to be produced. Batch formation is driven by the negotiated batching configuration and the active batching mode.

## Source Pad

A single source pad emits the batched output buffer. The output buffer carries references to the aggregated input buffers together with batch metadata that describes the batch contents.

## Processing Flow

### Caps Negotiation

During caps negotiation, `qtibatch` verifies that all sink pads use compatible media (audio or video) types and formats. It also derives the effective batching configuration, including the batch size and the expected input/output tensor or media characteristics required for downstream processing.

### Buffer Reception

Incoming video or audio buffers are received asynchronously on the sink pads and queued internally on a per-pad basis.

### Synchronization

`qtibatch` synchronizes buffers according to the selected batching mode:

* For **parallel streams**, the element waits for buffers that correspond to the same time interval across all active sink pads
* For **single stream**, the element waits for the required number of successive buffers from the same input stream

For parallel batching, if a buffer does not arrive on a given pad within the expected time window, that pad is treated as missing for the current batch. For single-stream batching, timeout-based skipping is not applied.

### Batch Formation

Buffers are grouped into a batch according to the selected batching strategy:

* Successive buffers from a single stream
* Time-aligned buffers from multiple streams

The resulting batch preserves buffer ordering and, where applicable, temporal alignment across inputs.

### Metadata Attachment

`qtibatch` attaches batch metadata to the output buffer. This metadata describes:

* The effective batch size
* The ordering of aggregated buffers
* Any missing inputs in parallel-stream mode

### Output Delivery

Once the batch is complete, the batched buffer is pushed downstream for further processing.

# Batching Behavior

Batch formation in `qtibatch` is determined by the negotiated batch size together with the active batching mode.

Incoming buffers are always queued internally on a per-pad basis. The batch completion logic depends on whether the element is aggregating buffers from a single stream or from multiple parallel streams.

## Sequential Batching (Single Stream)

In sequential mode, `qtibatch` aggregates successive buffers from a single input stream until the configured batch size is reached. Batch formation is count-based.

This mode supports a moving-window mechanism controlled by the `moving-window-size` property:

* A value of **0** produces non-overlapping batches
* A value **greater than 0** produces overlapping batches, reusing the specified number of buffers from the previous batch in the next batch

This behavior is useful for temporal inference workloads, such as action recognition or sequence-based models, where continuity across adjacent batches is important.

Sequential batching is strictly driven by buffer count and does not use timeout-based GAP insertion.

## Parallel Batching (Multiple Streams)

In parallel mode, `qtibatch` aggregates buffers arriving from multiple sink pads. The goal is to form a batch from buffers that belong to the same time interval across all input streams.

If a buffer is not received on one of the sink pads within the expected synchronization window, that input is treated as missing for the current batch. In this case, the batch metadata records the missing input so downstream elements can interpret the batch correctly — this is done through inserting a GAP buffer.

The synchronization window and timeout behavior are derived from buffer duration, which is calculated from the negotiated input framerate or audio timing information.

# Memory and Buffer Management

`qtibatch` is designed to integrate into high-performance and zero-copy multimedia pipelines.

## Buffer Handling

`qtibatch` does not copy media payloads during batch construction. Instead, the batched output buffer holds references to the original input buffers.

## Allocation and Pool Behavior

`qtibatch` does not require a custom allocator, a specific memory type, or a dedicated buffer pool. Buffer allocation and memory layout remain under the control of upstream and downstream elements.

This design allows `qtibatch` to fit into a wide range of pipeline topologies without imposing additional memory management constraints.

# Usage

### Four-Source Batched Object Detection with Compositor

Demo pipeline with batching of 4 sources. A detection inference is run and the results are overlaid via composer on the screen.

<img alt="use" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Yolov8 Detection W8A8 Batch 4 model | [Export from Qualcomm AI Hub](https://github.com/qualcomm/ai-hub-models/blob/v0.55.0/src/qai_hub_models/models/yolov8_det/README.md) | `yolov8_det_w8a8_batch_4.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp yolov8_det_w8a8_batch_4.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                           <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4             <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolov8_det_w8a8_batch_4.tflite
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME_1=ai_demo_sample.mp4
    export SRC_VIDEO_NAME_2=ai_demo_sample.mp4
    export SRC_VIDEO_NAME_3=ai_demo_sample.mp4
    export SRC_VIDEO_NAME_4=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash Run the pipeline theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qtimltflite name=inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,htp_performance_mode=(string)2;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME_1 ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw ! qtivtransform ! video/x-raw,format=NV12,width=640,height=360 ! queue ! tee name=tee_0 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME_2 ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw ! qtivtransform ! video/x-raw,format=NV12,width=640,height=360 ! queue ! tee name=tee_1 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME_3 ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw ! qtivtransform ! video/x-raw,format=NV12,width=640,height=360 ! queue ! tee name=tee_2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME_4 ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw ! qtivtransform ! video/x-raw,format=NV12,width=640,height=360 ! queue ! tee name=tee_3 \
    tee_0. ! video/x-raw,format=NV12 ! mixer. \
    tee_0. ! video/x-raw,format=NV12 ! batch. \
    tee_1. ! video/x-raw,format=NV12 ! mixer. \
    tee_1. ! video/x-raw,format=NV12 ! batch. \
    tee_2. ! video/x-raw,format=NV12 ! mixer. \
    tee_2. ! video/x-raw,format=NV12 ! batch. \
    tee_3. ! video/x-raw,format=NV12 ! mixer. \
    tee_3. ! video/x-raw,format=NV12 ! batch. \
    qtibatch name=batch ! queue ! qtimlvconverter ! queue ! inference. inference. ! queue ! qtimldemux name=mldemux \
    mldemux. ! queue ! qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! video/x-raw ! queue ! mixer. \
    mldemux. ! queue ! qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! video/x-raw ! queue ! mixer. \
    mldemux. ! queue ! qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! video/x-raw ! queue ! mixer. \
    mldemux. ! queue ! qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! video/x-raw ! queue ! mixer. \
    qtivcomposer name=mixer \
    sink_0::position="<0, 0>" sink_0::dimensions="<960, 540>" \
    sink_1::position="<960,  0>" sink_1::dimensions="<960, 540>" \
    sink_2::position="<0, 540>" sink_2::dimensions="<960, 540>" \
    sink_3::position="<960, 540>" sink_3::dimensions="<960, 540>" \
    sink_4::position="<0, 0>" sink_4::dimensions="<960, 540>" \
    sink_5::position="<960, 0>" sink_5::dimensions="<960, 540>" \
    sink_6::position="<0, 540>" sink_6::dimensions="<960, 540>" \
    sink_7::position="<960, 540>" sink_7::dimensions="<960, 540>" \
    mixer. ! video/x-raw,format=NV12 ! queue ! waylandsink sync=true fullscreen=true
    ```
  </Step>
</Steps>


# qticamsrc
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qticamsrc

Hardware-accelerated GStreamer camera source plugin for Qualcomm ISP camera access via the camera service.

<Note>
  qticamsrc is only available in `qcom-multimedia-proprietary-image`  <br />
  For more information on QLI images refer to [Qualcomm Linux release](https://dragonwingdocs.qualcomm.com/Key-Documents/Yocto-Guide/add-image-recipes#qualcomm-reference-images)
</Note>

## Overview

`qticamsrc` is a hardware-accelerated GStreamer **camera source plugin** that provides direct access to Qualcomm's Camera ISP (Image Signal Processor) capabilities through the `camera service`. It serves as the primary camera input element in Qualcomm's GStreamer multimedia stack, enabling applications to acquire live frames from camera hardware and integrate them into multimedia, imaging, and AI processing pipelines.

The plugin exposes Qualcomm camera ISP functionality through a standard GStreamer source interface, allowing applications to capture one or more camera streams in supported output formats while configuring both common capture settings and advanced ISP controls. This enables developers to build pipelines that combine camera acquisition with display, encoding, image capture, or downstream analytics processing.

`qticamsrc` operates as a client of the camera service, which is responsible for managing the underlying camera Hardware Abstraction Layer (HAL) and coordinating access to camera hardware resources. This client-server design provides a scalable foundation for both multi-client and multi-camera deployments, allowing multiple applications or pipelines to access one or more physical camera devices concurrently, subject to platform capabilities.

The plugin implements a request-pad, multi-output architecture with independently configurable `video_%u` and `image_%u` source pads. Each output pad can be configured with its own resolution, pixel format, frame rate, and codec-related settings, allowing a single camera session to produce multiple streams simultaneously. This makes it possible for one camera instance to support concurrent use cases such as:

* live preview,
* high-resolution video recording,
* still-image capture,
* machine learning inference pipelines.

`qticamsrc` supports multiple output formats to address both video and imaging use cases, including:

* NV12 and NV21,
* Raw Bayer,
* NV12 with UBWC (Universal Bandwidth Compression),
* JPEG output for snapshot capture.

The plugin also provides access to static and dynamic camera controls, enabling applications to configure image quality and other Qualcomm camera features directly through the GStreamer pipeline.

Beyond single-camera operation, `qticamsrc` supports multi-camera and multi-client usage models. Multiple plugin instances can be used within the same application to acquire frames from different physical cameras, enabling deployment scenarios such as:

* stereo camera processing,
* side-by-side composition,
* picture-in-picture,
* independent parallel camera streams.

For snapshot workflows, `qticamsrc` supports dedicated image capture paths with snapshot stream types that include:

* NV12 and NV21,
* Raw Bayer,
* JPEG blob output.

Internally, all camera access is coordinated through the `camera service`, which centralizes control of the camera HAL and associated ISP resources. This architecture simplifies resource management and enables scalable camera system designs, while the number of concurrently supported cameras and output streams remains dependent on the capabilities of the target Qualcomm chipset and ISP implementation and capabilities.

<img alt="" />

## Typical Usage Patterns

`qticamsrc` is commonly used in the following scenarios:

### Live camera preview

Capturing and streaming YUV frames for real-time display, or forwarding them directly to downstream encoding elements for preview and recording pipelines.

### Concurrent multi-stream capture

Producing multiple output streams from a single camera session, such as recording video while simultaneously capturing JPEG snapshots, and generating additional streams for AI inference, local storage, or network streaming.

### High-frame-rate video

Supporting high-frame-rate capture workflows, including pipelines operating above 60 FPS, with super-buffer mode used to reduce per-frame processing overhead in downstream elements.

### Raw Bayer capture

Acquiring unprocessed sensor data for custom imaging pipelines, computational photography workflows, or computer vision applications that require direct access to raw camera output.

### Multi-camera and logical-camera workflows

Accessing frames from multiple physical sensors, or working with coordinated camera configurations, for use cases such as stereo imaging, synchronized capture, and composite camera solutions.

### Still-image capture

Triggering single-shot or burst image capture workflows through GStreamer signals, with support for JPEG and RAW-based snapshot paths depending on platform configuration.

## Element Hierarchy

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />qticamsrc<br />

## Pad Templates

`qticamsrc` uses request pads — pads are not present by default and must be explicitly requested before the pipeline transitions to READY.

### Video Source Pad Template (video\_%u)

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { NV12, NV16, NV12_Q08C, RGB, YUY2, UYVY, P010_10LE, NV12_Q10LE32C }` <br /> `width: [min, max]` <br /> `height: [min, max]` <br /> `framerate: [0/1, max/1]` |
| `video/x-bayer` | `format:  { bggr, rggb, gbrg, grbg, mono }` <br /> `width: [min, max]` <br /> `height: [min, max]` <br /> `framerate: [0/1, max/1]` |
| `image/jpeg` | `format: NA` <br /> `width: [min, max]` <br /> `height: [min, max]` <br /> `framerate: [0/1, max/1]` |
| Availability: *On request* | |
| Direction: *source* | |
| Pad Name: `video_%u` | |

### Image Source Pad Template (image\_%u)

| Capabilities | |
| - | - |
| `image/jpeg` | `format: NA` <br /> `width: [min, max]` <br /> `height: [min, max]` <br /> `framerate: [0/1, max/1]` |
| `video/x-raw` | `format: { NV21, NV12, P010_10LE, NV12_Q10LE32C }` <br /> `width: [min, max]` <br /> `height: [min, max]` <br /> `framerate: [0/1, max/1]` |
| `video/x-bayer` | `format: { bggr, rggb, gbrg, grbg, mono }` <br /> `width: [min, max]` <br /> `height: [min, max]` <br /> `framerate: [0/1, max/1]` |
| Availability: *On request* | |
| Direction: *source* | |
| Pad Name: `image_%u` | |

<Note>
  Format availability is determined at runtime from camera static metadata. Resolution ranges and maximum framerate are also dynamically populated from camera static metadata at plugin load time.
</Note>

## Element Properties

### Core Camera Properties

| Property | Description |
| - | - |
| `active-sensor-size` | Returns the active pixel array rectangle in the format `<X, Y, WIDTH, HEIGHT>` of the sensor. Only populated after the element reaches READY state.<br /><br />`Type: GstValueArray of type gint`<br /> `Default: "<0, 0, 0, 0>"`<br /> `Flags: readable (available in READY and higher states)` |
| `camera` | Camera device ID to open. Selects which physical or logical camera device is used for all pads on this element instance.<br /><br />`Type: Unsigned Integer`<br />`Default: 0`<br /> `Range: 0 - 32` <br /> `Flags: readable/writable` |
| `sensor-mode` | Force a specific sensor mode index (0–15). A value of -1 means automatic selection by the camera HAL.<br /><br />`Type: Integer`<br />`Default: -1`<br />`Range: -1 - 15`<br /> `Flags: readable/writable` |
| `slave` | Set camera as slave device. In slave mode the camera session is subordinate to a master session, enabling synchronized multi-camera capture.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `static-metadata` | Returns the camera's static capabilities as a CameraMetadata object. The caller is responsible for releasing the object.<br /><br />`Type: Pointer (CameraMetadata)`<br />`Flags: readable` |
| `static-metas` | Returns a map of all connected camera IDs to their corresponding static metadata objects.<br /><br />`Type: GHashTable`<br />`Flags: readable` |

### Image Quality Properties (changeable in PLAYING)

| Property | Description |
| - | - |
| `contrast` | Image contrast strength.<br /><br />`Type: Integer`<br />`Default: 5`<br />`Range: 1 - 10`<br />`Flags: readable/writable` |
| `saturation` | Image saturation strength.<br /><br />`Type: Integer`<br />`Default: 5`<br />`Range: 0 - 10`<br />`Flags: readable/writable` |
| `sharpness` | Image sharpness strength.<br /><br />`Type: Integer`<br />`Default: 2`<br />`Range: 0 - 6`<br />`Flags: readable/writable` |

### 3A (Auto-Exposure, Auto-White-Balance, Auto-Focus) Properties

| Property | Description |
| - | - |
| `antibanding` | Antibanding mode to suppress flicker from artificial lighting: `off`, `50hz`, `60hz`, or `auto`.<br /><br />`Type: Enum`<br />`Default: 3, "auto"`<br /> `Range:` <br />     `(0): off - The camera device will not adjust exposure duration to avoid banding problems.` <br />     `(1): 50hz - The camera device will adjust exposure duration to avoid banding problems with 50Hz illumination sources.` <br />     `(2): 60hz - The camera device will adjust exposure duration to avoid banding problems with 60Hz illumination sources.` <br />     `(3): auto - The camera device will automatically adapt its antibanding routine to the current illumination condition.` <br /> `Flags: readable/writable (changeable in PLAYING state)`     <br /> `Example: antibanding="off" (or) antibanding=0` |
| `control-mode` | Top-level 3A control switch. `off` disables all 3A routines, `auto` enables them, and `use-scene-mode` delegates to the scene mode setting.<br /><br />`Type: Enum`<br />`Default: 1, "auto"`<br /> `Range:` <br />     `(0): off - Full application control of pipeline.` <br />     `(1): auto - Manual control of capture parameters is disabled.` <br />     `(2): use-scene-mode - Use a specific scene mode.` <br />     `(3): off-keep-state - Same as OFF mode, except that this capture will not be used by camera device background auto-exposure, auto-white balance and auto-focus algorithms (3A) to update their statistics.` <br /> `Flags: readable/writable (changeable in PLAYING state)`   <br /> `Example: control-mode="off" (or) control-mode=0` |
| `custom-exposure-table` | A serialized GstStructure string describing a custom exposure table (vendor tag `org.codeaurora.qcamera3.exposuretable`).<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable (changeable in PLAYING state)` |
| `exposure-compensation` | Exposure compensation in steps.<br /><br />`Type: Integer`<br />`Default: 0`<br />`Range: -12 - 12`<br />`Flags: readable/writable (changeable in PLAYING state)` |
| `exposure-lock` | Locks the current auto-exposure values, preventing further adjustments.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable (changeable in PLAYING state)` |
| `exposure-metering` | Exposure metering algorithm: `average`, `center-weighted`, `spot`, or `custom`.<br /><br />`Type: Enum`<br />`Default: 0, "average"`<br /> `Range:` <br />     `(0): average - The camera device's exposure metering is calculated as average from the whole frame.` <br />     `(1): center-weighted - The camera device's exposure metering is calculated from the center region of the frame.` <br />     `(2): spot - The camera device's exposure metering is calculated from a chosen spot.` <br />     `(6): custom - The camera device's exposure metering is calculated from a custom metering table.` <br /> `Flags: readable/writable (changeable in PLAYING state)`  <br /> `Example: exposure-metering="average" (or) exposure-metering=0` |
| `exposure-mode` | Auto-exposure mode. Set to `off` to use manual exposure time.<br /><br />`Type: Enum`<br />`Default: 1, "auto"`<br /> `Range:` <br />     `(0): off - The auto exposure routine is disabled. Manual exposure time will be used set via the 'exposure-time' property` <br />     `(1): auto - The auto exposure routine is active.` <br /> `Flags: readable/writable (changeable in PLAYING state)`  <br /> `Example: exposure-mode="off" (or) exposure-mode=0` |
| `focus-mode` | Auto-focus mode: `off`, `auto`, `macro`, `continuous`, or `edof`.<br /><br />`Type: Enum`<br />`Default: 0, "off"`<br /> `Range:` <br />     `(0): off - The auto focus routine is disabled.` <br />     `(1): auto - The auto focus routine is active.` <br />     `(2): macro - In this mode, the auto focus algorithm is optimized for focusing on objects very close to the camera.` <br />     `(3): continuous - In this mode, the AF algorithm modifies the lens position continually to attempt to provide a constantly-in-focus image stream.` <br />     `(4): edof - The camera device will produce images with an extended depth of field automatically; no special focusing operations need to be done before taking a picture.` <br />`Flags: readable/writable (changeable in PLAYING state)`  <br /> `Example: focus-mode="auto" (or) focus-mode=1` |
| `iso-mode` | ISO sensitivity mode. Set to `manual` to use `manual-iso-value`.<br /><br />`Type: Enum`<br />`Default: 0, "auto"`<br /> `Range:` <br />     `(0): auto - The ISO exposure mode will be chosen depending on the scene.` <br />     `(1): deblur - The ISO exposure sensitivity set to prioritize motion deblur.` <br />     `(2): 100 - The ISO exposure sensitivity set to prioritize level 100.` <br />     `(3): 200 - The ISO exposure sensitivity set to prioritize level 200.` <br />     `(4): 400 - The ISO exposure sensitivity set to prioritize level 400.` <br />     `(5): 800 - The ISO exposure sensitivity set to prioritize level 800.` <br />     `(6): 1600 - The ISO exposure sensitivity set to prioritize level 1600.` <br />     `(7): 3200 - The ISO exposure sensitivity set to prioritize level 3200.` <br />     `(8): manual - The ISO exposure value provided by manual-iso-value will be used.` <br /> `Flags: readable/writable (changeable in PLAYING state)`  <br /> `Example: iso-mode="deblur" (or) iso-mode=1` |
| `manual-exposure-time` | Manual exposure time in nanoseconds. Active only when `exposure-mode=off`.<br /><br />`Type: Integer64`<br />`Default: 33333333`<br /> `Range: 0 - 9223372036854775807` <br /> `Flags: readable/writable (changeable in PLAYING state)` |
| `manual-iso-value` | Manual ISO value. Active only when `iso-mode=manual`.<br /><br />`Type: Integer`<br />`Default: 800`<br />`Range: 100 - 3200`<br />`Flags: readable/writable (changeable in PLAYING state)` |
| `manual-wb-settings` | A serialized GstStructure string for manual white balance settings (color temperature or RGB gains). Used with `manual-cc-temp` or `manual-rgb-gains` modes.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable (changeable in PLAYING state)` |
| `white-balance-lock` | Locks the current auto white-balance values.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable (changeable in PLAYING state)` |
| `white-balance-mode` | White balance mode. Supports `auto`, `daylight`, `incandescent`, `fluorescent`, `manual-cc-temp`, `manual-rgb-gains`, and others.<br /><br />`Type: Enum`<br />`Default: 3, "auto"`<br /> `Range:` <br />     `(0): off - Both auto and manual white balance routines are disabled.` <br />     `(1): manual-cc-temp - The auto-white balance routine is inactive and manual color correction temperature is used which is set via the 'manual-wb-settings' property.` <br />     `(2): manual-rgb-gains - The auto-white balance routine is inactive and manual R/G/B gains are used which are set via the 'manual-wb-settings' property.` <br />     `(3): auto - The auto-white balance routine is active.` <br />     `(4): shade - The camera device uses shade light as the assumed scene illumination for white balance correction.` <br />     `(5): incandescent - The camera device uses incandescent light as the assumed scene illumination for white balance correction.` <br />     `(6): fluorescent - The camera device uses fluorescent light as the assumed scene illumination for white balance correction.` <br />     `(7): warm-fluorescent - The camera device uses warm fluorescent light as the assumed scene illumination for white balance correction.` <br />     `(8): daylight - The camera device uses daylight light as the assumed scene illumination for white balance correction.` <br />     `(9): cloudy-daylight - The camera device uses cloudy daylight light as the assumed scene illumination for white balance correction.` <br />     `(10): twilight - The camera device uses twilight light as the assumed scene illumination for white balance correction.` <br /> `Flags: readable/writable (changeable in PLAYING state)`  <br /> `Example: white-balance-mode="auto" (or) white-balance-mode=3` |

### Scene and Effect Properties

| Property | Description |
| - | - |
| `effect` | Color effect applied to frames such as `off`, `mono`, `negative`, `sepia`, `aqua`, and others.<br /><br />`Type: Enum `<br />`Default: 0, "off"`<br /> `Range:` <br />     `(0): off - No color effect will be applied.` <br />     `(1): mono - A 'monocolor' effect where the image is mapped into a single color.` <br />     `(2): negative - A 'photo-negative' effect where the image's colors are inverted.` <br />     `(3): solarize - A 'solarisation' effect (Sabattier effect) where the image is wholly or partially reversed in tone.` <br />     `(4): sepia - A 'sepia' effect where the image is mapped into warm gray, red, and brown tones.` <br />     `(5): posterize - A 'posterization' effect where the image uses discrete regions of tone rather than a continuous gradient of tones.` <br />     `(6): whiteboard - A 'whiteboard' effect where the image is typically displayed as regions of white, with black or grey details.` <br />     `(7): blackboard - A 'blackboard' effect where the image is typically displayed as regions of black, with white or grey details.` <br />     `(8): aqua - An 'aqua' effect where a blue hue is added to the image.` <br /> `Flags: readable/writable (changeable in PLAYING state)`    <br /> `Example: effect="aqua" (or) effect=8` |
| `scene` | Scene optimization mode. Options include `action`, `portrait`, `landscape`, `night`, `hdr`, and others.<br /><br />`Type: Enum `<br />`Default: 1, "face-priority"`<br /> `Range:` <br />     `(0): disabled - Indicates that no scene modes are set.` <br />     `(1): face-priority - Optimized for photos of with priority of people faces.` <br />     `(2): action - Optimized for photos of quickly moving objects.` <br />     `(3): portrait - Optimized for still photos of people.` <br />     `(4): landscape - Optimized for photos of distant macroscopic objects.` <br />     `(5): night - Optimized for low-light settings.` <br />     `(6): night-portrait - Optimized for still photos of people in low-light settings.` <br />     `(7): theatre - Optimized for dim, indoor settings where flash must remain off.` <br />     `(8): beach - Optimized for bright, outdoor beach settings.` <br />     `(9): snow - Optimized for bright, outdoor settings containing snow.` <br />     `(10): sunset - Optimized for scenes of the setting sun.` <br />     `(11): steady-photo - Optimized to avoid blurry photos due to small amounts of device motion (for example: due to hand shake).` <br />     `(12): fireworks - Optimized for nighttime photos of fireworks.` <br />     `(13): sports - Optimized for photos of quickly moving people.` <br />     `(14): party - Optimized for dim, indoor settings with multiple moving people.` <br />     `(15): candlelight - Optimized for dim settings where the main light source is a candle.` <br />     `(16): hdr - Turn on a device-specific high dynamic range (HDR) mode.` <br /> `Flags: readable/writable (changeable in PLAYING state)`  <br /> `Example: scene="hdr" (or) scene=16` |

### Advanced ISP / Hardware Properties

| Property | Description |
| - | - |
| `adrc` | Enable Automatic Dynamic Range Compression.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable (changeable in PLAYING state)` |
| `defog-table` | A serialized GstStructure for defog algorithm parameters (vendor tag `org.quic.camera.defog`).<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable (changeable in PLAYING state)` |
| `eis` | Electronic Image Stabilization. When EIS modes are enabled at build time, accepts values such as `eis-off`, `eis-on-single-stream`, or `eis-on-dual-stream`.<br /><br />`Type: Enum`<br />`Default:0, "eis-off"`<br /> `Range:` <br />     `(0): eis-off - EIS is not applied.` <br />     `(1): eis-on-single-stream - EIS is applied on first (non-snapshot) stream. Maximum number of each of preview, video and snapshot streams can be one.` <br />     `(2): eis-on-dual-stream - EIS is applied on both preview and video streams. Maximum number of each of preview, video and snapshot streams can be one.` <br /> `Flags: readable/writable`    <br /> `Example: eis="eis-off" (or) eis=0` |
| `ife-direct-stream` | Enable IFE (Image Front End) direct stream mode. Bypasses IPE and other downstream ISP modules, outputting directly from the IFE. Useful for low-latency or raw-like pipelines.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `infrared-mode` | Infrared LED and cut-filter control such as `off`, `on`, `auto`, `cut-filter-only`, or `cut-filter-disable`.<br /><br />`Type: Enum`<br />`Default: off`<br /> `Range:` <br />     `(0): off - The infrared LED is OFF and cut filter is applied i.e. infrared light is blocked.` <br />     `(1): on - The infrared LED is ON and cut filter is removed i.e. infrared light is allowed.` <br />     `(2): auto - The infrared LED and cut filter are turned ON or OFF dependingon the conditions.` <br />     `(3): cut-filter-only - The infrared LED is turned OFF and cut filter is applied i.e. IR light is blocked.` <br />     `(4): cut-filter-disable - Infrared cut filter is removed allowing IR light to pass. This mode is used for transitioning from 'cut-filter-only' mode i.e. disabling only the cut filter.` <br /> `Flags: readable/writable (changeable in PLAYING state)`  <br /> `Example: infrared-mode="auto" (or) infrared-mode=2` |
| `lcac` | Enable Lateral Chromatic Aberration Correction (LCAC). Reduces color fringing at high-contrast edges.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `ldc` | Enable Lens Distortion Correction (LDC). Corrects barrel and pincushion distortion introduced by the lens.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `ltm-data` | A serialized GstStructure for Local Tone Mapping (LTM) parameters (vendor tag `org.quic.camera.ltmDynamicContrast`).<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable (changeable in PLAYING state)` |
| `noise-reduction` | Temporal Noise Reduction (TNR) mode such as `off`, `fast`, or `hq` (high quality).<br /><br />`Type: Enum`<br />`Default: fast`<br /> `Range:` <br />     `(0): off - No noise reduction filter is applied.` <br />     `(1): fast - TNR (Temoral Noise Reduction) Fast Mode.` <br />     `(2): hq - TNR (Temoral Noise Reduction) High Quality Mode.` <br /> `Flags: readable/writable (changeable in PLAYING state)`  <br /> `Example: noise-reduction="fast" (or) noise-reduction=1` |
| `noise-reduction-tuning` | A serialized GstStructure for custom ANR tuning parameters (vendor tag `org.quic.camera.anr_tuning`).<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable (changeable in PLAYING state)` |
| `shdr` / `vhdr` | Super or Video High Dynamic Range. When VHDR modes are enabled at build time, supports values such as `off`, `shdr-raw`, `shdr-yuv`, `raw-shdr-switch`, `yuv-shdr-switch`, `qbc-hdr-video`, and `qbc-hdr-snapshot`.<br /><br />`Type: Enum `<br />`Default: false / off`<br /> `Range:` <br />     `(0): off - VHDR is disabled.` <br />     `(1): shdr-raw - Raw SHDR line interleaved mode with 2 frame. Use this mode for better performance on supporting sensor.` <br />     `(2): shdr-yuv - YUV SHDR virtual channel mode with 2 frames. Use this mode for better quality on supporting sensor. This mode may result in reduced framerate.` <br />     `(3): raw-shdr-switch - Enable Raw SHDR switch. Use this mode for enabling shdr switch in camera backend based on lux value. The switch is between linear and Raw SHDR based on support in camera.` <br />     `(4): yuv-shdr-switch - Enable YUV SHDR switch. Use this mode for enabling shdr switch in camera backend based on lux value. The switch is between linear and YUV SHDR based on support in camera.` <br />     `(5): qbc-hdr-video - Enable in-sensor HDR for video stream. This mode is applicable for sensor that support this feature only.` <br />     `(6): qbc-hdr-snapshot - Enable in-sensor HDR for snapshot. When enabled camera backend decides to enable in-sensor hdr for snapshot based on the scene. This mode is applicable for sensor that support this feature only.` <br /> `Flags: readable/writable (changeable in PLAYING state)` <br /> `Example: vhdr="off" (or) vhdr=0` |
| `sw-tnr` | Enable software-based Temporal Noise Reduction. Only available if the platform supports it.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |

### Zoom Properties

| Property | Description |
| - | - |
| `zoom` | Zoom rectangle in the format `<X, Y, WIDTH, HEIGHT>` defined in sensor active pixel array coordinates. Defaults to the full active sensor size (1× zoom). Setting a smaller rectangle results in digital zoom.<br /><br />`Type: GstValueArray of type gint`<br />`Default: sensor active size`<br />`Flags: readable/writable (changeable in PLAYING state)` |

### Frame Rate Control

| Property | Description |
| - | - |
| `frc-mode` | Stream frame rate control mode. `frame-skip` drops frames at the HAL level, while `capture-request` controls the rate via camera capture requests.<br /><br />`Type: Enum `<br />`Default: 0, "frame-skip"`<br /> `Range:` <br />     `(0): frame-skip - Control stream frame rate by frame skip` <br />     `(1): capture-request - Control stream frame rate by camera capture request` <br /> `Flags: readable/writable` <br /> `Example: frc-mode="frame-skip" (or) frc-mode=0` |

### Metadata Properties

| Property | Description |
| - | - |
| `image-metadata` | Returns the CameraMetadata object used for still image capture requests (read-only).<br /><br />`Type: Pointer (CameraMetadata)`<br />`Default: NULL`<br />`Flags: readable (available in PLAYING state)` |
| `session-metadata` | A CameraMetadata object used for stream configuration (write-only).<br /><br />`Type: Pointer (CameraMetadata)`<br />`Default: NULL`<br />`Flags: writable (changeable in PLAYING state)` |
| `video-metadata` | A CameraMetadata object used for per-frame capture request settings on video streams. The caller is responsible for releasing the object.<br /><br />`Type: Pointer (CameraMetadata)`<br />`Default: NULL`<br />`Flags: readable/writable (changeable in PLAYING state)` |

### MultiCamera Properties

| Property | Description |
| - | - |
| `camera-switch-index` | For logical multi-camera SAT mode, selects which physical sensor to use by index. A value of -1 selects the next valid physical camera. Only available on platforms with logical camera support.<br /><br />`Type: Integer`<br />`Default: -1`<br />`Flags: readable/writable (changeable in PLAYING state)` |
| `multicamera-hint` | Enables offline IFE mode for multi-camera setups where the number of sensors exceeds the available hardware ISP pipelines. This impacts memory usage and latency.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `multi-camera-exp-time` | Per-camera exposure times in the format `<exp-time-1, exp-time-2>` expressed in nanoseconds for multi-camera setups. Only available on platforms with logical camera support. Active when `exposure-mode=off`.<br /><br />`Type: GstValueArray of type gint64`<br />`Default: NA`<br />`Flags: readable/writable (changeable in PLAYING state)` |

### Operation Mode and ROI

| Property | Description |
| - | - |
| `input-roi-enable` | Enable Input Region of Interest (ROI) reprocessing use case.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `input-roi-info` | ROI coordinates provided as an array in the format `<X1, Y1, W1, H1, X2, Y2, W2, H2, …>`. Used only when `input-roi-enable=true`.<br /><br />`Type: GstValueArray of type gint`<br />`Default: NA`<br />`Flags: readable/writable (changeable in PLAYING state)` |
| `op-mode` | Camera operation mode flags. Supports values such as `none`, `frameselection`, and `fastswitch`. Multiple flags can be combined.<br /><br />`Type: Flags (GstFrameSelection)`<br />`Default: none`<br />`Flags: readable/writable` |

### Pad Activation Mode

| Property | Description |
| - | - |
| `video-pads-activation-mode` | Controls how video pad activation is managed. In `normal` mode, pads activate or deactivate automatically with the pipeline state. In `signal` mode, pad activation is controlled manually via the `video-pads-activation` signal together with `gst_pad_set_active()`.<br /><br />`Type: Enum `<br />`Default: normal`<br />`Flags: readable/writable (changeable in PLAYING state)` |

### Video Pad Properties (`video_%u`)

These properties are set on individual video pads using the GstChildProxy interface (e.g., `src.video_0::framerate=60`).

| Property | Description |
| - | - |
| `attach-cam-meta` | When enabled, attaches camera result metadata as a `GstCameraMeta` structure to each GstBuffer. Requires the downstream element to send a `need-metadata` custom query.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `crop` | Crop rectangle in the format `<X, Y, WIDTH, HEIGHT>`. Applicable for JPEG and YUY2 formats. A value of `<0,0,0,0>` means no crop.<br /><br />`Type: GstValueArray of type gint`<br />`Default: "<0,0,0,0>"`<br />`Flags: readable/writable (changeable in PLAYING state)` |
| `extra-buffers` | Number of additional buffers to allocate beyond the minimum required.<br /><br />`Type: Unsigned Integer`<br />`Default: 0`<br />`Flags: readable/writable (changeable in READY state)` |
| `framerate` | Target display framerate in frames per second.<br /><br />`Type: Double`<br />`Default: 30.0`<br />`Range: 0.0 - 30.0`<br />`Flags: readable/writable (changeable in PLAYING state)` |
| `logical-stream-type` | For logical multi-camera setups, selects a specific physical camera by index (`camera-index-0` to `camera-index-15`), a stitching layout (`sidebyside`, `panorama`), or `none` for the default logical output.<br /><br />`Type: Enum`<br />`Default: none`<br />`Flags: readable/writable (changeable in PAUSED state)` |
| `reprocess-enable` | Marks this pad as a reprocess pad. The stream will be used as input for a reprocessing pipeline.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable (changeable in READY state)` |
| `rotate` | Stream rotation such as `none`, `90CCW`, `180CCW`, or `270CCW`.<br /><br />`Type: Enum`<br />`Default: none`<br />`Flags: readable/writable` |
| `source-index` | Index of a source video pad to link this pad to as a reprocess input. A value of -1 means no source linkage.<br /><br />`Type: Integer`<br />`Default: -1`<br />`Flags: readable/writable (changeable in READY state)` |
| `super-buffer-mode` | Enables Super Buffer Mode for High Frame Rate (HFR) capture. In this mode, each buffer contains multiple frames. The batch size is calculated as `framerate / superframerate`. Requires a fixed framerate greater than or equal to 60 fps.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable (changeable in READY state)` |
| `type` | Stream tuning type such as `video` for encoding optimization or `preview` for display optimization.<br /><br />`Type: Enum`<br />`Default: video`<br />`Flags: readable/writable (changeable in PLAYING state)` |

### Image Pad Properties (`image_%u`)

| Property | Description |
| - | - |
| `logical-stream-type` | For logical multi-camera setups, selects a specific physical camera, a stitching layout, or `pd` (Phase Detection data stream).<br /><br />`Type: Enum `<br />`Default: none`<br />`Flags: readable/writable (changeable in PAUSED state)` |
| `rotate` | Image stream rotation such as `none`, `90CCW`, `180CCW`, or `270CCW`.<br /><br />`Type: Enum `<br />`Default: none`<br />`Flags: readable/writable` |

## Signals

qticamsrc exposes the following GObject signals for programmatic control of image capture and pipeline events.

### Action Signals (callable via `g_signal_emit_by_name`)

| Signals | Description |
| - | - |
| `cancel-capture` | Cancels any pending image capture requests.<br /><br />`Type: Signal (gboolean return)`<br />`Parameters: none` |
| `capture-image` | Triggers a still image capture. `imgtype` can be `video` (uses current video metadata) or `still` (uses per-frame metadata from metas). `n_images` specifies the burst count, and `metas` is an optional array of CameraMetadata objects, one per image.<br /><br />`Type: Signal (gboolean return)`<br />`Parameters: (GstImageCaptureMode imgtype, guint n_images, GPtrArray metas)` |
| `dynamic-capture-image` | Triggers a dynamic capture with a custom request group string. `req_group` defines the capture request grouping pattern. `imgtype`, `n_burst`, and `metas` behave similarly to `capture-image`.<br /><br />`Type: Signal (gboolean return)`<br />`Parameters: (GstImageCaptureMode imgtype, gchar req_group, guint n_burst, GPtrArray metas)` |
| `video-pads-activation` | Activates or deactivates specific video pads by name. Effective only when `video-pads-activation-mode=signal`.<br /><br />`Type: Signal (gboolean return)`<br />`Parameters: (gboolean activate, GPtrArray padnames)` |

### Notification Signals (emitted by the element)

| Property | Description |
| - | - |
| `device-status-change` | Emitted when a camera device's physical presence status changes, such as hot-plug or unplug events.<br /><br />`Type: Signal`<br />`Parameters: (guint camera_id, gboolean is_present)` |
| `result-metadata` | Emitted when a full result CameraMetadata frame is received from the camera HAL. The metadata pointer is a `const CameraMetadata*`.<br /><br />`Type: Signal`<br />`Parameters: (gpointer metadata)` |
| `urgent-metadata` | Emitted when an urgent (partial) CameraMetadata frame is received. Urgent metadata arrives before the full result and does not contain `ANDROID_SENSOR_TIMESTAMP`.<br /><br />`Type: Signal`<br />`Parameters: (gpointer metadata)` |

## Camera Event Handling

`qticamsrc` handles camera service events internally and propagates them as GStreamer element messages and signals:

| Event | Handling |
| - | - |
| `EVENT_SERVICE_DIED` | Posts `GST_ELEMENT_ERROR` (RESOURCE / NOT\_FOUND) |
| `EVENT_CAMERA_ERROR` | Posts `GST_ELEMENT_ERROR` (RESOURCE / FAILED) |
| `EVENT_CAMERA_CLOSING` | Pushes EOS to all source pads (slave mode only) |
| `EVENT_FRAME_ERROR` | Posts `GST_WARNING` (non-fatal frame drop) |
| `EVENT_METADATA_ERROR` | Posts `GST_WARNING` (non-fatal metadata drop) |
| `EVENT_SOF_FREEZE` | Logs warning |
| `EVENT_RECOVERYFAILURE` | Logs warning |
| `EVENT_RECOVERYSUCCESS` | Logs info |
| `EVENT_INTERNAL_RECOVERY` | Logs info |
| `EVENT_DEVICE_STATUS_CHANGE` | Emits `device-status-change` signal with `camera_id` and `is_present` |

## Internal Architecture Details

`qticamsrc` is built around a client-server camera architecture in which the GStreamer element acts as a client of the camera service. The camera service manages the underlying camera HAL, stream creation, buffer allocation, and capture control, while `qticamsrc` exposes those capabilities to the GStreamer pipeline through source pads, camera properties, and control signals.

At a high level, the element:

* opens and configures the selected camera device
* creates and manages video and image output streams
* receives captured frames from the `camera service`
* wraps camera buffers as `GstBuffer` objects
* pushes those buffers downstream through the corresponding source pads
* returns buffers to the camera service when downstream processing is complete

This design allows `qticamsrc` to provide zero-copy camera output while keeping ownership of camera buffers and stream resources in the service layer.

### Core Internal Components

`qticamsrc` is organized around four main internal components:

#### `GstQmmfContext` — camera session manager

A C++ wrapper around the camera service API. It is the primary interface between the plugin and the camera service, and is responsible for camera open and close operations, stream creation and deletion, image capture requests, translation of plugin properties into camera metadata, and per-frame metadata synchronization.

#### `GstQmmfSrcVideoPad` — video output pad implementation

A custom pad implementation used for each `video_%u` source pad. Each video pad owns its own queue, worker thread, and reusable buffer pool. Frames received from the camera service are queued per pad and then pushed downstream asynchronously.

#### `GstQmmfSrcImagePad` — image capture pad implementation

A pad implementation used for each `image_%u` source pad. It is similar in structure to the video pad implementation, but is used for still-image capture workflows rather than continuous video streaming.

#### `GstQmmfBufferPool` — reusable buffer wrapper pool

A lightweight buffer pool used to reuse GstBuffer wrapper objects. The pool does not allocate camera memory itself; instead, it provides reusable GstBuffer shells that are populated with camera-provided DMA-backed memory when frames arrive.

## Plugin State Management

The operation of `qticamsrc` is closely tied to normal GStreamer state transitions.

**`NULL → READY`**

The element opens and initializes the camera device selected by the `camera` property.

**`READY → PAUSED`**

The negotiated pad capabilities are translated into camera stream configuration, and the required streams are prepared through the camera service.

**`PAUSED → PLAYING`**

The prepared streams are activated. For video pads, capture requests are submitted and frame delivery begins.

This state-driven design ensures that camera initialization, stream setup, and frame acquisition are aligned with the pipeline lifecycle.

## Stream Types

`qticamsrc` supports two stream types: video streams and image streams.

### Video Pads

Video pads (`video_%u`) produce a continuous stream of buffers at the negotiated frame rate. They are typically used for:

* preview
* video encoding
* display
* streaming
* AI inference

Each video pad can be configured independently. The `camera service` selects an appropriate internal buffer count based on stream properties such as resolution, format, and frame rate. If additional buffering is required, it can be requested through the `extra-buffers` property.

Video orientation can be adjusted in 90-degree increments through the `rotate` property.

For high-frame-rate operation, when the negotiated frame rate is 60 FPS or higher, the stream enters HFR mode. In this mode, the `super-buffer-mode` property becomes available. When enabled, a single `GstBuffer` may contain multiple captured frames. The number of frames per buffer is computed as:

$$
\text{frames per buffer} = \frac{\text{negotiated frame rate}}{60}
$$

For example, at 120 FPS, each output buffer contains 2 frames.

By default, video pads begin streaming automatically when the element enters the PLAYING state. This behavior can be changed through `video-pads-activation-mode`. When configured for signal-based activation, streaming starts and stops only when controlled explicitly through the `video-pads-activation` signal.

### Image Pads

Image pads (`image_%u`) are used for on-demand image capture. Unlike video pads, they do not stream continuously. Image capture is triggered by the application through the `capture-image` signal, which specifies:

* the image type
* the number of images to capture
* optional camera metadata

If multiple image captures are requested, the operation can be canceled through the `cancel-capture` signal.

Image orientation can also be adjusted in 90-degree increments through the `rotate` property.

The number of simultaneously supported image pads is platform-dependent. All supported platforms allow multiple video pads, while support for multiple image pads depends on chipset capability.

## Buffer Handling and Data Flow

When a frame is received from the camera service, `qticamsrc` obtains a reusable GstBuffer wrapper from the corresponding pad's `GstQmmfBufferPool`. The incoming camera buffer is then wrapped as `GstFdMemory`, and the resulting GstBuffer is queued on the target pad.

Each source pad has its own internal queue and worker thread. The worker thread dequeues buffers and pushes them downstream through the connected GStreamer pad.

This design separates:

* camera callback handling
* buffer wrapping
* downstream `gst_pad_push()` execution

As a result, frame acquisition remains decoupled from downstream pipeline timing, which helps maintain stable capture behavior even when downstream elements operate at different speeds.

Buffers produced by `qticamsrc` are **DMA-backed file-descriptor buffers**, making them compatible with zero-copy downstream processing when the rest of the pipeline supports FD-backed memory.

### Output Buffer Pool Behavior

Each video pad and image pad owns a dedicated `GstQmmfBufferPool`. This pool is used only to reuse GstBuffer wrapper objects and reduce the overhead of repeatedly creating and destroying GStreamer buffers.

The pool behaves as follows:

* it maintains a small number of reusable GstBuffer shells
* it does not allocate the actual camera frame memory
* frame memory is attached later when a capture callback is received
* when a buffer is returned to the pool, the wrapper is cleaned by removing attached memory and auxiliary data before reuse

This keeps wrapper reuse efficient while ensuring that stale memory references are not retained across frames.

### DMA Buffer Lifecycle

The actual frame buffers are allocated and owned by camera service, not by `qticamsrc` plugin.

When a frame arrives:

* the camera-provided file descriptor is wrapped as `GstFdMemory`
* the memory is marked with `GST_FD_MEMORY_FLAG_DONT_CLOSE`
* GStreamer is therefore prevented from closing the underlying file descriptor

Additional buffer return information is stored with the GstBuffer so that the plugin can return the buffer to the camera service when downstream processing is complete.

When the final reference to the GstBuffer is released:

* a cleanup callback reconstructs the information needed to identify the original camera buffer
* the buffer is returned to the camera service
* any associated metadata references are released

This mechanism ensures that buffers are safely recycled back into the camera pipeline while preserving zero-copy delivery downstream.

### Metadata Handling

`qticamsrc` supports attachment of camera metadata to captured frames. When metadata attachment is enabled for a video pad, per-frame camera metadata is matched to the corresponding frame using timestamps before the buffer is pushed downstream.

The element also exposes a range of camera-control properties that map to underlying camera metadata settings, including controls for tuning and capture behavior. These settings provide a GStreamer-facing interface for configuring standard camera parameters and Qualcomm-specific camera features.

## Camera Selection and Multi-Instance Behavior

The camera device used by a `qticamsrc` instance is selected through the `camera` property. If no camera is specified, the element uses the first detected sensor by default.

Under normal operation, multiple instances of `qticamsrc` cannot open the same camera ID at the same time. An exception exists through **slave mode**:

* a primary instance owns the camera and controls its configuration and lifetime
* a secondary instance can access the same camera by enabling the `slave` property

In slave mode, the secondary instance can request additional output streams from the same camera, but it cannot:

* change camera tuning parameters
* modify camera settings
* control camera lifetime

This allows multiple GStreamer branches or clients to share the same physical camera while keeping ownership centralized in the primary instance.

### Multi-Camera and Multi-Client Operation

`qticamsrc` also supports multi-camera workflows, where each plugin instance corresponds to a single physical camera device. This allows applications to instantiate multiple `qticamsrc` elements to access and process streams from multiple cameras concurrently. For example, a dual-camera configuration can be implemented by creating two separate plugin instances, one for each physical camera. The maximum number of cameras that can be used simultaneously is determined by the hardware capabilities of the target Qualcomm chipset and its ISP capabilities.

<img alt="" />

## Usage

<Steps>
  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>
</Steps>

### Live Camera Preview

This example demonstrates a live camera preview pipeline that captures 1080p frames from the ISP camera using `qticamsrc` and renders them to the display using the `waylandsink` plugin.

```bash theme={null}
gst-launch-1.0 qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! waylandsink fullscreen=true sync=true
```

<img alt="" />

### Live Camera Preview and Image Capture

This example demonstrates live snapshot capture while a live preview stream continues to run in parallel. The `gst-pipeline-app` application is used to trigger the capture signal while the preview pipeline remains active.

```bash theme={null}
gst-pipeline-app -e qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! waylandsink sync=true fullscreen=true camsrc.image_1 ! image/jpeg,width=1920,height=1080,framerate=30/1 ! multifilesink location=$HOME/Downloads/qimsdk_samples/media/output/frame%d.jpg sync=true async=false
```

```
#Choose options in the order displayed.
#Use the alpha-numeric characters displayed corresponding to each of the below options:
PLAYING
Plugin Mode
camsrc
capture-image
still
3
```

<img alt="" />

### Two Streams Use Case

This example demonstrates simultaneous capture of two output streams from the camera source plugin. Both streams are processed in parallel, encoded independently, and then muxed into MP4 output.

```bash theme={null}
gst-launch-1.0 -e qticamsrc name=src \
  video_0::type=preview ! video/x-raw,format=NV12_Q08C,width=3840,height=2160,framerate=30/1 ! \
  queue ! v4l2h264enc ! h264parse ! mp4mux ! \
  filesink location=$HOME/Downloads/qimsdk_samples/media/output/4k_video.mp4 async=false \
  src.video_1 ! video/x-raw,format=NV12,width=1280,height=720,framerate=30/1 ! \
  queue ! v4l2h264enc ! h264parse ! mp4mux ! \
  filesink location=$HOME/Downloads/qimsdk_samples/media/output/720p_video.mp4 async=false
```

<img alt="" />

### Three Stream Use Case

This example demonstrates simultaneous camera capture for three parallel streams: a live preview stream, a 1080p H.264 recording stream muxed into an MP4 file, and a periodic JPEG snapshot capture stream.

```bash theme={null}
gst-pipeline-app -e \
  qticamsrc name=camsrc \
  video_0::type=preview \
  video_1::type=video \
  image_0::type=jpeg \
  \
  camsrc.video_0 ! queue ! \
  video/x-raw,format=NV12_Q08C,width=1280,height=720,framerate=30/1 ! \
  waylandsink sync=true fullscreen=true \
  \
  camsrc.video_1 ! queue ! \
  video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
  v4l2h264enc ! h264parse ! mp4mux ! \
  filesink location=$HOME/Downloads/qimsdk_samples/media/output/720p_video.mp4 async=false \
  \
  camsrc.image_2 ! \
  image/jpeg,width=1920,height=1080,framerate=5/1 ! \
  multifilesink location=$HOME/Downloads/qimsdk_samples/media/output/frame_%04d.jpg sync=true async=false
```

```
#Choose options in the order displayed.
#Use the alpha-numeric characters displayed corresponding to each of the below options:
PLAYING
Plugin Mode
camsrc
capture-image
still
3
```

<img alt="" />

### Dual Camera Use Case

This example demonstrates a dual-camera pipeline in which each camera produces two output streams. One stream from each camera is composited side-by-side for live display, while the second stream from each camera is encoded independently and saved as an MP4 file.

```bash theme={null}
gst-launch-1.0 -e \
  qticamsrc name=cam0 camera=0 \
  video_0::type=preview \
  video_1::type=video \
  \
  qticamsrc name=cam1 camera=1 \
  video_0::type=video \
  video_1::type=video \
  \
  qtivcomposer name=mixer \
  sink_0::position="<0,0>" sink_0::dimensions="<640,480>" \
  sink_1::position="<640,0>" sink_1::dimensions="<640,480>" \
  \
  mixer. ! queue ! waylandsink fullscreen=true \
  \
  cam0.video_0 ! queue ! \
  video/x-raw,format=NV12_Q08C,width=640,height=480,framerate=30/1 ! \
  mixer.sink_0 \
  \
  cam1.video_0 ! queue ! \
  video/x-raw,format=NV12_Q08C,width=640,height=480,framerate=30/1 ! \
  mixer.sink_1 \
  \
  cam0.video_1 ! queue ! \
  video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
  v4l2h264enc ! h264parse ! mp4mux ! \
  filesink location=$HOME/Downloads/qimsdk_samples/media/output/cam0.mp4 async=false \
  \
  cam1.video_1 ! queue ! \
  video/x-raw,format=NV12,width=1280,height=720,framerate=30/1 ! \
  v4l2h264enc ! h264parse ! mp4mux ! \
  filesink location=$HOME/Downloads/qimsdk_samples/media/output/cam1.mp4 async=false
```

<img alt="" />


# qtimetamux
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtimetamux

AI metadata synchronization and muxing plugin

# Overview

The **qtimetamux** element is a core component of an AI-enabled GStreamer pipeline. Its purpose is to
synchronize post-processed AI/CV results with the original media buffer and attach those results as
**GstMeta** using the standard metadata mechanism provided by GStreamer.

<img alt="" />

In practice, outputs generated by ML post-processing stages — such as:

* **Bounding box coordinates**
* **Class labels**
* **Segmentation masks**
* **Key points**
* **Motion vectors**
* **Other custom AI/CV metadata**

Can be associated with the corresponding video or audio frame and carried forward through the
pipeline as a single, unified buffer.

This design makes it easier to build pipelines where inference results remain tightly coupled with
the original frame. Downstream components can consume both the media buffer and its metadata without
needing separate synchronization logic.

By embedding metadata directly into the frame, **qtimetamux** enables several common AI pipeline patterns:

* **Live visualization** — Metadata can be consumed by overlay elements such as [`qtivoverlay`](/plugin-reference/qtivoverlay) to
  render bounding boxes, labels, and other inference results directly on the video output.
* **Daisy-chained AI pipelines** — The metadata-bearing buffer can be passed to a subsequent
  inference stage, allowing multi-stage AI workflows where the output of one model feeds the next.
* **Application-level access** — The resulting buffer can be sent to an `appsink`, giving a custom
  application access to both the media frame and the attached metadata for business logic or
  decision-making.
* **Metadata serialization and external integration** — The metadata can be forwarded to
  `qtimlmetaparser`, which converts it into JSON. That JSON can then be published to external
  systems such as MQTT, Kafka, or a REDIS server via [`qtiredissink`](/plugin-reference/qtiredissink).

In addition to AI inference outputs, **qtimetamux** is also capable of attaching other metadata
types such as **motion vectors**, making it useful for both AI and broader computer-vision-based
workflows.

<img alt="qtimetamux_workflow" />

## Example Pipeline

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp yolox_w8a8.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/models $HOME/Downloads/qimsdk_samples/labels $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/media/output
    export MODEL_NAME=yolox_w8a8.tflite
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
    t. ! queue ! qtimlvconverter ! queue ! \
    qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
      external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
    qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME \
      settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! obj_mux.
    ```
  </Step>
</Steps>

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />qtimetamux

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-raw(ANY)` | `format: NA` |
| `audio/x-raw(ANY)` | `format: NA` |
| Availability: *Always* | |
| Direction: *sink* | |
| Pad Name: `sink` | |

| Capabilities | |
| - | - |
| `text/x-raw` | `format: utf8` |
| `cv/x-optical-flow` | `format: NA` |
| Availability: *On request* | |
| Direction: *sink* | |
| Pad Name: `data_%u` | |

### src

| Capabilities | |
| - | - |
| `video/x-raw(ANY)` | `format: NA` |
| `audio/x-raw(ANY)` | `format: NA` |
| Availability: *Always* | |
| Direction: *source* | |

# Element Properties

| Property | Description |
| - | - |
| `latency` | Additional latency in nanoseconds to allow more time for upstream to produce metadata entries for the current position. Useful in sync mode when metadata generation takes longer than the default hold window.<br /><br />`Type: Unsigned Integer64`<br />`Default: 0`<br />`Range: 0 - 18446744073709551615`<br />`Flags: readable/writable (changeable only in NULL or READY state)` |
| `mode` | Controls the synchronization strategy used to associate metadata buffers with main media frames.<br /><br />`Type: Enum `<br />`Default: 0, "async"`<br />`Range:`<br />    `(0): async - No timestamp synchronization. The N-th incoming media frame is held until the N-th data buffer has been received on all data pads. Suitable for fixed, predictable sequences`<br />    `(1): sync - Timestamp-based synchronization. Each incoming frame is held for up to 1 / framerate (video) or 1 / rate (audio). Metadata with matching timestamps is attached before the frame is forwarded downstream`<br />`Flags: readable/writable (changeable only in NULL or READY state)` <br /> `Example: mode="sync" (or) mode=1` |
| `queue-size` | Sets the size of the internal input and output queues.<br /><br />`Type: Unsigned Integer`<br />`Default: 10`<br />`Range: 3 - 4294967295`<br />`Flags: readable/writable (changeable only in NULL or READY state)` |

## Main Buffer, Metadata Synchronization and Latency control

The plugin is designed with a **single main sink pad** that receives the primary video or audio
buffers, and **multiple auxiliary data pads** that collect ML post-processing results or CV motion
vectors. Data arriving on auxiliary pads may be provided in string or blob form and is parsed into
structured representations. Once parsed, the plugin matches each data buffer to its corresponding
main media frame and attaches the result as **GstMeta**.

## Async Mode

This is the default synchronization mode. No timestamp-based matching is performed. Instead,
metadata buffers are associated with main frames in strict **1:1 order**:

* The N-th incoming video/audio frame is held until the N-th data buffer has been received on **all** data pads.
* Once all required data for that frame is available, the metadata is attached.
* The enriched buffer is then pushed downstream.

This mode is suitable when media buffers and metadata buffers are produced in a fixed, predictable sequence.

## Sync Mode

In sync mode, the plugin performs **timestamp-based synchronization**. Each incoming main frame is
held for a limited time window of up to `1 / framerate` seconds (video) or `1 / rate` seconds
(audio). For example, at 30 fps, the frame may be held for approximately **33.3 ms**.

During this hold period, the plugin waits for data buffers on its auxiliary pads whose timestamps
match the timestamp of the main frame:

* If all expected data buffers arrive within the time window, they are attached before forwarding.
* If one or more auxiliary pads do not provide matching buffers in time, only the successfully
  matched metadata is attached and the main buffer is released downstream.

## Latency Control

In some use cases, the default hold period in sync mode may be too short — especially when metadata
generation takes longer than expected. The `latency` property extends the waiting period by
accepting an integer value in **nanoseconds**, allowing the plugin to wait longer for late-arriving
data buffers before forwarding the main frame.

<img alt="qtimetamux_latency_control" />

# Usage

### Person Detection

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp yolox_w8a8.tflite           <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolox_w8a8.tflite
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
    t. ! queue ! qtimlvconverter ! queue ! \
    qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
    external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
    qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME \
    settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! obj_mux.
    ```
  </Step>
</Steps>

### Detection-Classification Daisy Chain Pipeline

This pipeline demonstrates a cascaded inference approach where the output of one model (Detection) is used to crop regions of interest (ROIs) which are then fed into secondary models (Classification).

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | YOLO labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | MobileNet model | <a href="https://aihub.qualcomm.com/iot/models/mobilenet_v2">mobilenet-softmax</a> | `mobilenet_v2_w8a8.tflite` |
    | MobileNet labels | <a href="../labels/mobilenet.json">mobilenet.json</a> | `mobilenet_v2.json` |
    | Input video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
      scp yolox_w8a8.tflite                     <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp mobilenet_v2_w8a8.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp mobilenet_v2.json                            <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4                                    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/models $HOME/Downloads/qimsdk_samples/labels $HOME/Downloads/qimsdk_samples/media
    export MODEL_NAME_1=yolox_w8a8.tflite
    export LABELS_NAME_1=yolov8.json
    export MODEL_NAME_2=mobilenet_v2_w8a8.tflite
    export LABELS_NAME_2=mobilenet_v2.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
      qtimlvconverter name=det_conv \
      qtimltflite name=det_infer delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_1 \
      qtimlpostprocess name=det_post module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_1 settings="{\"confidence\": 51.0}" \
      qtimetamux name=det_mux \
      qtivoverlay name=main_overlay \
      qtimlvconverter name=cls_conv \
      qtimltflite name=cls_infer delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_2 \
      qtimlpostprocess name=cls_post module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_2 settings="{\"confidence\": 51.0}" \
      qtimetamux name=cls_mux \
      qtivoverlay name=cls_overlay \
      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! tee name=src_tee \
      src_tee. ! queue ! det_mux. \
      src_tee. ! queue ! det_conv. det_conv. ! queue ! det_infer. det_infer. ! queue ! det_post. det_post. ! text/x-raw ! queue ! det_mux. \
      det_mux. ! queue ! tee name=meta_tee \
      meta_tee. ! queue ! cls_mux. \
      meta_tee. ! queue ! cls_conv. cls_conv. ! queue ! cls_infer. cls_infer. ! queue ! cls_post. cls_post. ! text/x-raw ! queue ! cls_mux. \
      cls_mux. ! queue ! cls_overlay. cls_overlay. ! queue ! waylandsink fullscreen=true sync=true
    ```
  </Step>
</Steps>


# qtimlmetaparser
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtimetaparser

Serializes ML inference metadata from GStreamer buffers into text.

# Overview

`qtimlmetaparser` is a GStreamer element that converts machine learning metadata into a human-readable and portable text representation. Its primary purpose is to transform inference results attached to GStreamer buffers into **UTF-8 encoded text**, making the metadata easier to inspect, log, store, or forward to external systems.

Regardless of the input format, `qtimlmetaparser` focuses on extracting and converting the associated internal ML metadata into a standardized textual representation.

### Runtime Parser Architecture

`qtimlmetaparser` does not implement metadata translation logic internally. Instead, it serves as a lightweight wrapper that dynamically loads a runtime parser module and delegates the actual metadata conversion to that module. This design keeps the element extensible and allows different output formats to be supported without modifying the core element implementation.

One currently supported runtime module is the JSON parser module (`ml-meta-parser-json`). This module traverses the supported ML metadata structures and serializes them into a JSON document. The generated JSON output can then be used by downstream applications or external components for:

* logging and debugging
* result archival
* analytics pipelines
* inter-process communication
* integration with non-GStreamer software stacks

By separating the parsing backend from the element itself, `qtimlmetaparser` provides a flexible mechanism for converting ML metadata into portable text formats suitable for both developer inspection and system-level integration.

<img alt="qtimetamux_workflow" />

## Example Pipeline

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp yolox_w8a8.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolox_w8a8.tflite
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! tee name=t_out \
    t_out. ! queue ! waylandsink fullscreen=true sync=true \
    t_out. ! queue ! qtimlmetaparser module=json ! filesink location=$HOME/Downloads/qimsdk_samples/media/meta.json \
    t. ! queue ! qtimlvconverter ! queue ! \
    qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
      external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
    qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME \
      settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! obj_mux.
    ```
  </Step>
</Steps>

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />[GstBaseTransform](https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c)<br />
            <Icon icon="arrow-turn-down-right" />qtimlmetaparser

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `image/jpeg` | `format: NA` |
| `video/x-raw(ANY)` | `format: NA` |
| `text/x-raw` | `format: utf8` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `text/x-raw` | `format: utf8` |
| Availability: *Always* | |
| Direction: *source* | |

# Element Properties

| Property | Description |
| - | - |
| `module` | Module name used for parsing metadata.<br /><br />`Type: Enum `<br />`Default: 0, "none"`<br />`Range:`<br />    `(0): none - No module, default invalid mode`<br />    `(1): json - ml-meta-parser-json`<br />`Flags: readable/writable (changeable only in NULL or READY state)` `Example: module="json" (or) module=1` |
| `module-params` | Parameters specific to the chosen module for parsing buffer metadata. Format is a GstStructure string.<br /><br />`Type: String`<br />`Default: params`<br />`Flags: readable/writable` |

## Internal Architecture

`qtimlmetaparser` is organized as a three-layer architecture composed of:

1. GStreamer element
2. Dynamic module loader
3. Parser module

This separation of responsibilities keeps the element lightweight, simplifies extensibility, and allows metadata translation logic to evolve independently of the GStreamer integration layer.

### GStreamer Element

At the top level, `qtimlmetaparser` is exposed as a standard GStreamer element with two pads:

* **Sink pad** <br />
  Accepts input buffers from upstream elements. Supported buffer types include JPEG images, raw video frames, and UTF-8 text buffers.

* **Source pad** <br />
  Produces a UTF-8 encoded, human-readable textual representation of the machine learning metadata associated with the input buffer.

### Dynamic Module Loader

The module loader provides the abstraction layer between the GStreamer element and the parser backend. Its role is to load a parser implementation from a shared object at runtime and expose a small, stable interface to the element.

This design provides two key benefits:

* **Extensibility** <br />
  New metadata translation backends can be added without changing the core element implementation.

* **Simplified module development** <br />
  A parser backend only needs to implement the predefined module interface. It does not need to be implemented as a GStreamer element or include any pipeline-specific logic.

By decoupling parser logic from the element, the loader enables a modular architecture in which translation formats can be added, replaced, or updated independently.

### Parser Module Implementation

The parser module implementation defines how machine learning metadata is traversed, interpreted, and converted into UTF-8 text.

Each module is responsible for translating supported metadata structures into a portable textual representation suitable for downstream consumption.

The following section provides additional details about the **Module System** and the parser module interface..

## Module System

`qtimlmetaparser` uses a lightweight runtime module system to delegate metadata serialization to a shared library loaded at runtime. This allows new output formats to be added without modifying the element.

A parser module is responsible for converting supported ML metadata attached to an input buffer into UTF-8 text output.

Typical metadata categories include:

**Object Detection**— label, confidence, color, and normalized bounding box
**Landmarks / Pose** — keypoints and optional links
**Text** — OCR output, captions, or other textual annotations, including VLM results
**Classification** — labels and scores from image or audio classification

## JSON Module

The currently supported parser module is the JSON module. It is selected through the element’s module property:

```bash theme={null}
qtimlmetaparser module=json
```

The JSON module serializes the ML metadata attached to the input buffer into JSON text output. This allows downstream elements to store, display, or transmit inference results without needing to interpret GStreamer metadata directly.

The module also supports the attach-frame parameter through the module-params property. When enabled for a JPEG input buffer, the module maps the input buffer, Base64-encodes the JPEG data, and embeds it in the generated JSON output.

```bash theme={null}
qtimlmetaparser module=json module-params="params,attach-frame=true"
```

This allows inference metadata and the corresponding frame snapshot to be carried in a single JSON payload. If the input buffer is not JPEG-encoded, the frame is not attached even when attach-frame=true, and the output contains only the serialized metadata.

### Example JSON Output

The following example shows object detection metadata serialized as an object\_detection array, along with an attached Base64-encoded JPEG buffer. The buffer contents are truncated for readability.

```json theme={null}
{
  "object_detection": [
    {
      "label": "person",
      "confidence": 84.00863647460938,
      "color": 16711935,
      "rectangle": {
        "x": 0.26145833333333335,
        "y": 0.3037037037037037,
        "width": 0.10520833333333333,
        "height": 0.4824074074074074
      }
    },
    {
      "label": "person",
      "confidence": 84.00863647460938,
      "color": 16711935,
      "rectangle": {
        "x": 0.371875,
        "y": 0.33055555555555555,
        "width": 0.0703125,
        "height": 0.45555555555555555
      }
    }
  ],
  "buffer_base64": "/9j/4AAQSkZJRgABAQAAAQABAA...",
  "parameters": {
    "timestamp": "3403400000"
  }
}
```

### Module Parameters

<ResponseField name="attach-frame" type="Boolean">
  When enabled will map the input buffer and perform Base64 encoding of the data so it can be embedded alongside the metadata. Only valid for JPEG inputs<br />

  Flags: *readable, writable*<br />
  Default value: `false`
</ResponseField>

# Processing Flow

**Caps Negotiation** <br />
During pipeline setup, qtimlmetaparser negotiates supported sink and source caps with neighboring elements. The sink pad accepts multiple input types, including raw video, JPEG, and UTF-8 text, while the source pad always produces UTF-8 text.

**Property Parsing and Configuration** <br />
After caps negotiation, the element selects the parser module specified by the module property, loads it dynamically, and creates a module instance for the stream.

The element determines the input type from the sink caps, extracts width and height when applicable, and combines this information with the user-provided module-params. The resulting configuration is passed to the module before buffer processing begins.

**Allocation and Buffer Preparation** <br />
For raw video input, the element requests video meta support during allocation negotiation to preserve metadata across the pipeline.

For each input buffer, the element allocates a corresponding output buffer for the generated text. Input timestamps are preserved. If the input buffer is marked as GAP, the element propagates the GAP condition downstream.

**Module Execution** <br />
For each buffer, the element invokes the selected module to translate the attached ML metadata into UTF-8 text.

When the json module is used, it serializes supported metadata into a JSON document. Depending on configuration, it may also embed Base64-encoded JPEG frame data and additional parameters such as timestamps.

The serialized UTF-8 output is then written to the output buffer.

**Output** <br />
The element pushes the output buffer downstream on the source pad as text/x-raw.

# Usage

This example demonstrates a basic live-camera inference pipeline using a YOLO detection model. Video frames from the ISP camera are processed by the inference branch, and the resulting AI metadata is attached to the corresponding video buffers using the qtimetamux element. The attached metadata is then parsed by qtimlmetaparser, serialized into JSON format, and written to a file for storage or further processing.

<img alt="qtimetamux_workflow" />

## Main Buffer, Metadata Synchronization and Latency control

The plugin is designed with a **single main sink pad** that receives the primary video or audio
buffers, and **multiple auxiliary data pads** that collect ML post-processing results or CV motion
vectors. Data arriving on auxiliary pads may be provided in string or blob form and is parsed into
structured representations. Once parsed, the plugin matches each data buffer to its corresponding
main media frame and attaches the result as **GstMeta**.

### Async Mode

This is the default synchronization mode. No timestamp-based matching is performed. Instead,
metadata buffers are associated with main frames in strict **1:1 order**:

* The N-th incoming video/audio frame is held until the N-th data buffer has been received on **all** data pads.
* Once all required data for that frame is available, the metadata is attached.
* The enriched buffer is then pushed downstream.

This mode is suitable when media buffers and metadata buffers are produced in a fixed, predictable sequence.

### Sync Mode

In sync mode, the plugin performs **timestamp-based synchronization**. Each incoming main frame is
held for a limited time window of up to `1 / framerate` seconds (video) or `1 / rate` seconds
(audio). For example, at 30 fps, the frame may be held for approximately **33.3 ms**.

During this hold period, the plugin waits for data buffers on its auxiliary pads whose timestamps
match the timestamp of the main frame:

* If all expected data buffers arrive within the time window, they are attached before forwarding.
* If one or more auxiliary pads do not provide matching buffers in time, only the successfully
  matched metadata is attached and the main buffer is released downstream.

### Latency Control

In some use cases, the default hold period in sync mode may be too short — especially when metadata
generation takes longer than expected. The `latency` property extends the waiting period by
accepting an integer value in **nanoseconds**, allowing the plugin to wait longer for late-arriving
data buffers before forwarding the main frame.

<img alt="qtimetamux_latency_control" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLO model | [Qualcomm Yolo Model](https://aihub.qualcomm.com/iot/models/yolox) | `yolov8_det_w8a8.tflite` |
    | YOLO labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
      scp yolov8_det_w8a8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/models $HOME/Downloads/qimsdk_samples/labels $HOME/Downloads/qimsdk_samples/media
    export MODEL_NAME=yolov8_det_w8a8.tflite
    export LABELS_NAME=yolov8.json
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qticamsrc ! video/x-raw,width=1920,height=1080 ! queue ! \
    tee name=t ! queue ! \
    qtimetamux name=metamux ! queue ! \
    qtimlmetaparser module=json ! filesink location=$HOME/Downloads/qimsdk_samples/media/meta.json \
    t. ! queue ! \
    qtimlvconverter ! queue ! \
    qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME ! queue ! \
    qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! text/x-raw ! queue ! \
    metamux.
    ```
  </Step>
</Steps>


# qtimldemux
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtimldemux

GStreamer plugin for demultiplexing batched AI inference outputs in Vision AI SDK.

# Overview

`qtimldemux` is a GStreamer plugin designed for **batch-oriented AI inference pipelines**, where multiple independent inputs are processed together in a single inference execution. Its primary role is to **demultiplex batched output tensors** and restore them as **per-input results**, so downstream elements can continue processing each input independently. This is essential in multi-stream and batched AI workflows, where batched execution improves hardware utilization, but downstream stages must continue to operate on a per-stream or per-sample basis.

`qtimldemux` commonly operates in conjunction with [`qtibatch`](/plugin-reference/qtibatch). In such pipelines, [`qtibatch`](/plugin-reference/qtibatch) performs input aggregation before inference, and `qtimldemux` performs output demultiplexing after inference. This pairing allows pipelines to benefit from batched model execution without losing per-input result alignment.

* [`qtibatch`](/plugin-reference/qtibatch) aggregates multiple input streams or buffers into a single batched input
* `qtimldemux` splits the resulting batched output back into per-input tensors or results

Together, these elements enable efficient batched inference while preserving the association between each inference result and its originating input.

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />qtimldemux

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `neural-network/tensors` | `format: { INT8, UINT8, INT32, UINT32, FLOAT16, FLOAT32 }` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `neural-network/tensors` | `format: { INT8, UINT8, INT32, UINT32, FLOAT16, FLOAT32 }` |
| Availability: *On request* | |
| Direction: *source* | |

## Why Batch Inference Requires Output Demultiplexing

Many machine learning models are designed to process multiple inputs in a single inference execution. This execution model is commonly referred to as **batch inference**. In batch-based models, the input tensor includes an explicit **batch dimension**, allowing the model to process a fixed number of independent inputs together rather than one input at a time.

Batch inference is widely used because it:

* improves hardware utilization
* reduces per-input inference overhead
* increases accelerator efficiency through better scheduling
* matches the fixed input shape requirements of many deployed models

For these models, the batch size is typically defined by the model itself. It is not treated as a dynamic runtime parameter. As a result, the runtime must provide input data in a form that exactly matches the model’s expected batch shape.

### Constructing Batched Input

Before inference can be executed, multiple independent inputs must be collected and combined into a single batched input. In a streaming pipeline, this usually involves:

* receiving data from multiple logical input sources, such as separate streams or sensors
* selecting one input unit from each source, such as a video frame or audio buffer
* assembling those inputs into a single batched representation that matches the model input shape

The inputs grouped into a batch do not need to originate from a single source. They may come from different streams and may arrive at slightly different times. As a result, batch construction is a pipeline-level operation that groups multiple logical inputs into one inference unit.

### Batched Output Representation

When inference is executed on a batched input, the model produces **batched output tensors**. These output tensors contain the inference results for all inputs in the batch, organized according to the same batch structure used at the input.

At this stage:

* the inference results for all inputs are grouped into a single output
* each result is identified only by its position in the batch
* the original stream-level or input-level separation is no longer explicit

This output form is efficient for model execution, but it is not ideal for most downstream pipeline stages.

### Why Demultiplexing Is Needed

Most downstream elements do not operate on batched results. Post-processing, metadata generation, visualization, tracking, and application logic typically expect results on a **per-input** basis. These stages usually require:

* inference output corresponding to a single logical input
* correct association between each result and its originating stream or sample
* independent downstream processing for each input

Because batched output does not preserve this separation in a directly consumable form, it must be split back into individual per-input results before further processing.

### Role of Demultiplexing

The demultiplexing stage restores the logical separation that existed before batch inference. It:

* extracts the result corresponding to each batch element
* re-establishes the mapping between inference results and their original inputs
* allows downstream elements to continue operating in a per-stream or per-sample manner

This step is essential in batch-based inference pipelines whenever downstream processing is not designed to operate on batched tensors directly.

## Usage

### Multi stream batched mode AI Inference

This example demonstrates a four-stream batched inference pipeline using video files as input. Four file sources feed the same video into [`qtibatch`](/plugin-reference/qtibatch), which aggregates frames into a single batched input. [`qtimlvconverter`](/plugin-reference/qtimlvconverter) prepares the batched tensors, and [`qtimltflite`](/plugin-reference/qtimltflite) performs batched inference. `qtimldemux` then restores per-stream outputs so that post-processing can run independently on each stream. The resulting metadata is combined with the corresponding video streams by [`qtivcomposer`](/plugin-reference/qtivcomposer), and the final 2×2 composition is displayed using [`waylandsink`](/plugin-reference/waylandsink).

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Yolov8 Detection W8A8 Batch 4 model | [Export from Qualcomm AI Hub](https://github.com/qualcomm/ai-hub-models/blob/v0.55.0/src/qai_hub_models/models/yolov8_det/README.md) | `yolov8_det_w8a8_batch_4.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
      scp yolov8_det_quantized_batch_4.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                           <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4             <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolov8_det_w8a8_batch_4.tflite
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qtimltflite name=inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,htp_performance_mode=(string)2;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME \
    qtibatch name=batch ! queue ! qtimlvconverter ! queue ! inference. inference. ! queue ! qtimldemux name=mldemux_1 \
    qtivcomposer name=mixer \
    sink_0::position="<0, 0>" sink_0::dimensions="<960, 540>" \
    sink_1::position="<960,  0>" sink_1::dimensions="<960, 540>" \
    sink_2::position="<0, 540>" sink_2::dimensions="<960, 540>" \
    sink_3::position="<960, 540>" sink_3::dimensions="<960, 540>" \
    sink_4::position="<0, 0>" sink_4::dimensions="<960, 540>" \
    sink_5::position="<960, 0>" sink_5::dimensions="<960, 540>" \
    sink_6::position="<0, 540>" sink_6::dimensions="<960, 540>" \
    sink_7::position="<960, 540>" sink_7::dimensions="<960, 540>" \
    mixer. ! video/x-raw,format=NV12 ! queue ! waylandsink sync=true fullscreen=true \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! tee name=split_1 ! queue ! batch. split_1. ! queue ! mixer. \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! tee name=split_2 ! queue ! batch. split_2. ! queue ! mixer. \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! tee name=split_3 ! queue ! batch. split_3. ! queue ! mixer. \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! tee name=split_4 ! queue ! batch. split_4. ! queue ! mixer. \
    mldemux_1. ! queue ! qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! video/x-raw ! queue ! mixer. \
    mldemux_1. ! queue ! qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! video/x-raw ! queue ! mixer. \
    mldemux_1. ! queue ! qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! video/x-raw ! queue ! mixer. \
    mldemux_1. ! queue ! qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! video/x-raw ! queue ! mixer.
    ```
  </Step>
</Steps>


# qtimlonnx
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtimlonnx

ONNX Runtime inference element for GStreamer AI and multimedia pipelines.

<Note>
  **qtimlonnx** is not available in the current build. Support for this plugin will be enabled in a future release.
</Note>

# Overview

**qtimlonnx** is a GStreamer inference element that executes **ONNX** models as part of AI and multimedia pipelines. The element operates entirely in **tensor mode:** it accepts input tensors on its sink pad and produces output tensors on its source pad according to the model's input and output specifications.

The element is limited to **model execution**. It does not perform preprocessing, tensor reshaping, batching, layout conversion, or model-specific post-processing. These functions are expected to be handled by adjacent elements in the pipeline. As a result, upstream elements must provide tensors that already match the model requirements, and downstream elements must interpret the output tensors produced by inference.

**qtimlonnx** supports multiple ONNX Runtime execution providers, including **CPU, GPU, and Qualcomm AI accelerator / NPU** execution through the QNN runtime. This allows the same pipeline structure to be deployed across different hardware targets and optimized for different performance, latency, and power requirements. The element is intended for real-time and embedded AI pipelines where inference is one stage in a larger modular processing flow.

### Key Responsibilities

**qtimlonnx** is responsible for:

* Loading and executing an ONNX model via the ONNX Runtime
* Accepting preformatted input tensors from upstream elements
* Producing output tensors that match the model output signature
* Negotiating tensor data types and dimensions with adjacent elements
* Propagating tensor metadata required by downstream elements
* Automatically extracting quantization parameters (scale and zero-point) from the ONNX model graph and dequantizing quantized outputs to `FLOAT32` when required
* Automatically detecting NCHW or NHWC memory layout for 4-D output tensors and advertising the layout in output caps

In practice, **qtimlonnx** serves as the inference stage in the pipeline, while tensor preparation and result interpretation are handled externally.

## Example Pipeline

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.onnx` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp yolox_w8a8.onnx          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolox_w8a8.onnx
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
    t. ! queue ! qtimlvconverter ! queue ! qtimlonnx model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME execution-provider=qnn backend-path="/usr/lib/libQnnHtp.so" ! queue ! \
    qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! obj_mux.
    ```
  </Step>
</Steps>

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />[GstBaseTransform](https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c)<br />
            <Icon icon="arrow-turn-down-right" />qtimlonnx

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `neural-network/tensors` | `format: { INT8, UINT8, INT16, UINT16, INT32, UINT32, INT64, UINT64, FLOAT16, FLOAT32 }` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `neural-network/tensors` | `format: { INT8, UINT8, INT16, UINT16, INT32, UINT32, INT64, UINT64, FLOAT16, FLOAT32 }` |
| Availability: *Always* | |
| Direction: *source* | |

# Element Properties

| Property | Description |
| - | - |
| `backend-path` | Absolute path to the QNN backend shared library such as `libQnnHtp.so`. Used only when `execution-provider=qnn`. Determines which Qualcomm hardware accelerator is targeted.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable (construct)` |
| `execution-provider` | Selects the ONNX Runtime Execution Provider (EP) used for inference.<br /><br />`Type: Enum `<br />`Default: 0, "cpu"`<br />`Range:`<br />    `(0): cpu - Default ONNX Runtime CPU execution. Runs all operations on the host CPU`<br />    `(1): qnn - Qualcomm QNN Execution Provider. Offloads inference to Qualcomm hardware via the QNN SDK. Requires backend-path to be set`<br />`Flags: readable/writable` |
| `htp-performance-mode` | Controls the power and performance trade-off on the Qualcomm Hexagon HTP. Applicable only when `execution-provider=qnn`.<br /><br />`Type: Enum `<br />`Default: 0, "default"`<br />`Range:`<br />    `(0): default - Default performance mode`<br />    `(1): burst - Maximum performance with highest power consumption`<br />    `(2): balanced - Balanced performance and power`<br />    `(3): low-balanced - Lower balanced performance`<br />    `(4): high-performance - High performance mode`<br />    `(5): extreme-power - Extreme power performance`<br />    `(6): low-power - Lowest power consumption`<br />    `(7): sustained-high-performance - Sustained high performance without throttling`<br />`Flags: readable/writable` |
| `model` | Path to the ONNX model file. This property is required and must reference a valid `.onnx` model file. The model is loaded when the element transitions from `NULL` to `READY` state.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable (construct)` |
| `optimization-level` | Controls the ONNX Runtime graph optimization level applied when loading the model. Higher levels may reduce inference latency but increase model load time.<br /><br />`Type: Enum `<br />`Default: 2, "enable-extended"`<br />`Range:`<br />    `(0): disable-all - Disable all graph optimizations`<br />    `(1): enable-basic - Basic optimizations such as constant folding`<br />    `(2): enable-extended - Extended optimizations including operator fusion`<br />    `(3): enable-all - All optimizations including layout and memory optimizations`<br />`Flags: readable/writable` |
| `threads` | Number of intra-operation threads assigned to the ONNX Runtime session. Primarily affects CPU execution. May have limited impact when using QNN execution provider.<br /><br />`Type: Unsigned Integer`<br />`Default: 1`<br />`Range: 1 - 16`<br />`Flags: readable/writable (construct)` |

# Input and Output Behavior

### Input Tensors

**qtimlonnx** exposes a single sink pad, but it supports both single-input and multi-input models. For multi-input models, all required tensors are delivered through the same sink pad as a tensor set.

Input tensors must be fully prepared before they reach **qtimlonnx**. Expected tensor layout, shape, data type, and batch size are determined by:

* the ONNX model input signature (read at engine initialization)
* caps negotiation with upstream elements

Typical upstream elements include:

* [`qtimlvconverter`](/plugin-reference/qtimlvconverter) for scaling, color conversion, normalization, and quantization
* [`qtibatch`](/plugin-reference/qtibatch) for batch construction

**qtimlonnx** does not modify, reshape, batch, or reinterpret incoming tensors. It wraps them directly as ONNX Runtime input tensors (zero-copy) and passes them to the runtime as received.

### Output Tensors

**qtimlonnx** exposes a single source pad and produces output tensors according to the model output signature. The single source pad does not limit the element to a single tensor. Models with multiple output tensors are fully supported, and all outputs are emitted together on the same pad.

The element supports:

* single-output and multi-output models
* arbitrary tensor ranks, including batch and depth dimensions
* quantized and floating-point outputs

Output tensors are typically consumed by downstream post-processing elements, which decode model-specific results such as classification scores, detection boxes, segmentation masks, landmarks, or other structured outputs.

# Quantization and Dequantization

**qtimlonnx** can optionally dequantize quantized output tensors (such as `UINT8` or `INT8`) into `FLOAT32`. This conversion uses quantization parameters (scale and zero-point) extracted directly from the ONNX model graph at engine initialization time.

### Quantization Parameter Extraction

At initialization, **qtimlonnx** parses the ONNX model graph and locates `QuantizeLinear` nodes whose outputs match the model's declared output tensor names. For each such node, the plugin reads:

* `scale` — from the second input of the `QuantizeLinear` node (a graph initializer)
* `zero-point` — from the third input of the `QuantizeLinear` node (a graph initializer)

These values are stored per output tensor and used during inference to perform dequantization.

### Conditional Output Dequantization

Dequantization is performed only when the downstream path requires `FLOAT32` tensors. In practice, this is enabled when downstream caps negotiation indicates that floating-point output is needed.

When dequantization is applied, **qtimlonnx**:

* reads the per-tensor `scale` and `zero_point` extracted from the model graph
* applies the standard dequantization formula:

```text theme={null}
output_float = (quantized_value - zero_point) × scale
```

* produces `FLOAT32` tensors for downstream processing

### When Dequantization Is Skipped

Dequantization is not performed when:

* downstream elements accept only quantized tensor types
* no downstream element negotiates `FLOAT32`
* the model output tensor does not contain a `QuantizeLinear` node with valid quantization metadata

In these cases, the output tensor is forwarded in its original quantized representation.

This behavior allows the same downstream processing path to support both quantized and floating-point models where applicable, while avoiding unnecessary conversion.

# Supported Data Types

**qtimlonnx** supports the tensor data types provided by the ONNX Runtime and the selected execution provider, subject to caps negotiation with adjacent elements.

Supported data types include:

* `INT8`
* `UINT8`
* `INT16`
* `UINT16`
* `INT32`
* `UINT32`
* `INT64`
* `UINT64`
* `FLOAT16`
* `FLOAT32`

The element does not impose additional data-type restrictions beyond those required by the ONNX Runtime, the selected execution provider, and the negotiated pipeline caps.

# NCHW / NHWC Layout Detection

For 4-D output tensors, **qtimlonnx** automatically detects the memory layout (NCHW or NHWC) at engine initialization by traversing the ONNX model graph backwards from each output tensor and inspecting `Transpose` node permutations:

* A `Transpose` with `perm=[0,2,3,1]` indicates the output is **NHWC** (the node converted NCHW→NHWC).
* A `Transpose` with `perm=[0,3,1,2]` indicates the output is **NCHW** (the node converted NHWC→NCHW).
* If no `Transpose` node is found, the output defaults to **NCHW** (the ONNX standard).

When any output tensor is detected as NCHW, the plugin adds `layout=nchw` to the source pad caps. Downstream elements (such as [`qtimlpostprocess`](/plugin-reference/qtimlpostprocess)) use this field to perform the necessary in-place NCHW→NHWC transpose before processing.

# Batch and Depth Model Support

**qtimlonnx** supports models with batch and multi-dimensional tensor inputs and outputs, including tensors with explicit batch and depth dimensions.

Examples include:

* batched tensors: `N × H × W × C`
* multi-dimensional tensors: `N × D × H × W × C`

The element treats these dimensions transparently and passes tensors to the ONNX Runtime according to the negotiated shape. It does not construct batches, reshape tensors, or reinterpret tensor dimensions internally.

Batch construction must be handled by upstream elements such as [`qtibatch`](/plugin-reference/qtibatch).

This behavior keeps inference predictable across single-frame, batched, and higher-dimensional workflows.

# Execution Providers

An ONNX Runtime **Execution Provider (EP)** defines the hardware backend used to run a model. Execution providers allow **qtimlonnx** to offload inference from the default CPU interpreter to an optimized backend such as Qualcomm's HTP/DSP.

**qtimlonnx** supports two execution providers. The provider is selected through the `execution-provider` property and controls how the ONNX Runtime dispatches model operations during inference.

### CPU

Runs the model on the default ONNX Runtime CPU interpreter.

* **Backend**: CPU
* **Use case**: reference execution, debugging, or systems without hardware acceleration
* **Additional configuration**: none required

### QNN (Qualcomm Neural Network)

Offloads inference to the AI Accelerator / NPU via the ONNX Runtime's QNN Execution Provider.

* **Backend**: Qualcomm AI Accelerator / NPU
* **Use case**: hardware-accelerated inference on Qualcomm SoCs for quantized and floating-point models
* **Additional configuration required**:
  * `backend-path` — absolute path to the QNN backend shared library (e.g. `libQnnHtp.so`)
  * `htp-performance-mode` — optional power/performance trade-off setting

When `execution-provider=qnn`, the plugin performs the following initialization sequence:

1. Registers `libonnxruntime_providers_qnn_abi.so` with the ONNX Runtime environment using `RegisterExecutionProviderLibrary`.
2. Enumerates available QNN EP devices using `GetEpDevices`.
3. Attaches the QNN EP to the session options using `SessionOptionsAppendExecutionProvider_V2`, passing `backend_path` and `htp_performance_mode` as provider options.
4. Creates the ONNX Runtime session with the configured QNN EP.

# Runtime Memory Behavior and GAP Handling

**qtimlonnx** operates within the memory model of the ONNX Runtime. Input buffers from the pipeline are mapped read-only and wrapped directly as ONNX Runtime input tensors using `CreateTensorWithDataAsOrtValue`, avoiding a copy of the input data. Output tensors are written into DMA-backed output buffers allocated from the element's `GstMLBufferPool`.

### ONNX Runtime Memory Model

The ONNX Runtime manages its own internal memory for:

* intermediate activation tensors
* output tensors (allocated by the runtime and then copied into the pipeline output buffer)

### GAP Buffer Handling

**qtimlonnx** is GAP-aware and correctly handles input buffers marked with `GST_BUFFER_FLAG_GAP`.

When a GAP buffer is received, the element skips inference and forwards the buffer downstream. This preserves timing and synchronization while explicitly indicating that no valid inference input is available for that timestamp.

GAP buffers commonly appear in conditional AI pipelines, such as cascaded workflows where later inference stages run only when earlier stages produce valid regions of interest.

# Usage

### Single-Stage AI Inference on Live Camera Stream

This example demonstrates real-time ONNX inference on a live camera stream using a single **qtimlonnx** instance with the QNN execution provider. Inference results are attached to each `GstBuffer` as `MLMeta`, allowing downstream elements to access synchronized metadata directly from the frame. An overlay stage renders annotations such as bounding boxes, labels, or keypoints before display.

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.onnx` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"
      scp yolox_w8a8.onnx            <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json   <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolox_w8a8.onnx
    export LABELS_NAME=yolov8.json
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash Run the pipeline theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qticamsrc ! video/x-raw,width=1920,height=1080,format=NV12,framerate=30/1 ! queue ! tee name=t \
    t. ! queue ! qtimetamux name=metamux ! queue ! qtivoverlay ! waylandsink fullscreen=true sync=true \
    t. ! queue ! qtimlvconverter ! queue ! qtimlonnx model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME execution-provider=qnn backend-path="libQnnHtp.so" ! queue ! \
    qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw ! queue ! metamux.
    ```
  </Step>
</Steps>

### Two-Stage Daisy Chain AI Inference on File Stream

This example demonstrates a two-stage ONNX inference workflow using two **qtimlonnx** instances. The first model operates on full video frames after preprocessing by a [`qtimlvconverter`](/plugin-reference/qtimlvconverter) configured for full-frame input. Inference results, such as detected objects, are attached to the corresponding video buffer and propagated downstream. The second model runs once for each object detected by the first stage. A second [`qtimlvconverter`](/plugin-reference/qtimlvconverter), configured for ROI-based processing, crops each detected region from the input frame and prepares it as input for the second **qtimlonnx** instance.

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Detection model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.onnx` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Classification model (InceptionV3) | [Qualcomm AI Hub — InceptionV3](https://aihub.qualcomm.com/iot/models/inception_v3) | `mobilenet_v2_w8a8.onnx` |
    | Classification labels | <a href="../labels/mobilenet.json">mobilenet.json</a> | `mobilenet.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
      scp <yolox_w8a8.onnx>      <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json               <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp <mobilenet_v2_w8a8.onnx>      <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp mobilenet.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME_1=yolox_w8a8.onnx
    export LABELS_NAME_1=yolov8.json
    export MODEL_NAME_2=<mobilenet_v2_w8a8.onnx>
    export LABELS_NAME_2=mobilenet.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash Run the pipeline theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qtimlvconverter name=stage_01_preproc \
    qtimlonnx model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_1 execution-provider=qnn backend-path="libQnnHtp.so" htp-performance-mode=1 name=stage_01_inference \
    qtimlpostprocess name=stage_01_postproc results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_1 \
    qtimlvconverter name=stage_02_preproc \
    qtimlonnx model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_2 execution-provider=qnn backend-path="libQnnHtp.so" htp-performance-mode=1 name=stage_02_inference \
    qtimlpostprocess name=stage_02_postproc module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! tee name=t_split_1 \
    t_split_1. ! queue ! metamux_1. \
    t_split_1. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! stage_01_inference. stage_01_inference. ! queue ! stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! metamux_1. \
    qtimetamux name=metamux_1 ! queue ! qtiobjtracker algo=bytetrack ! queue ! tee name=t_split_2 \
    t_split_2. ! queue ! metamux_2. \
    t_split_2. ! queue ! stage_02_preproc. stage_02_preproc. ! queue ! stage_02_inference. stage_02_inference. ! queue ! stage_02_postproc. stage_02_postproc. ! text/x-raw ! queue ! metamux_2. \
    qtimetamux name=metamux_2 ! queue ! qtivoverlay ! queue ! waylandsink fullscreen=true
    ```
  </Step>
</Steps>


# qtimlpostprocess
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtimlpostprocess

AI post-processing plugin

# Overview

**Output tensors** produced by inference models typically require **post-processing** to make the results usable for downstream components or interpretable by applications. For example:

* **Image classification** outputs are arrays of confidence scores that need interpretation, such as selecting the top classes exceeding a specified threshold.
* **Object detection** outputs should be converted into a set of bounding boxes with associated labels.
* **Pose estimation** outputs should be transformed into a set of keypoints and connections between them.
* **Image segmentation** outputs should be converted into RGBA image masks that can be overlaid on the original frame.
* **Raw tensor** data may require conversion into formats expected by subsequent plugins or processing stages.

Within the QIM SDK, the **qtimlpostprocess** element manages post-processing tasks. This plugin converts raw model outputs into **GStreamer ML metadata.** It is a customizable plugin that provides a library interface for post-processing the tensor output of inference plugins. The **post-processing library** is solely responsible for tensor parsing and outputs a list of predictions. We refer to this as the post-processing (PP) module.

<img alt="Postprocess diagram" />

The **qtimlpostprocess** element receives a list of tensors as input, each encapsulated in a GST Buffer. Metadata describing the tensors—such as the number of tensors, their shapes, timestamps, batching indexes, and more—is attached as GStreamer metadata.

The post-processing plugin configuration ( GStreamer properties):

***

## Example Pipeline

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp yolox_w8a8.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/models $HOME/Downloads/qimsdk_samples/labels $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/media/output
    export MODEL_NAME=yolox_w8a8.tflite
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
    t. ! queue ! qtimlvconverter ! queue ! \
    qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
      external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
    qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME \
      settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! obj_mux.
    ```
  </Step>
</Steps>

# Element Properties

| Property | Description |
| - | - |
| `module` | Post-processing module name. This mandatory property specifies how the tensor will be parsed. It does not define the plugin output type, which is determined during pipeline caps negotiation.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `labels` | Path to the label file. The file is passed directly to the module without interpretation by the plugin. Supports JSON and newline-separated formats.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `results` | Limits the number of output results. If the detected results exceed this value, lower-confidence results are discarded automatically by the plugin.<br /><br />`Type: Integer`<br />`Default: 5`<br />`Range: 0 - 50`<br />`Flags: readable/writable` |
| `settings` | JSON string or path to a JSON file containing module-specific configuration such as confidence thresholds, keypoints, or other parameters.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `bbox-stabilization` | Enable stabilization of bounding boxes (bboxes) to reduce jitter across frames.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |

### Label File Examples

#### Newline-separated format

```
tench
goldfish
great white shark
tiger shark
hammerhead
electric ray
stingray
cock
hen
ostrich
```

#### JSON format

```json theme={null}
[
  {"id": 3, "color": "0x00FF00FF", "label": "tiger shark"},
  {"id": 4, "color": "0x00FF00FF", "label": "hammerhead"},
  {"id": 5, "color": "0x00FF00FF", "label": "electric ray"},
  {"id": 6, "color": "0x00FF00FF", "label": "stingray"},
  {"id": 17, "color": "0x00FF00FF", "label": "jay"}
]
```

### Settings Example (Pose Estimation)

```json theme={null}
{
  "confidence": 51.0,
  "connections": [
    {"id": 5, "connection": 6},
    {"id": 6, "connection": 12},
    {"id": 7, "connection": 5},
    {"id": 8, "connection": 6}
  ]
}
```

The output of the **qtimlpostprocess** element can be in one of the following formats:

* **text** - The post-processing plugin serializes ML metadata to text. This metadata can either be used as-is by other plugins or attached to the source stream using qtimetamuxer.
* **image mask** - The post-processing plugin can generate an image mask with overlaid text, bounding boxes, dots, lines, and other visual elements. This is a transparent frame that contains only ML results. For example, if the post-processing type is object detection, the plugin will draw bounding boxes with labels. The image mask can then be blitted onto the source video stream using the qtivcomposer plugin.
* **tensor** - The post-processing plugin can generate tensors. This is useful when the output tensor from one inference stage needs to be passed to the next stage, but the tensor shapes don't match exactly. For example, the first stage might produce four output tensors, while the next stage might require only three of them.

Output format is selected during GStreamer pipeline caps negotiation. Most settable formats are usually negotiated automatically but developers can specify them manually via GStreamer caps filter. The plugin supports only one source pad. If the pipeline requires two or more of the supported formats simultaneously, then the post-processing plugin should be added and executed twice within the pipeline.

# Post Processing Callback

The `process-*` signals enable **external post-processing**: instead of loading a built-in module, the application parses the inference output tensors itself. This path is engaged only when **both** conditions are met:

1. The `module` property is **not** set, so no built-in module claims the tensors, and
2. A handler is connected to one of the `process-*` signals.

The element selects its processing type from whichever signal you connect, so connect **only** the signal that matches your model.

<Warning>If the `module` property is set, the built-in module performs the parsing and the connected handler is **not** invoked.</Warning>

All `process-*` signals share the same callback layout, differing only in the type of the result container passed as the third parameter:

| Parameter | Type | Description |
| - | - | - |
| *(instance)* | `qtimlpostprocess` | The element emitting the signal. |
| `mlframe` | `GstMLFrame` | Input inference output tensors. |
| `mlparam` | `GstStructure` | Per-batch parameters describing how the input was prepared. |
| *(result)* | *(see table below)* | The result container to populate (or output `GstMLFrame` for `process-tensors`). |
| *(return)* | `gboolean` | `TRUE` on success, `FALSE` to abort the pipeline. |

The available signals, the result container passed as the third argument, and the entry type you append to it:

| Signal | Result container (3rd argument) | Entry type to append (C / Python) |
| - | - | - |
| `process-image-classification` | `GstMLClassifications` | `GstMLClassification` / `GstQtiML.Classification` |
| `process-audio-classification` | `GstMLClassifications` | `GstMLClassification` / `GstQtiML.Classification` |
| `process-object-detection` | `GstMLDetections` | `GstMLDetection` / `GstQtiML.Detection` |
| `process-pose-estimation` | `GstMLPoses` | `GstMLPose` / `GstQtiML.Pose` |
| `process-segmentation` | `GstMLSegmentations` | `GstMLSegmentation` / `GstQtiML.Segmentation` |
| `process-depth-estimation` | `GstMLDepthMaps` | `GstMLDepthMap` / `GstQtiML.DepthMap` |
| `process-tensors` | `GstMLFrame` / `GstQtiML.Frame` | — (write output tensors directly; no entries) |

`Flags: Run Last`

<Info>For the result-structure field reference and C/C++ and Python examples, see [Guidelines for Writing an External Post-Processing Callback](#guidelines-for-writing-an-external-post-processing-callback) below.</Info>

# Post Processing Module

The **post processing module** is solely responsible for tensor parsing and outputs a list of predictions. Each post-processing module implements parsing logic tailored to a specific class of models. For example, a single module is responsible for all variants of YoloV8 detection models. The plugin manages the execution of the module, the generation of outputs (ML metadata or image mask), batching, chained AI models, and other related tasks.

The post processing module currently supports the following output data types:

* object-detection
* image-classification
* image-segmentation
* depth-estimation
* super-resolution
* pose-estimation
* audio-classification
* tensor

The post-processing module is a loadable entity. The QIM SDK provides a comprehensive set of post-processing modules, but application developers can also write their own custom modules and deploy them on the device. Each module is built as a shared library.

All post-processing module shared libraries must be deployed in a dedicated folder on the target device, typically:

```
/usr/lib/gstreamer-1.0/ml/modules
```

The **qtimlpostprocess** element automatically detects supported post-processing modules when they are deployed in this location.

Below you can find list of currently supported AI modules.

<AccordionGroup>
  <Accordion title="Image Classification">
    * `mobilenet-softmax`
    * `mobilenet`
    * `ocr-recognizer`
    * `ocr`
    * `qfr-softmax`
    * `qfr`
  </Accordion>

  <Accordion title="Object Detection">
    * `easy-textdt`
    * `easy-ocr-detector`
    * `mediapipe-pose`
    * `qfd`
    * `qpd`
    * `ssd-mobilenet`
    * `yolo-nas`
    * `yolov5`
    * `yolov8`
    * `palmd`
  </Accordion>

  <Accordion title="Semantic Segmentation">
    * `deeplab-argmax`
    * `yolov8-seg`
  </Accordion>

  <Accordion title="Depth Estimation">
    * `midas-v2`
  </Accordion>

  <Accordion title="Pose Estimation">
    * `hrnet`
    * `lite-3dmm`
    * `posenet`
    * `hlandmark`
    * `mediapipe-pose-landmark`
  </Accordion>

  <Accordion title="Super Resolution">
    * `srnet`
  </Accordion>

  <Accordion title="Audio Classification">
    * `wave2vec`
    * `yamnet`
  </Accordion>

  <Accordion title="Tensor Generation">
    * `tensor`
  </Accordion>
</AccordionGroup>

The supported models for all the above categories can be found in the [Supported Models](/supportedmodels/) section.

**Important:** It is very common for one post-process module to support more than one ML model. For example:

* yolov8 module can be used for both YoloV8 and YoloX models, because both Yolo models have the same output and need the same post processing implementation. The same applies to YoloV3 and YoloV5.
* mobilenet module can be used for MobileNet, ResNet and other Image Classification ML models because classification post processing is very common across ML models.

A list of supported post-processing modules, along with their supported input tensor shapes and data types, can be checked directly on the device.
To view the full list of supported modules, use the following command:

```
gst-inspect-1.0 qtimlpostprocess
```

This information is updated immediately when a new post-processing module is deployed or removed.

Example output:

```
module              : Module name that is going to be used for processing the tensors
                      flags: readable, writable
                      Enum "GstMLPostProcessModules" Default: 0, "none"
                         (0): none             - No module, default invalid mode
                         (1): ssd-mobilenet    -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 10, 4
                                Tensor 1: 1, 10
                                Tensor 2: 1, 10
                                Tensor 3: 1

                         (2): hrnet            -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 1-256, 1-256, 1-17

                         (3): srnet            -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 32-4096, 32-4096
                                Type: FLOAT32
                                Tensor 0: 1, 32-4096, 32-4096, 1-3

                         (4): yolov8-seg       -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 21-42840, 4
                                Tensor 1: 1, 21-42840
                                Tensor 2: 1, 21-42840, 1-32
                                Tensor 3: 1, 21-42840
                                Tensor 4: 1, 1-32, 32-2048, 32-2048

                         (5): posenet          -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 5-251, 5-251, 1-17
                                Tensor 1: 1, 5-251, 5-251, 2-34
                                Tensor 2: 1, 5-251, 5-251, 4-64

                         (6): east-textdt      -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 8-480, 8-480, 1-5
                                Tensor 1: 1, 8-480, 8-480, 1-5

                         (7): qfr              -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 512
                                Tensor 1: 1, 32
                                Tensor 2: 1, 2
                                Tensor 3: 1, 2
                                Tensor 4: 1, 2
                                Tensor 5: 1, 2

                         (8): deeplab-argmax   -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 32-2048, 32-2048
                                Type: FLOAT32
                                Tensor 0: 1, 32-2048, 32-2048, 1-150

                         (9): yolov8           -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 21-42840, 4
                                Tensor 1: 1, 21-42840
                                Tensor 2: 1, 21-42840
                                Type: FLOAT32
                                Tensor 0: 1, 4, 21-42840
                                Tensor 1: 1, 1-1001, 21-42840
                                Type: FLOAT32
                                Tensor 0: 1, 5-1005, 21-42840

                         (10): mobilenet        -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 1000-1001

                         (11): lite-3dmm        -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 512
                                Tensor 1: 1, 265
                                Type: FLOAT32
                                Tensor 0: 1, 265

                         (12): ocr              -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 26, 1, 37
                                Type: FLOAT32
                                Tensor 0: 1, 26-48, 37

                         (13): yolov5           -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 1-136, 1-136, 18-3018
                                Tensor 1: 1, 1-136, 1-136, 18-3018
                                Tensor 2: 1, 1-136, 1-136, 18-3018
                                Type: FLOAT32
                                Tensor 0: 1, 3, 1-136, 1-136, 6-85
                                Tensor 1: 1, 3, 1-136, 1-136, 6-85
                                Tensor 2: 1, 3, 1-136, 1-136, 6-85
                                Type: FLOAT32
                                Tensor 0: 1, 21-72828, 6-85

                         (14): mobilenet-softmax -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 1000-1001

                         (15): yolo-nas         -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 21-42840, 4
                                Tensor 1: 1, 21-42840
                                Tensor 2: 1, 21-42840
                                Type: FLOAT32
                                Tensor 0: 1, 21-42840, 2
                                Tensor 1: 1, 21-42840, 2
                                Tensor 2: 1, 21-42840, 81
                                Type: FLOAT32
                                Tensor 0: 1, 5-1005, 21-42840
                                Type: FLOAT32
                                Tensor 0: 1, 21-42840, 1-1001
                                Tensor 1: 1, 21-42840, 4
                                Type: FLOAT32
                                Tensor 0: 1, 21-42840, 4
                                Tensor 1: 1, 21-42840, 1-1001

                         (16): qfd              -
                              Supported tensors:
                                Type: UINT8, FLOAT32
                                Tensor 0: 1, 60, 80, 1
                                Tensor 1: 1, 60, 80, 1
                                Tensor 2: 1, 60, 80, 10
                                Tensor 3: 1, 60, 80, 4
                                Type: UINT8, FLOAT32
                                Tensor 0: 1, 120, 160, 1
                                Tensor 1: 1, 120, 160, 10
                                Tensor 2: 1, 120, 160, 4
                                Type: UINT8, FLOAT32
                                Tensor 0: 1, 60, 80, 4
                                Tensor 1: 1, 60, 80, 10
                                Tensor 2: 1, 60, 80, 1
                                Type: UINT8, FLOAT32
                                Tensor 0: 1, 60, 80, 1
                                Tensor 1: 1, 60, 80, 4
                                Tensor 2: 1, 60, 80, 10

                         (17): yamnet           -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 521

                         (18): midas-v2         -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 128-2048, 128-2048, 1
                                Type: FLOAT32
                                Tensor 0: 1, 128-2048, 128-2048

                         (19): qfr-softmax      -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 512
                                Tensor 1: 1, 32
                                Tensor 2: 1, 2
                                Tensor 3: 1, 2
                                Tensor 4: 1, 2
                                Tensor 5: 1, 2

                         (20): qpd              -
                              Supported tensors:
                                Type: FLOAT32
                                Tensor 0: 1, 120, 160, 3
                                Tensor 1: 1, 120, 160, 12
                                Tensor 2: 1, 120, 160, 34
                                Tensor 3: 1, 120, 160, 17
```

Lets take the yolov8 post processing module as an example:

```
(9): yolov8           -
    Supported tensors:
      Type: FLOAT32
      Tensor 0: 1, 21-42840, 4
      Tensor 1: 1, 21-42840
      Tensor 2: 1, 21-42840
      Type: FLOAT32
      Tensor 0: 1, 4, 21-42840
      Tensor 1: 1, 1-1001, 21-42840
      Type: FLOAT32
      Tensor 0: 1, 5-1005, 21-42840
```

**1. Models with three output tensors**

These models produce three separate tensors. The second dimension is dynamic and depends on the number of classes used during training. This flexibility allows a single post-processing module to support multiple YOLOv8 models with different class configurations.

Example shapes:

* 1, (21–42840), 4
* 1, (21–42840)
* 1, (21–42840)

**2. Models with two output tensors**

These models combine bounding box and classification data across two tensors.

Example shapes:

* 1, 4, (21–42840)
* 1, (1–1001), (21–42840)

**3. Models with one output tensor**

These models output all relevant data in a single tensor.

Example shapes:

* 1, (5–1005), (21–42840)

*Data format is Float32 in all cases.*

## Batching & Daisy Chaining

QIM SDK supports advanced features such as **batched models** and **daisy chaining** (executing models sequentially). These complex tasks are handled automatically by the SDK, allowing the post-processing module code to remain generic and focused solely on core post-processing logic.

**Examples:**

* If a model has a batch size of 4, the **qtimlpostprocess** element will execute the **post-processing module** four times—once for each item in the batch. This means the same module can be used in both simple scenarios (processing one frame at a time) and more complex ones (processing multiple sources in parallel).

* In a sequential model setup, where the first model detects objects and the second performs pose estimation, the second model is executed for each detected object from the first model. In this case, **QIM SDK** manages the complexity of invoking the **post-processing module** for each inference result and mapping the output back to the original frame.

## Overview of AI Post Processing use cases

QIM SDK (Qualcomm Intelligent multimedia SDK) is a framework that provides necessary building blocks to construct AI, Multimedia, and CV pipelines for end application. To build AI workflow three components/ GStreamer plugins are needed.

<img alt="AI Post Process overview" />

1. The preprocess element converts the data stream into a tensor.
2. The inference element performs inference on an AI model and eventually applies dequantization to the output tensor. There is no additional preprocessing or postprocessing involved, other than dequantization.
3. The post-processing is a plugin that parses tensors and creates a buffer containing either ML metadata or an image mask. ML metadata can be handled by the QIM SDK in two different ways: it can either be attached to the source stream using qtimetamuxer, or used directly and streamed to RTSP, RTMP, Redis, etc. The image mask can be overlaid on the source video frame using qtivcomposer.

Examples legend:

<img alt="AI Post Process Legend" />

Example 1: The ML metadata is used directly (source stream is not propagated after inference plugin):

<img alt="AI Post Process Example 1" />

Example2: The ML metadata is attached to the source video. The overlay uses the attached ML metadata to draw bounding boxes, text, and other visual elements. The result is either displayed on screen or streamed over the network.

<img alt="AI Post Process Example 1" />

Example3: The ML metadata is converted into an image mask, which is then blitted on top of the source stream.

<img alt="AI Post Process Example 1" />

## Guidelines for Writing an External Post-Processing Callback

External post-processing lets you decode a model's raw output tensors **directly in your application**, instead of writing and deploying a separate C++ module. You connect a single function — the *callback* — to the `qtimlpostprocess` element, and from then on the SDK hands that function every inference result and takes care of everything around it: batching, daisy-chaining, and delivering your results to the downstream plugins. You only write the model-specific decoding in the middle.

This is the quickest way to bring up a new or custom model, and it can be done entirely in Python — no compilation or on-device deployment required.

**To enable it, you only need to:**

1. Build the pipeline **without** setting the `module` property, so no built-in module claims the tensors.
2. Connect a handler to the single `process-*` signal that matches your model type (see [Post Processing Callback](#post-processing-callback) for the full list).

The element then invokes your handler once for every inference result. Downstream plugins consume the entries you produce exactly as if a built-in module had generated them.

<Warning>If you set the `module` property, the built-in module performs the parsing and your callback is **never** invoked. Leave `module` unset to route the tensors to your callback.</Warning>

### Callback Signature

Whatever the model, the callback always has the same shape — it receives the same arguments and returns a boolean. Only the decoding logic in the middle changes. Using object detection as an example, the handler looks like this:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    def on_object_detection(postprocess, mlframe, mlparam, detections):
        ...
        return True
    ```
  </Tab>

  <Tab title="C/C++">
    ```c theme={null}
    static gboolean
    on_object_detection (GstElement * postprocess, GstMLFrame * mlframe,
        GstStructure * mlparam, GstMLDetections * detections, gpointer userdata)
    {
      ...
      return TRUE;
    }
    ```
  </Tab>
</Tabs>

Each argument plays a specific role:

| Argument | Type (C / Python) | What it is |
| - | - | - |
| `postprocess` | `GstElement` / `qtimlpostprocess` | The element that emitted the signal. Rarely needed inside the callback. |
| `mlframe` | `GstMLFrame` / `GstQtiML.Frame` | **The input.** The model's raw output tensors that you decode. See [ML Frame](#ml-frame). |
| `mlparam` | `GstStructure` | Per-batch info describing how the stream was fitted into the model input — needed to map coordinates back to the frame. See [ML Parameters](#ml-parameters). |
| result container | e.g. `GstMLDetections` / a `GstQtiML` list | **The output.** An empty list you populate with one entry per prediction; its type depends on the signal you connected. See [ML Structures](#ml-structures). |
| `userdata` *(C only)* | `gpointer` | The pointer you passed to `g_signal_connect()`. Absent in Python. |
| *return value* | `gboolean` / `bool` | `TRUE` to let the pipeline continue, `FALSE` to abort it. |

And every callback follows the same four steps:

1. **Read** the tensor(s) from `mlframe`.
2. **Decode** them into predictions — this is your model-specific logic.
3. **Append** one entry per prediction to the result container, filling in its fields (coordinates, label, confidence, …).
4. **Return** `TRUE`.

The three sections that follow describe the two inputs (`mlframe`, `mlparam`) and the output containers in detail, and the [Example](#example) at the end puts them together into a working handler.

### ML Frame

**This is the input to your callback** — the `mlframe` argument, of type `GstMLFrame` / `GstQtiML.Frame`. It holds the model's raw output tensors, which you decode in step 2. A frame carries one or more tensors. For example, output from object-detection model may produce separate tensors for boxes, scores and class indices.

| Field / accessor | Type (C / Python) | Description |
| - | - | - |
| number of tensors | `GST_ML_FRAME_N_TENSORS()` / `mlframe.info.n_tensors` | How many tensors the frame holds. |
| individual tensor | `gst_ml_frame_get_tensor(mlframe, i)` / `mlframe.get_tensor(i)` | The `i`-th tensor. |

In C, `gst_ml_frame_get_tensor()` returns a `GstMLTensor` wrapper for direct access to the tensor:

| Field | Type | Description |
| - | - | - |
| `type` | `GstMLType` | Element data type (e.g. `GST_ML_TYPE_FLOAT32`). |
| `n_dimensions` | `guint` | Number of valid entries in `dimensions`. |
| `dimensions` | `guint[]` | Tensor shape, e.g. `{1, 8400, 4}`. |
| `data` | `guint8 *` | Pointer to the mapped tensor data. |
| `size` | `gsize` | Size of `data` in bytes. |

In Python, `mlframe.get_tensor(i)` returns a **zero-copy NumPy array** already shaped to the tensor dimensions and typed to match the tensor's data type, so it can be indexed and sliced directly.

### ML Parameters

Alongside the tensors, each callback receives the `mlparam` argument, a `GstStructure` of per-batch parameters describing how the input stream was fitted into the model input tensor. You need this because the stream resolution and aspect ratio often differ from the tensor shape: the frame is usually scaled and letter-boxed into the tensor, so a prediction's position in the tensor must be mapped back to the original frame. These fields tell you how. The available fields:

| Field | Type | Description |
| - | - | - |
| `input-tensor-width`, `input-tensor-height` | uint | Full dimensions of the model input tensor. |
| `input-region-x`, `input-region-y`, `input-region-width`, `input-region-height` | int | The rectangle inside the input tensor that was actually filled with stream data; the rest is padding. |

### ML Structures

**This is the output of your callback.** The last argument is an empty **result container** — an ordered, growable list. In step 3 you fill it with one **entry** per prediction, setting the entry's fields (coordinates, label, confidence, and so on).

Which entry type you use is fixed by the signal you connected — a `process-object-detection` handler fills `Detection` entries, `process-pose-estimation` fills `Pose` entries, and so on. Find your signal below and fill the matching fields; you only ever need the one that applies to your model. Coordinates are always **relative** (`0.0`–`1.0`), so they stay correct regardless of the source resolution.

**Classification**

Populated in `process-image-classification` and `process-audio-classification`.

| Field | Type (C / Python) | Description |
| - | - | - |
| `name` | `GQuark` / `str` | Predicted class label. |
| `confidence` | `gfloat` / `float` | Percentage certainty, `0.0`–`100.0`. |
| `color` | `guint32` / `int` | RGBA8888 (`0xRRGGBBAA`) visualization color. |
| `xtraparams` (optional) | `GstStructure` / `Gst.Structure` | Extra parameters passed downstream. |

**Detection**

Populated in `process-object-detection`.

| Field | Type (C / Python) | Description |
| - | - | - |
| `name` | `GQuark` / `str` | Class label. |
| `confidence` | `gfloat` / `float` | Percentage certainty, `0.0`–`100.0`. |
| `color` | `guint32` / `int` | RGBA8888 (`0xRRGGBBAA`) visualization color. |
| `left`, `top`, `right`, `bottom` | `gfloat` / `float` | Bounding-box corners, relative `0.0`–`1.0`. |
| `landmarks` (optional) | `GArray<GstMLKeypoint>` / `list[Keypoint]` | Per-object key points (e.g. facial landmarks). |
| `xtraparams` (optional) | `GstStructure` / `Gst.Structure` | Extra parameters passed downstream. |

**Keypoint**

Used inside `Pose.keypoints` and `Detection.landmarks`; not appended to a container on its own.

| Field | Type (C / Python) | Description |
| - | - | - |
| `name` | `GQuark` / `str` | Keypoint name. |
| `confidence` | `gfloat` / `float` | Percentage certainty, `0.0`–`100.0`. |
| `color` (optional) | `guint32` / `int` | RGBA8888 (`0xRRGGBBAA`) visualization color. |
| `x`, `y` | `gfloat` / `float` | Keypoint location, relative `0.0`–`1.0`. |

**KeypointLink**

Used inside `Pose.links` to connect two keypoints; not appended to a container on its own.

| Field | Type (C / Python) | Description |
| - | - | - |
| `l_kp` | `GstMLKeypoint` / `Keypoint` | First endpoint of the link. |
| `r_kp` | `GstMLKeypoint` / `Keypoint` | Second endpoint of the link. |
| `color` (optional) | `guint32` / `int` | RGBA8888 (`0xRRGGBBAA`) color of the connecting line. |

**Pose**

Populated in `process-pose-estimation`.

| Field | Type (C / Python) | Description |
| - | - | - |
| `name` | `GQuark` / `str` | Class label. |
| `confidence` | `gfloat` / `float` | Overall confidence for the pose, `0.0`–`100.0`. |
| `keypoints` | `GArray<GstMLKeypoint>` / `list[Keypoint]` | Detected key points. |
| `links` (optional) | `GArray<GstMLKeypointLink>` / `list[KeypointLink]` | Connections between key points. |
| `xtraparams` (optional) | `GstStructure` / `Gst.Structure` | Extra parameters passed downstream. |

**Segmentation**

Populated in `process-segmentation`.

| Field | Type (C / Python) | Description |
| - | - | - |
| `labels` | `GArray<GQuark>` / `list[str]` | Class label for each cell of the mask. |
| `colors` (optional) | `GArray<guint32>` / `list[int]` | RGBA8888 color for each cell of the mask. |
| `n_rows` | `guint` / `int` | Number of rows in the mask. |
| `n_columns` | `guint` / `int` | Number of columns in the mask. |
| `xtraparams` (optional) | `GstStructure` / `Gst.Structure` | Extra parameters passed downstream. |

The `labels` and `colors` lists are stored row-major and contain `n_rows` × `n_columns` entries — one per cell.

**DepthMap**

Populated in `process-depth-estimation`.

| Field | Type (C / Python) | Description |
| - | - | - |
| `values` | `GArray<gdouble>` / `list[float]` | Depth value for each cell of the map. |
| `colors` (optional) | `GArray<guint32>` / `list[int]` | RGBA8888 color for each cell of the map. |
| `n_rows` | `guint` / `int` | Number of rows in the map. |
| `n_columns` | `guint` / `int` | Number of columns in the map. |
| `xtraparams` (optional) | `GstStructure` / `Gst.Structure` | Extra parameters passed downstream. |

The `values` and `colors` lists are stored row-major and contain `n_rows` × `n_columns` entries — one per cell.

### Example

The handler below puts the four steps together for an object-detection model: it reads the input tensor from the `mlframe`, reads the fitting parameters from `mlparam`, decodes the tensor into boxes (the model-specific part, omitted here), and appends one `Detection` per box to the result container. The final lines show how the handler is connected to the element — note that the `module` property is **not** set.

<Tabs>
  <Tab title="Python application">
    In Python the signals are available through GObject introspection, and the ML result types (`Detection`, `Pose`, `Classification`, `Keypoint`, …) come from the `GstQtiML` overrides. The input tensor is exposed as a zero-copy NumPy array via `Frame.get_tensor()`.

    ```python theme={null}
    import gi
    gi.require_version('Gst', '1.0')
    gi.require_version('GstVideo', '1.0')
    gi.require_version('GstQtiML', '1.0')
    from gi.repository import Gst, GstVideo, GstQtiML

    def on_object_detection(postprocess, mlframe, mlparam, detections):
        # Input inference tensor as a zero-copy NumPy array.
        tensor = mlframe.get_tensor(0)

        # mlparam is a GstStructure — access fields with .get_value("key")
        # Fetch the width and height of the model input image tensor
        width = mlparam.get_value("input-tensor-width")
        height = mlparam.get_value("input-tensor-height")

        # Fetch the region in the model input image tensor populated with actual data
        region = GstVideo.VideoRectangle()

        region.x = mlparam.get_value("input-region-x")
        region.y = mlparam.get_value("input-region-y")
        region.w = mlparam.get_value("input-region-width")
        region.h = mlparam.get_value("input-region-height")

        # Decode `tensor` into bounding boxes (model-specific logic omitted).
        # For every detected object, build a Detection and append it.
        detection = GstQtiML.Detection()
        detection.name = "person"
        detection.confidence = 98.0
        detection.color = 0x00FF00FF

        # Coordinates are relative (0.0 - 1.0).
        detection.left = 0.1
        detection.top = 0.1
        detection.right = 0.4
        detection.bottom = 0.4

        detections.append(detection)

        return True  # return False to signal an error and stop the pipeline

    # ... after building the pipeline (note: the 'module' property is NOT set) ...
    postprocess = pipeline.get_by_name("postprocess")
    postprocess.connect("process-object-detection", on_object_detection)
    ```
  </Tab>

  <Tab title="C/C++ application">
    In C/C++ connect a callback to the matching signal, read the input tensor from the `GstMLFrame`, and append your results to the provided container.

    ```c theme={null}
    #include <gst/ml/ml-frame.h>
    #include <gst/ml/ml-post-process-detection.h>

    static gboolean
    on_object_detection (GstElement * postprocess, GstMLFrame * mlframe,
        GstStructure * mlparam, GstMLDetections * detections, gpointer userdata)
    {
      // Input inference tensor.
      GstMLTensor tensor = gst_ml_frame_get_tensor (mlframe, 0);

      // mlparam is a GstStructure — access fields with gst_structure_get_*
      // Fetch the width and height of the model input image tensor
      gint width = 0, height = 0;

      gst_structure_get_int (mlparam, "input-tensor-width", &width);
      gst_structure_get_int (mlparam, "input-tensor-height", &height);

      // Fetch the region in the model input image tensor populated with actual data
      GstVideoRectangle region = { 0, };

      gst_structure_get_int (mlparam, "input-region-x", &region.x);
      gst_structure_get_int (mlparam, "input-region-y", &region.y);
      gst_structure_get_int (mlparam, "input-region-width", &region.w);
      gst_structure_get_int (mlparam, "input-region-height", &region.h);

      // Decode `tensor.data` using `tensor.dimensions` (model-specific logic
      // omitted). For every detected object, fill a detection and append it.
      GstMLDetection detection = { 0, };

      detection.name = g_quark_from_string ("person");
      detection.confidence = 98.0f;
      detection.color = 0x00FF00FF;

      // Coordinates are relative (0.0 - 1.0).
      detection.left = 0.1f;
      detection.top = 0.1f;
      detection.right = 0.4f;
      detection.bottom = 0.4f;

      gst_ml_detections_append (detections, &detection);

      return TRUE; // return FALSE to signal an error and stop the pipeline
    }

    // ... after creating the element (note: the 'module' property is NOT set) ...
    g_signal_connect (postprocess, "process-object-detection",
        G_CALLBACK (on_object_detection), NULL);
    ```
  </Tab>
</Tabs>

## Guidelines for Writing a Custom Post-Processing Module

If you cannot find a suitable post-processing module for your AI model, you can implement your own. You can build a post-processing module completely independently from the QIM SDK — all you need are the interface header files and a toolchain. Once the module is built, it should be deployed to the following location on the device: /usr/lib/gstreamer-1.0/ml/modules/. The post-processing plugin will automatically detect it, and users can select it in the GStreamer pipeline.

[Post processing module header file(s)](https://git.codelinaro.org/clo/le/platform/vendor/qcom-opensource/gst-plugins-qti-oss/-/blob/imsdk.lnx.2.0.0/gst-plugin-mlpostprocess/modules/qti-ml-post-process.h?ref_type=heads)

### Module/library naming

Post-processing module shared libraries must follow the naming convention: libml-postprocess-\<module-name>.so. This is required to avoid duplication of post-processing module names. For example, the shared library for the YoloV8 module should be named libml-postprocess-yolov8.so. The same \<module-name> is used when configuring the post-processing plugin, for example: **module=yolov8.**

```bash theme={null}
gst-launch-1.0 -e --gst-debug=2 filesrc location=$HOME/Downloads/qimsdk_samples/media/ai_demo_sample.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! qtimlvconverter ! qtimltflite name=inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/yolov8_det_w8a8.tflite ! qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/yolov8_json.labels ! text/x-raw ! filesink location=$HOME/data/ml-results.txt
```

### AI Post Processing Module Inference

```
class IModule {
 public:
  virtual ~IModule() {};

  virtual std::string Caps() = 0;

  virtual bool Configure(const std::string& labels_file, const std::string& json_settings) = 0;

  virtual bool Process(const Tensors& tensors, Dictionary& mlparams, std::any& output) = 0;
};
```

The **post-processing modules** expose a C++ API. Since C++ classes cannot be directly instantiated from shared libraries, class creation is encapsulated in a C-style function. Module developers must include the following code in the source file:

```
IModule* NewModule(LogCallback logger) {
  return new Module(logger);
}
```

The developer only needs to implement the following APIs in the Module class, which derives from the IModule interface:

* **Constructor / Destructor** - For initialization and cleanup.
* **Caps** - This function must return the module type (e.g., image classification, object detection), supported tensor dimensions, and supported data types (e.g., uint8, float32).
* **Configuration** - Called once during initialization. Handles label and configuration files.
* **Process** - Called after each inference. This is where the tensor is converted into a prediction result in one of the supported formats.

**std::string Caps()**

This API returns the module type and the supported tensor shapes as a JSON string. The tensor shape is not fixed but defined within a range, represented using square brackets. For example, \[1, \[21, 42840], 4] indicates that the second dimension can vary between 21 and 42840.

Example definition of post processing module capabilities. Module implements "object-detection" post-processing in this example. Three different tensor outputs are support: one tensor, two tensors, three tensors. Supported tensor format is only FLOAT32 in this example.

```
static const std::string kModuleCaps = R"(
{
  "type": "object-detection",
  "tensors": [
    {
      "format": ["FLOAT32"],
      "dimensions": [
        [1, [21, 42840], 4],
        [1, [21, 42840]],
        [1, [21, 42840]]
      ]
    },
    {
      "format": ["FLOAT32"],
      "dimensions": [
        [1, 4, [21, 42840]],
        [1, [1, 1001], [21, 42840]]
      ]
    },
    {
      "format": ["FLOAT32"],
      "dimensions": [
        [1, [5, 1005], [21, 42840]]
      ]
    }
  ]
}
)";
```

Supported Post Processed Module types:

* object-detection
* image-classification
* image-segmentation
* depth-estimation
* super-resolution
* pose-estimation
* audio-classification
* tensor

Supported Tensor Types:

* FLOAT32
* FLOAT16
* INT8
* UINT8
* INT16
* UINT16
* INT32
* UINT32
* INT64
* UINT64

More than one format could be specified in the same time. Example:

```
...
     {
      "format": ["FLOAT32", "INT8"],
      "dimensions": [
        [1, 4, [21, 42840]],
        [1, [1, 1001], [21, 42840]]
      ]
    },
...
```

**bool Configure(const std::string& labels, const std::string& settings)**

* **labels** - (optional) а string that holds the path to a file containing labels. If the user does not provide a label file, the string remains empty. The label file can be in any format. The IM SDK includes parsers for both newline-separated labels and JSON-formatted labels.
* **settings** - (optional) a JSON string containing module-specific setting. These settings are provided by the user through the settings property of the post-processing GStreamer plugin. It will be empty if user does not provide any settings.

**bool Process(const Tensors& tensors, Dictionary& mlparams, std::any& output)**
The module takes as input: tensors, tensors shape, information how input tensor is filled. The output should be a list of predictions in one of the supported formats

* object-detection
* image-classification
* image-segmentation
* depth-estimation
* super-resolution
* pose-estimation
* audio-classification
* tensors

<Note>
  Tensor output is a special case where the post-processing plugin and module generate tensors instead of predictions. This is used when two ML models are chained together and the output tensor from the first model needs to be modified before it is passed to the next model. If the output tensor does not require modification, then both inference plugins can be linked directly, one after the other, and the post-processing plugin is not needed in that case.
</Note>

#### Understanding post processing module input

The input is split into two fields:

1. **tensor** – This field holds the inference output tensors and describes their structure. Each output tensor is represented as an entry in a vector. For example, in the case of YOLOv8, which produces three output tensors (boxes, scores, class indices), the vector will contain three entries.
   * type – float, uint8, etc
   * name - tensor name. This field is useful when two or more output tensors have the same shape. Tensor names are unique and guarantee that an exact tensor is selected.
   * dimensions – this field describes the tensor's shape. For example, YoloV8 with three output tensors:  \[1,8400,4], \[1,8400], \[1,8400]
   * data – pointer to tensor
2. **mlparams** – Additional parameters that may be required for tensor processing. These may not be applicable to all submodules. This field also provides information about how the input stream is processed, which is particularly important because the resolution and aspect ratio of the stream often do not match the shape of the input tensor. This field is a dictionary implemented using std::any. The module developer must know the expected key and its corresponding return type. The use of std::any ensures that the returned value matches the type associated with the given key. Example usage:

```
auto& resolution = std::any_cast<Resolution&>(mlparams["input-tensor-dimensions"]);
auto& region = std::any_cast<Region&>(mlparams["input-tensor-region"]);
```

Supported keys:

* Key: "input-tensor-region"<br />
  Type: Region<br />
  Description: This parameter indicates which portion of the input tensor is filled with actual data from the stream. The remaining area is considered padding<br />

* Key: "input-tensor-dimensions"<br />
  Type: Resolution<br />
  Description: Specifies the size of the input tensor. This is useful when the post-processing algorithm produces output in absolute coordinates. Since post-processing modules are required to output relative coordinates, the input tensor size is needed to convert absolute values to relative ones.<br />

#### Generating post processing module output

The output is array of array of results. Arrays are nested because of the batching case. Only inner array is filled if there is no batching. Inner array size match to number of found result.  Results are always in relative dimension. Result type depends on module type:

1. **Image/Audio Classification**
   * name – class label. Predicated category or class the image/audio belong to.
   * confidence – class probability / confidence score
   * color – RGBA8888 color for visualization in overlay plugin
   * xtraparams – (optional) additional parameters in #Dictionary (key/value pair) which the user can export arbitrary extra results from the module and be passed downstream.
2. Object Detection:
   * left, top, right, bottom – bounding box coordinates
   * name – class label. Predicated category or class the image/audio belong to.
   * landmarks – (optional) list of key points. For example, face detection model can output face point along with bounding box.
   * confidence – class probability / confidence score
   * color – RGBA8888 color for visualization in overlay plugin
   * xtraparams – (optional) additional parameters in #Dictionary (key/value pair) which the user can export arbitrary extra results from the module and be passed downstream.
3. Pose Estimation:
   * name – class label. Predicated category or class the image/audio belong to.
   * confidence – class probability / confidence score
   * keypoints – vector of key points
   * Links – (optional) vector of links between key points.
   * color – RGBA8888 color for visualization in overlay plugin
   * xtraparams – (optional) additional parameters in #Dictionary (key/value pair) which the user can export arbitrary extra results from the module and be passed downstream.
4. Image Segmentation:
   * labels – list of class labels, one per cell (paxel) of the segmentation mask.
   * color – list of RGBA8888 colors, one per cell, for visualization in the overlay plugin.
   * n\_rows / n\_columns – dimensions of the segmentation mask produced for the processed region.
   * xtraparams – (optional) additional parameters in #Dictionary (key/value pair) which the user can export arbitrary extra results from the module and be passed downstream.
5. Depth Estimation:
   * values – list of per-cell depth values for the processed region.
   * color – list of RGBA8888 colors, one per cell, for visualization in the overlay plugin.
   * n\_rows / n\_columns – dimensions of the depth map produced for the processed region.
   * xtraparams – (optional) additional parameters in #Dictionary (key/value pair) which the user can export arbitrary extra results from the module and be passed downstream.
6. Super Resolution:
   * Output is image frame/mask
7. Tensor
   * list of tensors

### Module helper tools

As part of interface header files we also provide label and JSON parsers. User is not obligated to use neither them. They are provided for convenience only. Developer can use any label and/or JSON parser but module must be linked statically with them.

**Label parser** – This parser support two formats. Label parser takes path to file with labels and automatically detects formatting:

* New line separated format. Line number is class id.
* JSON format. Class index, label, visualization color should be set in this format. This format is more flexible because user can pass only some classes. The rest of the classes will be automatically filtered out.

**JSON parser** – Settings are passed in JSON string. So this utility is useful to parse settings. This implementation is also used in our label parser in case of JSON format.

### Logging

The post-processing module can output logs to the GStreamer log system without having a direct dependency on GStreamer. A logging object is passed to the module via its constructor. This object, along with a LOG macros, can be used to output logs directly to the GStreamer log. Supported log levels include: Error, Warning, Info, Debug, Trace, and Log..

LOG macro:

```
#define LOG(logger, level, fmt, ...)
```

Example of logging usage:

```
LOG(logger_, kError, "ML frame with unsupported post-processing procedure!");
LOG(logger_, kLog, "Threshold: %f", threshold_);
```

## How to Compile the Post-Processing Module standalone

Prerequisite: Ubuntu22.04 or Ubuntu24.04 PC

1. Install tools

```
sudo apt-get install g++-aarch64-linux-gnu
sudo apt-get install cmake
```

2. Put QIM SDK headers and module sources in one folder.

```
ml-postprocess-yolov8.cc
ml-postprocess-yolov8.h
qti-json-parser.h
qti-labels-parser.h
qti-ml-post-process.h
```

3. Create a CMakeLists.txt file. Example:

```
cmake_minimum_required(VERSION 3.8.2)
project(QTI_OSS_ML_MODULES LANGUAGES C CXX)

set(CMAKE_INCLUDE_CURRENT_DIR ON)

# Common compiler flags.
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_FLAGS "${CMAKE_C_FLAGS} -Wall -Wextra -Werror")
set(CMAKE_CXX_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-parameter")

include_directories(
  $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}>
)

set(CMAKE_INCLUDE_CURRENT_DIR ON)

set(TARGET_NAME ml-postprocess-yolov8)

add_library(${TARGET_NAME} SHARED
  ml-postprocess-yolov8.cc
)
```

<Note>
  Post-processing module shared libraries must follow the naming convention: libml-postprocess-\<module-name>.so
  For example, the shared library for the YoloV8 module should be named libml-postprocess-yolov8.so
</Note>

4. Create a toolchain file e.g. aarch64-toolchain.cmake. For example:

```
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)
set(CMAKE_CXX_COMPILER aarch64-linux-gnu-g++)
set(CMAKE_CXX_FLAGS "-march=armv8-a")
```

5. Configure and build project

```
mkdir build
cd build
cmake -DCMAKE_TOOLCHAIN_FILE=../aarch64-toolchain.cmake ..
cmake --build .
```

## How to Deploy and Test the Post-Processing Module

1. Deploy module on device

```
scp libml-postprocess-yolov8.so  <user>@<device-ip>:/usr/lib/gstreamer-1.0/ml/modules/
```

2. Run GST inspect and check if your module appears in supported modules list. You have to see you post processing module in supported modules list along with supported tensors shape.

```
gst-inspect-1.0 qtimlpostprocess
```

3. Once you have the post-processing module, you need to build a GStreamer pipeline. You must select your post-processing module using the module property of the qtimlpostprocess plugin. If your module requires a label file or configuration, you must pass them accordingly via the label and settings properties. <br />
   Below is an example pipeline for running a YOLOv8 model. An offline video is used as the video source. The video is decoded to YUV format using the v4l2h264dec decoder. YUV frames are preprocessed by the qtimlvconverter plugin. The qtimltflite plugin is used to run inference with the TensorFlow Lite YOLOv8 model. The post-processing plugin loads the YOLOv8 module and passes a label file in JSON format. The ML results are saved to a file.

```bash theme={null}
gst-launch-1.0 -e --gst-debug=2 filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! qtimlvconverter  ! qtimltflite name=inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/yolov8_det_w8a8.tflite ! qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/yolov8_json.labels ! text/x-raw ! filesink location=$HOME/data/ml-results.txt
```


# qtimlqnn
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtimlqnn

GStreamer inference element that executes neural network models using QNN runtime.

<Note>
  qtimlqnn is only available in `qcom-multimedia-proprietary-image`  <br />
  For more information on QLI images refer to [Qualcomm Linux release](https://dragonwingdocs.qualcomm.com/Key-Documents/Yocto-Guide/add-image-recipes#qualcomm-reference-images)
</Note>

# Overview

**qtimlqnn** is a GStreamer inference element that executes neural network models using the **Qualcomm AI Engine Direct (QNN)** runtime. The element operates entirely in **tensor mode**: it accepts input tensors on its sink pad and produces output tensors on its source pad according to the model's declared input and output specifications.

**qtimlqnn** is designed to run models prepared for the QNN runtime, typically in the form of a **QNN context binary**. To use this element, the model must first be exported to a QNN-compatible format using the **Qualcomm AI Runtime (QAIRT) SDK**. For additional details, refer to the [QAIRT documentation](https://docs.qualcomm.com).

The element is limited to **model execution**. It does not perform preprocessing, tensor reshaping, batching, layout conversion, or model-specific post-processing. These functions are expected to be handled by adjacent elements in the pipeline.

**qtimlqnn** supports multiple QNN execution backends, including **CPU**, **GPU**, and **NPU** targets. This allows the same pipeline structure to be deployed across different hardware configurations and tuned for different performance, latency, and power requirements. The element is intended for **real-time and embedded AI pipelines**, where inference is one stage in a larger modular processing flow.

### Key Responsibilities

**qtimlqnn** is responsible for:

* Loading and executing a QNN model artifact, such as a model library (`.so`) or cached binary (`.bin`)
* Accepting preformatted input tensors from upstream elements
* Producing output tensors that match the model output signature
* Negotiating tensor data types and dimensions with adjacent pipeline elements
* Propagating tensor metadata required by downstream elements
* Managing DMA-backed buffers through `GstMLBufferPool` to reduce unnecessary memory copies

In practice, **qtimlqnn** serves as the inference stage in the pipeline, while tensor preparation and result interpretation are handled externally.

<img alt="" />

## Example Pipeline

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Yolov8 Detection W8A8 model | [Export from Qualcomm AI Hub](https://aihub.qualcomm.com/iot/models/yolov8_det) | `yolov8_det_w8a8.bin` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp yolov8_det_w8a8.bin          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/models $HOME/Downloads/qimsdk_samples/labels $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/media/output
    export MODEL_NAME=yolov8_det_w8a8.bin
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    tee name=t ! queue ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
    t. ! queue ! qtimlvconverter ! queue ! \
    qtimlqnn model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME backend=/usr/lib/libQnnHtp.so tensors="<boxes,scores,class_idx>" ! queue ! \
    qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! obj_mux.
    ```
  </Step>
</Steps>

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />[GstBaseTransform](https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c)<br />
            <Icon icon="arrow-turn-down-right" />qtimlqnn

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `neural-network/tensors` | `format: { INT8, UINT8, INT16, UINT16, INT32, UINT32, FLOAT16, FLOAT32 }` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `neural-network/tensors` | `format: { INT8, UINT8, INT16, UINT16, INT32, UINT32, FLOAT16, FLOAT32 }` |
| Availability: *Always* | |
| Direction: *source* | |

# Element Properties

| Property | Description |
| - | - |
| `backend` | Path to the QNN backend library. Selects the execution backend used for inference. Supported backends include CPU, HTP or NPU, GPU, and DSP implementations depending on the library used.<br /><br />`Type: String`<br />`Default: "/usr/lib/libQnnCpu.so"`<br />`Flags: readable/writable` |
| `backend-device-id` | Backend device selector. Platform dependent and used for some DSP or HTP variants to select a specific hardware instance.<br /><br />`Type: Unsigned Integer`<br />`Default: 0`<br />`Flags: readable/writable` |
| `model` | Path to the QNN model file. This property is required and must reference a valid `.so` model or cached `.bin` file.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `system` | Path to the QNN system library required for QNN runtime initialization.<br /><br />`Type: String`<br />`Default: "/usr/lib/libQnnSystem.so"`<br />`Flags: readable/writable` |
| `tensors` | Output tensor filter. When set, only the specified output tensor names are emitted on the source pad. When empty, all model outputs are emitted.<br /><br />`Type: GstValueArray of type gchararray`<br />`Default: "< >"`<br />`Flags: readable/writable` |

# Input and Output Behavior

### Input Tensors

**qtimlqnn** exposes a single sink pad, but it supports both single-input and multi-input models. For multi-input models, all required tensors are delivered through the same sink pad as a tensor set.

Input tensors must be fully prepared before they reach **qtimlqnn**. Expected tensor layout, shape, data type, and batch size are determined by:

* The QNN model input signature
* Caps negotiation with upstream elements

Typical upstream elements include:

* [`qtimlvconverter`](/plugin-reference/qtimlvconverter) — for scaling, color conversion, normalization, and quantization (if required)

**qtimlqnn** does not modify, reshape, batch, or reinterpret incoming tensors. It passes them to the QNN runtime as received.

### Output Tensors

**qtimlqnn** exposes a single source pad and produces output tensors that follow the model's declared output signature. Models with multiple output tensors are fully supported, and all outputs are emitted together on the source pad.

Supported output behavior includes:

* Single-tensor and multi-tensor outputs
* Arbitrary tensor shapes and ranks, including batch and depth dimensions
* Both quantized and floating-point tensor types
* Selective emission of output tensors using the `tensors` property

The generated output tensors are intended for downstream post-processing stages, which are responsible for decoding model-specific results such as classification outputs, detection results, segmentation masks, landmark data, and other structured inference outputs.

# Supported Data Types

**qtimlqnn** supports the tensor data types provided by the QNN runtime and the selected execution backend, subject to caps negotiation with adjacent elements.

Supported data types include:

* `INT8`
* `UINT8`
* `INT16`
* `UINT16`
* `INT32`
* `UINT32`
* `FLOAT16`
* `FLOAT32`

The element does not impose additional data-type restrictions beyond those required by the runtime, the selected backend, and negotiated pipeline caps.

# Backends

A QNN **backend** defines the hardware target used to run a model. Backends allow **qtimlqnn** to offload inference from the default CPU interpreter to an optimized hardware accelerator. The backend is selected through the `backend` property and controls how the QNN runtime dispatches model operations during inference.

### NPU — libQnnHtp.so

Runs the model on the AI accelerator (NPU).

* **Backend**: Qualcomm AI Accelerator / NPU
* **Use case**: Preferred backend where available. Best performance and power efficiency for quantized models.
* **Additional configuration**: set `backend=libQnnHtp.so`; optionally set `backend-device-id` for multi-device platforms

### GPU — libQnnGpu.so

Runs supported operations through the QNN GPU backend.

* **Backend**: GPU
* **Use case**: Floating-point models and workloads that benefit from GPU parallelism.
* **Additional configuration**: set `backend=/usr/lib/libQnnGpu.so`

### CPU — libQnnCpu.so

Runs the model on the default QNN CPU backend.

* **Backend**: CPU
* **Use case**: Reference execution, debugging, bring-up, or systems without hardware acceleration.
* **Additional configuration**: none required

# Runtime Memory Behavior and GAP Handling

### QNN Memory Model

**qtimlqnn** operates within the memory model of the QNN runtime. The element uses DMA buffers via `GstMLBufferPool` to minimize memory copies and maintain zero-copy transport where possible.

QNN uses runtime-managed memory to allocate:

* Input tensors
* Intermediate activation tensors
* Output tensors

The element discovers input/output tensor metadata (count, shape, type) at model load time and configures buffer pools accordingly.

### GAP Buffer Handling

**qtimlqnn** is GAP-aware and correctly handles input buffers marked with `GST_BUFFER_FLAG_GAP`.

When a GAP buffer is received, the element skips inference and forwards the buffer downstream. This preserves timing and synchronization while explicitly indicating that no valid inference input is available for that timestamp.

GAP buffers commonly appear in conditional AI pipelines, such as cascaded workflows where later inference stages run only when earlier stages produce valid regions of interest.

# Usage

### Single-Stage AI Inference on Live Camera Stream (HTP)

This example demonstrates real-time inference on a live camera stream using a single instance of **qtimlqnn** with the HTP backend. Inference results are attached to each `GstBuffer` as `MLMeta`, allowing downstream elements to access synchronized metadata directly from the frame. An overlay stage then uses this metadata to render annotations such as bounding boxes, labels, or key-points before display or further processing.

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Yolov8 Detection W8A8 model | [Export from Qualcomm AI Hub](https://aihub.qualcomm.com/iot/models/yolov8_det) | `yolov8_det_w8a8.bin` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"
      scp yolov8_det_w8a8.bin   <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json           <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/models $HOME/Downloads/qimsdk_samples/labels
    export MODEL_NAME=yolov8_det_w8a8.bin
    export LABELS_NAME=yolov8.json
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash Run the pipeline theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! \
    tee name=t ! queue ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
    t. ! queue ! qtimlvconverter ! queue ! \
    qtimlqnn model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME backend=/usr/lib/libQnnHtp.so tensors="<boxes,scores,class_idx>" ! queue ! \
    qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! obj_mux.
    ```
  </Step>
</Steps>

### Single-Stage AI Inference on Live Camera Stream (GPU)

This example demonstrates the same single-stage inference workflow using the GPU backend instead of HTP. This is suitable for floating-point models or workloads that benefit from GPU parallelism.

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Yolov8 Detection Float model | [Export from Qualcomm AI Hub](https://aihub.qualcomm.com/iot/models/yolov8_det) | `yolov8_det_float.bin` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"
      scp yolov8_det_float.bin   <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json            <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolov8_det_float.bin
    export LABELS_NAME=yolov8.json
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash Run the pipeline theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! \
    tee name=t ! queue ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
    t. ! queue ! qtimlvconverter ! queue ! \
    qtimlqnn model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME backend=/usr/lib/libQnnGpu.so tensors="<boxes,scores,class_idx>" ! queue ! \
    qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! obj_mux.
    ```
  </Step>
</Steps>


# qtimlsnpe
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtimlsnpe

neural network models using Qualcomm’s SNPE

<Note>
  qtimlsnpe is only available in `qcom-multimedia-proprietary-image`  <br />
  For more information on QLI images refer to [Qualcomm Linux release](https://dragonwingdocs.qualcomm.com/Key-Documents/Yocto-Guide/add-image-recipes#qualcomm-reference-images)
</Note>

## Overview

qtimlsnpe executes neural network models using Qualcomm’s Snapdragon Neural Processing Engine (SNPE). Models are packaged as DLC files and, once loaded, the runtime exposes the model’s input and output signature (tensor count, shapes, and element types). SNPE provides multiple execution targets—including CPU, NSP, GPU (Adreno), AIP —so the same model can be deployed across hardware with different performance, latency, and power characteristics. qtimlsnpe surfaces SNPE’s key runtime controls as simple tunables: a delegate selector (CPU/DSP/GPU) to choose the preferred target, performance profiles to trade power for throughput or latency, optional profiling levels for runtime diagnostics, and an execution priority hint. These settings do not change model accuracy by themselves; they help match runtime behavior to your deployment goals. Inputs and outputs flow as neural-network/tensors. The element derives exact tensor shapes and types from the DLC at runtime, ensuring downstream components receive tensors that reflect the model’s declared outputs. When needed by downstream algorithms, outputs can be requested as FLOAT32 even if the model is quantized, enabling dequantization without changing the model artifact.

## Key Responsibilities

qtimlsnpe is responsible for:

* loading and executing an SNPE DLC model on CPU, DSP (Hexagon), GPU (Adreno), or AIP

* accepting preformatted input tensors from upstream elements

* producing output tensors that match the model output signature

* negotiating tensor data types and dimensions with adjacent pipeline elements

* propagating tensor metadata required by downstream elements

* managing buffers through SNPE user buffer mode to reduce unnecessary memory copies

* exposing runtime controls including delegate, performance-profile, profiling-level, and priority

* supporting explicit output filtering by named layers or tensors (order-preserving)

In practice, qtimlsnpe serves as the inference stage in the pipeline, while tensor preparation and result interpretation are handled externally.

## Example Pipeline

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX model | [Qualcomm AI hub model](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.dlc` |
    | YOLO labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Input video | [Input Video](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/) | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
      scp yolox_w8a8.dlc <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4 <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolox_w8a8.dlc
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! queue ! tee name=split \
    split. ! queue ! qtivcomposer name=mixer sink_1::dimensions="<1920,1080>" ! queue ! waylandsink fullscreen=true sync=true \
    split. ! queue ! video/x-raw,format=NV12 ! qtimlvconverter ! queue ! qtimlsnpe delegate=dsp tensors="<boxes,scores,class_idx>" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME ! queue ! qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! video/x-raw ! queue ! mixer.
    ```
  </Step>
</Steps>

## Plugin Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />[GstBaseTransform](https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c)<br />
            <Icon icon="arrow-turn-down-right" />qtimlsnpe

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `neural-network/tensors` | `type: { INT8, UINT8, INT16, UINT16, INT32, UINT32, INT64, UINT64, FLOAT16, FLOAT32 }` |

Availability: *Always*
Direction: *sink*

***

### src

| Capabilities | |
| - | - |
| `neural-network/tensors` | `type: { INT8, UINT8, INT16, UINT16, INT32, UINT32, INT64, UINT64, FLOAT16, FLOAT32 }` |

Availability: *Always*
Direction: *source*

***

# Element Properties

| Property | Description |
| - | - |
| `delegate` | Delegate the graph execution to a runtime backend.<br /><br />`Type: Enum`<br />`Default: DEFAULT_PROP_DELEGATE`<br />`Range:`<br />    `(0): none - CPU execution (fallback always available)`<br />    `(1): dsp - DSP execution`<br />    `(2): gpu - GPU execution`<br />    `(3): aip - AIP execution`<br />`Flags: readable/writable` |
| `layers` | List of output layer names.<br /><br />`Type: Array of String`<br />`Default: []`<br />`Note: Mutually exclusive with tensors`<br />`Flags: readable/writable` |
| `model` | Path to the SNPE DLC model file.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable, construct` |
| `performance-profile` | Request a performance profile.<br /><br />`Type: Enum`<br />`Default: DEFAULT_PROP_PERF_PROFILE`<br />`Range:`<br />    `(0): default - Default performance`<br />    `(1): balanced - Balanced performance and power`<br />    `(2): high-performance - Maximum performance`<br />    `(3): power-saver - Lower power usage`<br />    `(4): system-settings - System defined behavior`<br />    `(5): sustained-high-performance - Sustained performance mode`<br />    `(6): burst - Short bursts of high performance`<br />    `(7): low-power-saver - Aggressive power saving`<br />    `(8): high-power-saver - Moderate power saving`<br />    `(9): low-balanced - Lower balanced mode`<br />`Flags: readable/writable` |
| `priority` | Execution priority hint for SNPE runtime.<br /><br />`Type: Enum`<br />`Default: DEFAULT_PROP_EXEC_PRIORITY`<br />`Range:`<br />    `(0): normal - Default priority`<br />    `(1): high - Higher priority execution`<br />    `(2): low - Lower priority execution`<br />`Flags: readable/writable` |
| `profiling-level` | Set profiling level for runtime statistics.<br /><br />`Type: Enum`<br />`Default: DEFAULT_PROP_PROFILING_LEVEL`<br />`Range:`<br />    `(0): off - No profiling`<br />    `(1): basic - Minimal profiling`<br />    `(2): moderate - Medium level profiling`<br />    `(3): detailed - Full profiling`<br />`Flags: readable/writable` |
| `tensors` | List of output tensor names. Outputs follow defined order.<br /><br />`Type: Array of String`<br />`Default: []`<br />`Note: Mutually exclusive with layers`<br />`Flags: readable/writable` |

<Note>Layers vs tensors: Set only one of these. If your model exposes named output tensors, prefer tensors for precise ordering. If both are set sequentially, the last one written takes effect (the other is cleared).</Note>

## Input and Output Behavior

### Input Tensors

qtimlsnpe exposes a single sink pad, but it supports both single-input and batch-input models. For batch-input models, all required tensors are delivered through the same sink pad as a tensor set in a single gstbuffer.

Input tensors must be fully prepared before they reach qtimlsnpe. Expected tensor layout, shape, data type, and batch size are determined by:

* the SNPE DLC model input signature
* caps negotiation with upstream elements

Typical upstream elements include:

* qtimlvconverter for scaling, color conversion, normalization, and quantization (if required).

qtimlsnpe does not modify, reshape, batch, or reinterpret incoming tensors. It maps input tensor blocks into SNPE user buffers and passes them to the SNPE runtime as received.

### Output Tensors

qtimlsnpe exposes a single source pad and produces output tensors that follow the model's declared output signature. This single-pad design does not limit the element to a single output. Models with batch output tensors are fully supported, and all outputs are emitted together on the source pad.

Supported output behavior includes:

* single-tensor and batch-tensor outputs
* arbitrary tensor shapes and ranks, including batch and depth dimensions.
* both quantized and floating-point tensor types
* selective emission of output tensors using the layers or tensors property
* FLOAT32 dequantization: if the model's native output type is not FLOAT32, output caps will include a type list \[FLOAT32, native] to enable downstream negotiation for dequantization without changing the model artifact

The generated output tensors are intended for downstream post-processing stages, which are responsible for decoding model-specific results such as classification outputs, detection results, segmentation masks, landmark data, and other structured inference outputs.

## Delegates

A SNPE **delegate** defines the execution hardware used to run a model. Backends allow `qtimlsnpe` to offload inference from the default CPU interpreter to an optimized hardware accelerator, such as NPU, GPU, AIP.

`qtimlsnpe` supports multiple backend options. The backend is selected through the `backend` property by specifying the path to the corresponding shared library.

### DSP

Runs the model on the AI accelerator (NPU).

* **Use case:** Preferred backend where available. Best performance and power efficiency for quantized models.

### GPU

Runs supported operations through the snpe GPU backend.

* **Use case**: Floating-point models and workloads that benefit from GPU parallelism.

### AIP

Runs supported operations through the snpe AIP backend.

**Use case**:
Hybrid acceleration combining DSP + CPU + other HW blocks

Best for complex models with mixed operator support where pure DSP may fall back frequently

Useful when targeting maximum throughput with balanced power efficiency

Recommended for production pipelines where model partitioning across accelerators is beneficial

### CPU

Runs the model on the default snpe CPU backend.

**Use case**:

Fallback backend when other accelerators (DSP/GPU/AIP) are not available or unsupported

Ideal for debugging, validation, and functional correctness testing

Useful for small models or low-throughput workloads

Works with all model types (quantized + floating point) without operator support limitations

Preferred when deterministic performance and ease of deployment matter more than efficiency

## Profiling Level

Enables SNPE diagnostics collection. Available levels: `off, basic, detailed, moderate`

### Runtime Memory Behavior and GAP Handling

`qtimlsnpe` operates within the memory model of the snpe runtime. The element uses DMA buffers via `GstMLBufferPool` to minimize memory copies and maintain zero-copy transport where possible.

### SNPE Memory Model

SNPE uses runtime-managed memory to allocate:

* input tensors
* intermediate activation tensors
* output tensors

The element discovers input/output tensor metadata(count,shape,type) at model load time and configures buffer pools accordingly.

### GAP Buffer Handling

`qtimlsnpe` is GAP-aware and correctly handles input buffers marked with `GST_BUFFER_FLAG_GAP`.

When a GAP buffer is received, the element skips inference and forwards the buffer downstream. This preserves timing and synchronization while explicitly indicating that no valid inference input is available for that timestamp.

GAP buffers commonly appear in conditional AI pipelines, such as cascaded workflows where later inference stages run only when earlier stages produce valid regions of interest.

## Uses

### Single-Stage AI Inference on Live Camera Stream (HTP):

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX model | [Qualcomm AI hub model](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.dlc` |
    | YOLO labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Input video | [Input Video](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/) | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
      scp yolox_w8a8.dlc <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4 <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolox_w8a8.dlc
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! queue ! tee name=split \
    split. ! queue ! qtivcomposer name=mixer sink_1::dimensions="<1920,1080>" ! queue ! waylandsink fullscreen=true sync=true \
    split. ! queue ! video/x-raw,format=NV12 ! qtimlvconverter ! queue ! qtimlsnpe delegate=dsp tensors="<boxes,scores,class_idx>" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME ! queue ! qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! video/x-raw ! queue ! mixer.
    ```
  </Step>
</Steps>

### Single-Stage AI Inference on Live Camera Stream (GPU):

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Inception model | [Qualcomm AI Hub model](https://aihub.qualcomm.com/iot/models/inception_v3) | `inception_v3_float.dlc` |
    | MobileNet labels | <a href="../labels/mobilenet.json">mobilenet.json</a> | `mobilenet.json` |
    | Input video | [Input Video](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/) | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
      scp inception_v3_float.dlc <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp mobilenet.json <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4 <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=inception_v3_float.dlc
    export LABELS_NAME=mobilenet.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! queue ! tee name=split \
    split. ! queue ! qtivcomposer name=mixer sink_1::dimensions="<1920,1080>" ! queue ! waylandsink fullscreen=true sync=true \
    split. ! queue ! video/x-raw,format=NV12 ! qtimlvconverter ! queue ! qtimlsnpe delegate=gpu tensors="<class_logits>" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME ! queue ! qtimlpostprocess results=1 module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! video/x-raw ! queue ! mixer.
    ```
  </Step>
</Steps>


# qtimltflite
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtimltflite

TensorFlow Lite inference element for GStreamer AI and multimedia pipelines.

# Overview

**qtimltflite** is a GStreamer inference element that executes **TensorFlow Lite** models as part of AI and multimedia pipelines. The element operates entirely in **tensor mode:** it accepts input tensors on its sink pad and produces output tensors on its source pad according to the model’s input and output specifications.

The element is limited to **model execution**. It does not perform preprocessing, tensor reshaping, batching, layout conversion, or model-specific post-processing. These functions are expected to be handled by adjacent elements in the pipeline. As a result, upstream elements must provide tensors that already match the model requirements, and downstream elements must interpret the output tensors produced by inference.

**qtimltflite** supports multiple TensorFlow Lite execution backends through delegates, including **CPU, GPU, and external delegate** configurations. This allows the same pipeline structure to be deployed across different hardware targets and optimized for different performance, latency, and power requirements. The element is intended for real-time and embedded AI pipelines where inference is one stage in a larger modular processing flow.

### Key Responsibilities

**qtimltflite** is responsible for:

* Loading and executing a TensorFlow Lite model
* Accepting preformatted input tensors from upstream elements
* Producing output tensors that match the model output signature
* Negotiating tensor data types and dimensions with adjacent elements
* Propagating tensor metadata required by downstream elements

In practice, **qtimltflite** serves as the inference stage in the pipeline, while tensor preparation and result interpretation are handled externally.

<img alt="" />

## Example Pipeline

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp yolox_w8a8.tflite           <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolox_w8a8.tflite
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
    t. ! queue ! qtimlvconverter ! queue ! \
    qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
    qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! text/x-raw ! queue ! obj_mux.
    ```
  </Step>
</Steps>

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />[GstBaseTransform](https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c)<br />
            <Icon icon="arrow-turn-down-right" />qtimltflite

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `neural-network/tensors` | `format: { INT8, UINT8, INT16, UINT16, INT32, UINT32, INT64, UINT64, FLOAT16, FLOAT32 }` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `neural-network/tensors` | `format: { INT8, UINT8, INT16, UINT16, INT32, UINT32, INT64, UINT64, FLOAT16, FLOAT32 }` |
| Availability: *Always* | |
| Direction: *source* | |

# Element Properties

| Property | Description |
| - | - |
| `model` | Path to the TensorFlow Lite model file. This property is required and must reference a valid `.tflite` model.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `delegate` | Selects the execution backend used for inference.<br /><br />`Type: Enum `<br />`Default: 0, "none"`<br />`Range:`<br />    `(0): none - Default TensorFlow Lite CPU execution`<br />    `(5): gpu - TensorFlow Lite GPU delegate`<br />    `(6): xnnpack - Optimized CPU execution through XNNPACK`<br />    `(7): external - External delegate loaded through external-delegate-path and external-delegate-options`<br />`Flags: readable/writable` |
| `external-delegate-path` | Absolute path to the external delegate shared library. Used only when `delegate=external`.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `external-delegate-options` | Delegate-specific initialization options passed to the external delegate. Used only when `delegate=external`.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `priority` | Selects the execution preference for supported delegates, typically when there is a trade-off between latency and precision.<br /><br />`Type: Enum `<br />`Default: 0, "min-latency"`<br />`Range:`<br />    `(0): min-latency - Reduce latency at the cost of precision`<br />    `(1): max-precision - Increase precision at the cost of higher latency`<br />`Flags: readable/writable` |
| `threads` | Number of threads assigned to the TensorFlow Lite interpreter. Primarily affects CPU-based execution including XNNPACK.<br /><br />`Type: Unsigned Integer`<br />`Default: 1`<br />`Range: 1 - 4`<br />`Flags: readable/writable` |

# Input and Output Behavior

### Input Tensors

**qtimltflite** exposes a single sink pad, but it supports both single-input and multi-input models. For multi-input models, all required tensors are delivered through the same sink pad as a tensor set.

Input tensors must be fully prepared before they reach `qtimltflite`. Expected tensor layout, shape, data type, and batch size are determined by:

* the TensorFlow Lite model input signature
* caps negotiation with upstream elements

Typical upstream elements include:

* [`qtimlvconverter`](/plugin-reference/qtimlvconverter) for scaling, color conversion, normalization, and quantization
* [`qtibatch`](/plugin-reference/qtibatch) for batch construction

`qtimltflite` does not modify, reshape, batch, or reinterpret incoming tensors. It passes them to the TensorFlow Lite runtime as received.

### Output Tensors

`qtimltflite` exposes a single source pad and produces output tensors according to the model output signature. The single source pad does not limit the element to a single tensor. Models with multiple output tensors are fully supported, and all outputs are emitted together on the same pad.

The element supports:

* single-output and multi-output models
* arbitrary tensor ranks, including batch and depth dimensions
* quantized and floating-point outputs

Output tensors are typically consumed by downstream post-processing elements, which decode model-specific results such as classification scores, detection boxes, segmentation masks, landmarks, or other structured outputs.

# Quantization and Dequantization

`qtimltflite` can optionally dequantize quantized output tensors, such as `UINT8` or `INT8`, into `FLOAT32`. This conversion uses the quantization parameters stored in the TensorFlow Lite model metadata.

### Conditional Output Dequantization

Dequantization is performed only when the downstream path requires `FLOAT32` tensors. In practice, this is enabled when downstream caps negotiation indicates that floating-point output is needed.

When dequantization is applied, `qtimltflite`:

* reads the tensor `scale`
* reads the tensor `zero_point`
* applies the standard TensorFlow Lite dequantization formula:

```text theme={null}
output_float = scale × (quantized_value - zero_point)
```

* produces `FLOAT32` tensors for downstream processing

### When Dequantization Is Skipped

Dequantization is not performed when:

* downstream elements accept only quantized tensor types
* no downstream element negotiates `FLOAT32`
* the model output tensor does not contain valid quantization metadata

In these cases, the output tensor is forwarded in its original quantized representation.

This behavior allows the same downstream processing path to support both quantized and floating-point models where applicable, while avoiding unnecessary conversion.

# Supported Data Types

`qtimltflite` supports the tensor data types provided by the TensorFlow Lite runtime and the selected execution backend, subject to caps negotiation with adjacent elements.

Commonly used data types include:

* `UINT8`
* `INT8`
* `INT32`
* `FLOAT16`
* `FLOAT32`

The element does not impose additional data-type restrictions beyond those required by the runtime, the selected delegate, and negotiated pipeline caps.

# Batch and Depth Model Support

`qtimltflite` supports models with batch and multi-dimensional tensor inputs and outputs, including tensors with explicit batch and depth dimensions.

Examples include:

* batched tensors: `N × H × W × C`
* multi-dimensional tensors: `N × D × H × W × C`

The element treats these dimensions transparently and passes tensors to TensorFlow Lite according to the negotiated shape. It does not construct batches, reshape tensors, or reinterpret tensor dimensions internally.

Batch construction must be handled by upstream elements such as [`qtibatch`](/plugin-reference/qtibatch).

This behavior keeps inference predictable across single-frame, batched, and higher-dimensional workflows.

# Delegates

A TensorFlow Lite **delegate** defines the execution backend used to run a model. Delegates allow qtimltflite to offload inference from the default TensorFlow Lite CPU interpreter to an optimized backend, such as GPU, an optimized CPU runtime, or NPU.

`qtimltflite` supports multiple delegate options. The delegate is selected through the `delegate` property and controls how TensorFlow Lite dispatches model operations during inference.

### Built-in Delegate Options

### none

Runs the model on the default TensorFlow Lite CPU interpreter.

* **Backend**: CPU
* **Use case**: reference execution, debugging, or systems without acceleration

### gpu

Runs supported operations through the TensorFlow Lite GPU delegate.

* **Backend**: GPU
* **Use case**: floating-point models/workloads that benefit from GPU parallelism

### xnnpack

Runs inference through the XNNPACK optimized CPU backend.

* **Backend**: Optimized CPU
* **Use case**: improved CPU performance for supported floating-point and quantized models

### External Delegate Support

External delegate is used to accelerate models on Qualcomm’s NPU.

When an external delegate is selected, `qtimltflite` loads and configures the delegate at runtime using the following properties:

* `external-delegate-path`
  * Path to the external delegate shared library.
* `external-delegate-options`
  * Delegate-specific initialization options passed to the external delegate.

After initialization, supported model operations are offloaded to the NPU backend implemented by the external delegate.

# Runtime Memory Behavior and GAP Handling

`qtimltflite` operates within the memory model of the TensorFlow Lite runtime. Although the surrounding pipeline may use zero-copy transport for tensor buffers, TensorFlow Lite execution requires input and output tensors to reside in runtime-managed memory.

### TensorFlow Lite Memory Model

TensorFlow Lite uses an internal memory arena to allocate:

* input tensors
* intermediate activation tensors
* output tensors

### GAP Buffer Handling

`qtimltflite` is GAP-aware and correctly handles input buffers marked with `GST_BUFFER_FLAG_GAP`.

When a GAP buffer is received, the element skips inference and forwards the buffer downstream. This preserves timing and synchronization while explicitly indicating that no valid inference input is available for that timestamp.

GAP buffers commonly appear in conditional AI pipelines, such as cascaded workflows where later inference stages run only when earlier stages produce valid regions of interest.

# Usage

### Single-Stage AI Inference on Live Camera Stream

This example demonstrates real-time inference on a live camera stream using a single instance of `qtimltflite`. Inference results are attached to each `GstBuffer` as `MLMeta`, allowing downstream elements to access synchronized metadata directly from the frame. An overlay stage then uses this metadata to render annotations such as bounding boxes, labels, or key-points before display or further processing.

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"
      scp yolox_w8a8.tflite    <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json          <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolox_w8a8.tflite
    export LABELS_NAME=yolov8.json
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash Run the pipeline theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! \
    tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
    t. ! queue ! qtimlvconverter ! queue ! \
    qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
    qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! obj_mux.
    ```
  </Step>
</Steps>

<img alt="" />

### Two-Stage Daisy Chain AI Inference on Live Camera Stream

This example demonstrates a two-stage TensorFlow Lite inference workflow using two qtimltflite instances. The first model operates on full video frames after preprocessing by a [`qtimlvconverter`](/plugin-reference/qtimlvconverter) configured for full-frame input. Inference results, such as detected objects, are attached to the corresponding video buffer and propagated downstream. The second model runs once for each object detected by the first stage. A second [`qtimlvconverter`](/plugin-reference/qtimlvconverter), configured for ROI-based processing, crops each detected region from the input frame and prepares it as input for the second `qtimltflite` instance.

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Detection model (YOLOX) | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Classification model (InceptionV3) | [Qualcomm AI Hub — InceptionV3](https://aihub.qualcomm.com/iot/models/inception_v3) | `inception_v3_w8a8.tflite` |
    | Classification labels | <a href="../labels/mobilenet.json">mobilenet.json</a> | `mobilenet.json` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"
      scp yolox_w8a8.tflite            <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp inception_v3_w8a8.tflite     <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp mobilenet.json               <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME_1=yolox_w8a8.tflite
    export LABELS_NAME_1=yolov8.json
    export MODEL_NAME_2=inception_v3_w8a8.tflite
    export LABELS_NAME_2=mobilenet.json
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash Run the pipeline theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
      qtimlvconverter name=stage_01_preproc \
      qtimltflite name=stage_01_inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
      external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" \
      model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_1 \
      qtimlpostprocess name=stage_01_postproc module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_1 \
      settings="{\"confidence\": 51.0}" \
      qtimetamux name=metamux_1 \
      qtivoverlay name=main_overlay \
      qtimlvconverter name=stage_02_preproc \
      qtimltflite name=stage_02_inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
      external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" \
      model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_2 \
      qtimlpostprocess name=stage_02_postproc module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_2 \
      settings="{\"confidence\": 51.0}" \
      qtimetamux name=metamux_2 \
      qtivoverlay name=cls_overlay \
      qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! \
      tee name=t_split_1 \
      t_split_1. ! queue ! metamux_1. \
      t_split_1. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! stage_01_inference. stage_01_inference. ! queue ! \
      stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! metamux_1. \
      metamux_1. ! queue ! tee name=t_split_2 \
      t_split_2. ! queue ! metamux_2. \
      t_split_2. ! queue ! stage_02_preproc. stage_02_preproc. ! queue ! stage_02_inference. stage_02_inference. ! queue ! \
      stage_02_postproc. stage_02_postproc. ! text/x-raw ! queue ! metamux_2. \
      metamux_2. ! queue ! cls_overlay. cls_overlay. ! queue ! waylandsink sync=true fullscreen=true
    ```
  </Step>
</Steps>

### Four-Stage Daisy Chain AI Inference on Live Camera Stream

The example demonstrates a multi-stage inference workflow for live Hand-Gesture recognition use case built with four TensorFlow Lite models executed through four qtimltflite instances.

* **Stage 1** performs full-frame palm detection. The input video frame is preprocessed, passed through inference, and post-processed to generate metadata describing the detected palm.
* **Stage 2** performs per-ROI hand landmark inference. Regions detected in the first stage are cropped and batched for processing. Two post-processing paths are used: one generates visualization metadata, while the other reformats the output tensors for the next stage.
* **Stage 3** chains two models in tensor-only mode, without additional preprocessing or post-processing. The first model consumes multiple tensors produced by the previous stage, and its output is passed directly to the second model.
* **Stage 4** performs final gesture classification and converts the model output into metadata for downstream use.

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    Download the gesture recognizer models from Google MediaPipe:

    ```bash theme={null}
    # Download the gesture recognizer task
    wget https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/latest/gesture_recognizer.task 

    # Extract the .task file
    unzip gesture_recognizer.task
    # This gives you two more task files: hand_landmarker.task and hand_gesture_recognizer.task

    # Extract hand_landmarker.task
    unzip hand_landmarker.task
    # This gives you hand_detector.tflite and hand_landmarks_detector.tflite

    # Extract hand_gesture_recognizer.task
    unzip hand_gesture_recognizer.task
    # This gives you gesture_embedder.tflite and canned_gesture_classifier.tflite
    ```

    <Note>These are FLOAT precision models.</Note>

    | File | Download | Save as |
    | - | - | - |
    | Palm detection model | See download steps above | `hand_detector.tflite` |
    | Palm detection labels | <a href="../labels/palmd_labels.json">palmd\_labels.json</a> | `palmd_labels.json` |
    | Palm detection settings | <a href="../labels/palmd_settings.json">palmd\_settings.json</a> | `palmd_settings.json` |
    | Hand landmark model | See download steps above | `hand_landmarks_detector.tflite` |
    | Hand landmark labels | <a href="../labels/hlandmark_labels.json">hlandmark\_labels.json</a> | `hlandmark_labels.json` |
    | Hand landmark settings | <a href="../labels/hlandmark_settings.json">hlandmark\_settings.json</a> | `hlandmark_settings.json` |
    | Gesture embedder model | See download steps above | `gesture_embedder.tflite` |
    | Gesture classifier model | See download steps above | `canned_gesture_classifier.tflite` |
    | Gesture labels | <a href="../labels/gesture_labels.json">gesture\_labels.json</a> | `gesture_labels.json` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"
      scp hand_detector.tflite               <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp palmd_labels.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp palmd_settings.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp hand_landmarks_detector.tflite     <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp hlandmark_labels.json              <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp hlandmark_settings.json            <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp gesture_embedder.tflite            <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp canned_gesture_classifier.tflite   <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp gesture_labels.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME_1=hand_detector.tflite
    export LABELS_NAME_1=palmd_labels.json
    export LABELS_NAME_2=palmd_settings.json
    export MODEL_NAME_2=hand_landmarks_detector.tflite
    export LABELS_NAME_3=hlandmark_labels.json
    export LABELS_NAME_4=hlandmark_settings.json
    export MODEL_NAME_3=gesture_embedder.tflite
    export MODEL_NAME_4=canned_gesture_classifier.tflite
    export LABELS_NAME_5=gesture_labels.json
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash Run the pipeline theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qtimlvconverter name=stage_01_preproc \
    qtimltflite name=stage_01_inference delegate=gpu model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_1 \
    qtimlpostprocess name=stage_01_postproc results=1 module=palmd labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_1 settings=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_2 \
    qtimlvconverter name=stage_02_preproc mode=roi-batch-non-cumulative \
    qtimltflite name=stage_02_inference delegate=gpu model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_2 \
    qtimlpostprocess name=stage_02_1_postproc results=6 module=hlandmark labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_3 settings=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_4 \
    qtimlpostprocess name=stage_02_2_postproc results=6 module=tensor \
    qtimltflite name=stage_03_1_inference delegate=gpu model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_3 \
    qtimltflite name=stage_03_2_inference delegate=gpu model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_4 \
    qtimlpostprocess name=stage_03_postproc results=8 module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_5 \
    qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! tee name=t_split_1 \
    t_split_1. ! queue ! qtimetamux name=metamux_1 ! queue ! qtimetatransform module=roi-palmd ! queue ! tee name=t_split_2 \
    t_split_1. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! stage_01_inference. stage_01_inference. ! queue ! stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! metamux_1. \
    t_split_2. ! queue ! qtimetamux name=metamux_2 ! queue ! qtivoverlay ! waylandsink fullscreen=true sync=true \
    t_split_2. ! queue ! stage_02_preproc. stage_02_preproc. ! queue ! stage_02_inference. stage_02_inference. ! queue ! tee name=t_split_3 \
    t_split_3. ! queue ! stage_02_1_postproc. stage_02_1_postproc. ! text/x-raw ! metamux_2. \
    t_split_3. ! queue ! stage_02_2_postproc. stage_02_2_postproc. ! queue ! stage_03_1_inference. stage_03_1_inference. ! stage_03_2_inference. stage_03_2_inference. ! stage_03_postproc. stage_03_postproc. ! text/x-raw ! metamux_2.
    ```
  </Step>
</Steps>


# qtimlvconverter
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtimlvconverter

AI pre-processing plugin

# Overview

The **qtimlvconverter** element is an essential component of the AI pipeline, responsible for AI preprocessing—specifically, preparing video frame data for neural network inference. It efficiently processes incoming buffers in YUV or RGB formats and converts them into tensors that are compatible with machine learning models.

<img alt="mlvconverter" />

This transformation encompasses several critical operations:

* **Cropping**: Selects a specific region of the input frame to focus on. This uses provided ROI metadata to determine the crop region for each frame.
* **Rescaling**: Adjusts the spatial dimensions of input frames to match the expected tensor size for the target model to ensure compatibility and consistent performance.
* **Format Conversion**: Translates pixel data between supported formats (e.g., YUV to RGB) to meet the input requirements of the target model.
* **Batching**: Aggregates multiple frames or images into batches to optimize inference throughput and leverage parallel processing capabilities.
* **Normalization**: Applies pixel value scaling and normalization techniques (such as mean subtraction and standard deviation division) to standardize input data for improved model accuracy.
* **Temporal Batching**: Prepares input tensors with shape NDHWC by stacking multiple temporally consecutive frames along the depth dimension. This enables compatibility with video-based models, which require temporal context across frames.

To perform efficient transformation the **qtimlvconverter** runs on GPU and executes these steps in a chain. It resizes the entire input frame or the cropped region to fit the target tensor dimensions while preserving the full field of view. In this step, only the resolution changes, while the format and data range remain the same. By default it maintains the original aspect ratio, ensuring that geometric shapes of the objects remain undistorted in the output. If the input frame’s aspect ratio differs from that of the target tensor, the resized frame is positioned in the top-left corner, and any remaining area is filled with a black background. This method ensures that the entire tensor is populated, preserves the integrity of the original image content, and avoids cropping or distortion.  This makes it well-suited for models that require consistent spatial representation.

The plugin also intelligently analyzes the dimensional order, count, and size to automatically select the appropriate tensor layout. Output tensors may be produced in either interleaved (*NHWC*) or planar (*NCHW*) formats, supporting *RGBA*, *RGB*, or *GRAYSCALE* pixel arrangements. This selection is performed seamlessly by the plugin, eliminating the need for manual configuration.

In addition to conventional four-dimensional tensor outputs, the plugin also supports five-dimensional tensors where the fifth dimension represents depth (*NDHWC*). The detection and application of this advanced configuration are managed automatically, further enhancing flexibility for complex inference models.

To accommodate models trained with alternative pixel arrangements, the default tensor format - typically *RGBA*, *RGB*, or *GRAYSCALE* - can be modified via the subpixel-layout property. This feature enables support for formats such as *BGRA* or *BGR*, ensuring compatibility with a diverse range of neural network architectures and training methodologies.

Through this automated and highly configurable approach, the plugin streamlines the transformation of video data into neural network-ready tensors, facilitating robust and efficient inference across varied deployment scenarios.
<Tip>More details about tensor options available at the end of this page</Tip>

## Example Pipeline

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp yolox_w8a8.tflite           <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolox_w8a8.tflite
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
    t. ! queue ! qtimlvconverter ! queue ! \
    qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
    external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
    qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME \
    settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! obj_mux.
    ```
  </Step>
</Steps>

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />[GstBaseTransform](https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c)<br />
            <Icon icon="arrow-turn-down-right" />qtimlvconverter

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { RGBA, BGRA, ABGR, ARGB, RGBx, BGRx, xRGB, xBGR, BGR, RGB, GRAY8, NV12, NV21, YUY2, UYVY, NV12_Q08C }` <br /> `width: [1, 32767]` <br /> `height: [1, 32767]` <br /> `framerate: [0/1, 255/1]` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `neural-network/tensors` | `format: { INT8, UINT8, INT16, UINT16, INT32, UINT32, INT64, UINT64, FLOAT16, FLOAT32 }` <br /> `width: [1, 32767]` <br /> `height: [1, 32767]` <br /> `framerate: [0/1, 255/1]` |
| Availability: *Always* | |
| Direction: *source* | |

# Element Properties

| Property | Description |
| - | - |
| `engine` | Engine backend used for the conversion operations.<br /><br />`Type: Enum `<br />`Default: 2, "gles"`<br />`Range:`<br />    `(0): none - No backend used`<br />    `(2): gles - Use OpenGLES based video converter`<br />    `(3): fev - Use FastCV based video converter`<br />    `(4): ocv - Use OpenCV based video converter`<br />`Flags: readable/writable` |
| `image-disposition` | Aspect ratio and placement of the image inside the output tensor.<br /><br />`Type: Enum `<br />`Default: 0, "top-left"`<br />`Range:`<br />    `(0): top-left - Preserve aspect ratio during resize  and place it in the top-left corner of the output tensor`<br />    `(1): centre - Preserve aspect ratio during resize and place it in the centre of the output tensor`<br />    `(2): stretch - Ignore aspect ratio and if required stretch it's AR in order to fit completely inside the output tensor`<br />    `(3): centre-crop - Ignore the aspect ratio and if required crop the source around its center to fit completely inside the output tensor` <br /> `Flags: readable/writable` |
| `mean` | Channel mean subtraction values for FLOAT tensors such as `{R, G, B}`, `{R, G, B, A}`, or `{<G>}`.<br /><br />`Type: GstValueArray of type gdouble`<br /> `Default: "<  >"` <br /> `Flags: readable/writable` |
| `mode` | Conversion mode.<br /><br />`Type: Enum `<br />`Default: 0, "image-batch-non-cumulative"`<br />`Range:`<br />    `(0): image-batch-non-cumulative - ROI metadata is ignored.Immediately process incoming buffers irrelevant of whether there are enough image memory blocks to fill the requested tensor batch size.`<br />    `(1): image-batch-cumulative - ROI metadata is ignored. Accumulate buffers until there are enough image memory blocks to fill the requested tensor batch size. Accumulation is interrupted early if a GAP buffer is received.`<br />    `(2): roi-batch-non-cumulative - Use only ROI metas to fill tensor batch size. Immediately process incoming buffers irrelevant of whether there are enough ROI metas to fill the requested tensor batch size. In case no ROI meta is present a GAP buffer will be produced.`<br />    `(3): roi-batch-cumulative - Use only ROI metas to fill tensor batch size. Accumulate buffers until there are enough ROI metas to fill the requested tensor batch size. Accumulation is interrupted early if a GAP buffer is received or if there are no ROI metas present inside the received buffer.`<br />`Flags: readable/writable` |
| `sigma` | Channel divisor values for FLOAT tensors such as `{R, G, B}`, `{R, G, B, A}`, or `{<G>}`.<br /><br />`Type: GstValueArray of type gdouble`<br /> `Default: "<  >"` <br /> `Flags: readable/writable` |
| `subpixel-layout` | Arrangement of the image pixels in the output tensor.<br /><br />`Type: Enum `<br />`Default: 0, "regular"`<br />`Range:`<br />    `(0): regular - RGB, RGBA, RGBx`<br />    `(1): reverse - BGR, BGRA, BGRx`<br />`Flags: readable/writable` |

# Pre Process Callback

Emitted once per frame when **external processing** is enabled, allowing the application to perform the conversion itself instead of a built-in engine. The handler reads the input video blits, produces the output tensor, and returns `TRUE` on success or `FALSE` to abort the pipeline.

External processing is engaged only when **both** conditions are met:

1. The `engine` property is set to `none`, so no built-in engine claims the conversion, and
2. A handler is connected to the `process` signal.

<Warning>If a handler is connected but `engine` is left at its default backend, the built-in engine performs the conversion and the handler is **not** invoked.</Warning>

| Parameter | Type | Description |
| - | - | - |
| *(instance)* | `qtimlvconverter` | The element emitting the signal. |
| `blits` | `GstVideoBlits` | Input buffers, each with its crop (source) and placement (destination) rectangles. |
| `sample` | `GstMLBundle` | Container holding the output ML buffer (tensor) and its ML info. |
| *(return)* | `gboolean` | `TRUE` on success, `FALSE` to abort the pipeline. |

`Flags: Run Last`

<Info>For the input/output structure reference and C/C++ and Python examples, see [Guidelines for Writing an External Preprocessing Callback](#guidelines-for-writing-an-external-preprocessing-callback) below.</Info>

# Image/Video Tensor Characteristics

An image or video tensor has several key characteristics that define how data is represented and processed in machine learning pipelines:

```bash theme={null}
🛈 Layouts
 N - Tensor batch size. How many frames can be processed at the same time by the model.
 D - Tensor depth. Represent the number of history frames from single source/stream.
 H - Tensor height in pixels.
 W - Tensor width in pixels.
 C - Number of pixel components(4 == RGBA/BGRA, 3 == RGB/BGR, 1 == GRAYSCALE).

```

**Tensor Dimensions / Descriptor**

This defines the shape and layout of the tensor. Common descriptors include:

* **HWC (Height, Width, Channels)**
  The single-image equivalent of **NHWC** — the same interleaved arrangement, but without a leading batch (N) dimension. Used by models that accept a three-dimensional input tensor for one image at a time.
  For example, in RGB:
  * Pixel 1: R, G, B
  * Pixel 2: R, G, B
  * and so on

<img alt="HWC" />

* **CHW (Channels, Height, Width)**
  The single-image equivalent of **NCHW** — the same planar arrangement, but without a leading batch (N) dimension.
  For example, in RGB:
  * First all Red values for the entire image
  * Then all Green values
  * Then all Blue values

<img alt="CHW" />

* **NHWC(Batch, Height, Width, Channels)**
  Channels are interleaved per pixel. Common in TensorFlow and QIM SDK because it aligns better with memory access patterns for certain hardware accelerators.
  For example, in RGB:
  * Pixel 1: R, G, B
  * Pixel 2: R, G, B
  * and so on

<img alt="NHWC" />

* **NCHW  (Batch, Channels, Height, Width)**:
  All pixels for one channel are stored contiguously. Preferred by many deep learning frameworks (e.g., PyTorch, Caffe) because it optimizes convolution operations on GPUs.
  For example, in RGB:
  * First all Red values for the entire image
  * Then all Green values
  * Then all Blue values

<img alt="NCHW" />

* **NDHWC (Batch,Depth,Height,Width,Channels)**
  This format extends the NHWC layout by adding a Depth (D) dimension, making it suitable for video sequences or volumetric data (e.g., temporal stacks for action recognition).
  Frames are stored sequentially, and within each frame, pixels are arranged in NHWC order (interleaved channels per pixel).
  For example:
  * Frame 1: R,G,B for each pixel
  * Frame 2: R,G,B for each pixel
  * and so on.

<img alt="NDHWC" />

* **NDCHW (Batch, Depth, Channels, Height, Width)**
  The planar counterpart of **NDHWC**. Frames are stored sequentially, and within each frame the channels are stored in planar (CHW) order rather than interleaved. Suitable for video/volumetric models that expect planar per-frame data.
  For example:
  * Frame 1: all R, then all G, then all B
  * Frame 2: all R, then all G, then all B
  * and so on.

<img alt="NDCHW" />

<Info>
  **The plugin detects the tensor layout automatically** from the number of dimensions and their sizes — no manual configuration is required.

  The number of dimensions selects the family of layouts:

  | Dimensions | Possible layouts |
  | - | - |
  | 3 | **HWC** / **CHW** |
  | 4 | **NHWC** / **NCHW** |
  | 5 | **NDHWC** / **NDCHW** |

  To pick between the **interleaved** and **planar** variant, the plugin looks at where the small "channels" dimension sits — an image has only a handful of channels (1, 3, or 4) but many pixels across its width and height:

  * If the small dimension comes **first** (with a large dimension last), the tensor is **planar** — **CHW**, **NCHW**, **NDCHW**.
  * Otherwise it is **interleaved** — **HWC**, **NHWC**, **NDHWC**.

  If the shape doesn't correspond to a sensible image — for example the resulting width or height would be smaller than 32 pixels — the layout is treated as **unknown** (see the note below).
</Info>

**NCHW** and **NHWC** formats contain the same data but **organize it differently**. The order of the letters indicates how the data is arranged within the tensor. Lets take as example **\[1,3,480,640]** and **\[1,480,640,3]** tensors.  In both cases the batch size is 1, width is 640, height is 480 and the number of channels is 3. Let's assume that image format is RGB. The key difference in these formats is how RGB pixels are arranged within the buffer.

**Data Format**

Specifies the type of values stored in the tensor like  **UINT8, INT8, UINT16, FLOAT32, FLOAT16**, etc

**Data Range**

Defines the expected value range of the tensor elements:

* 0-255 - Typical for raw **UINT8** image data
* -128-127 - Typical for raw **INT8** image data
* 0.0-1.0 - Common for **normalized floating-point** data for FLOAT16 and FLOAT32 tensors
* -1.0-1.0 - Used in some models that expect centered data, the user must explicitly configure normalization parameters(offset and scale)
* Custom ranges may be used depending on the model's training setup

**Color Format**

Indicates how color information is represented:

* RGB – Red, Green, Blue (standard format for most models)
* BGR – Blue, Green, Red (used by some OpenCV-based models)
* Grayscale – Single-channel format for monochrome images

**Supported Tensors by qtimlvconverter**

| Property | Description |
| - | - |
| Tensor Shape | HWC, CHW, NHWC, NCHW, NDHWC, NDCHW |
| Data Format | uint8, int8, float32, float16 |
| Data Range | any |
| Color Format | RGB, BGR, Grayscale |

<Note>**Unknown tensor layouts.** When the output tensor shape does not match any of the layouts above, the plugin cannot determine how to place the image data on its own. In this case the layout is treated as **unknown**, and the conversion is supported **only** through [external processing](../qimsdk-overview/preprocessing#external-preprocessing) — the application connects the `process` signal and populates the tensor itself. If the layout is unknown and no `process` handler is connected, caps negotiation fails.</Note>

# Image Placement Matters in Model Performance

When working with computer vision models, we often focus on the content of the image - but where the image is placed inside the input tensor can also make a big difference. By default, it maintains the original aspect ratio, ensuring that geometric shapes of the objects remain undistorted in the output. If the input frame’s aspect ratio differs from that of the target tensor, the resized frame is positioned in the top-left corner, and any remaining area is filled with a black background. This method ensures that the entire tensor is populated, preserves the integrity of the original image content, and avoids cropping or distortion, making it well-suited for models that require consistent spatial representation.

<img alt="NV12" />

However, some models perform better when the image is **centered or stretched** to fill entire tensor without padding. To support such use cases, **qtimlvconverter** provides three configurable placement modes:

* **top-left** (default) - Keeps the original aspect ratio of the image and places it in the **top-left corner** of the output tensor.
* **centre** - Also preserves the aspect ratio, but centers the image within the tensor. This is a common choice for models that expect the main object to be in the middle.
* **stretch** - Ignores the original aspect ratio and stretches the image to completely fill the tensor. This can introduce distortion but ensures full coverage, which some models require
* **centre-crop** - Ignore the source image AR (Aspect Ratio) and if required crop the source around its center to fit completely inside the output tensor.

<img alt="centre" />

**What if You Need to Crop Instead?**

In some cases, rather than resizing or repositioning the entire image, you may need to **crop a specific region** - for example, a random or targeted part of the input image - and feed only that portion to the model. This approach is useful when the model is trained to focus on localized features. While **image-disposition** helps with placement and aspect ratio handling, cropping is a separate preprocessing step that gives you more control over **which part** of the image the model sees.

To achieve this, you can insert a [qtivtransform](https://qualcomm-confluence.atlassian.net/wiki/spaces/VAISDK/pages/1669368059/qtivtransform) step before **qtimlvconverter**. It allows you to perform the necessary cropping operation on the input image before it reaches the converter. This gives you precise control over which part of the image is used.

<img alt="ml_preprocess_with_crop" />

Here's how each component works:

* **source** : This is your input stream.
* [qtivtransform](https://qualcomm-confluence.atlassian.net/wiki/spaces/VAISDK/pages/1669368059/qtivtransform): This stage allows you to apply transformations to the image, such as cropping, resizing, or rotating. In this context, you can define a **crop region** via properties passed directly to **qtivtransform**, enabling you to extract a specific part of the input image before it reaches the model.

```bash theme={null}
⚠️ Important: The output resolution of qtivtransform must be specified manually.
To avoid stretching or distortion, the aspect ratio of the output resolution
must match the aspect ratio of the crop window. If you're unsure what
resolution to choose, you can simply set the crop width and height as
the output resolution - this ensures a 1:1 mapping without scaling.
```

* **qtimlvconverter**: After cropping, this component prepares the image for inference. It handles scaling, color conversion, normalization and positioning based on the **image-disposition** property (e.g., **top-left, centre, stretch**)
* **sink** - This is the next element where the tensor is passed

By combining **input cropping via qtivtransform** with placement control via **image-disposition** in qtimlvconverter, you can achieve any desired transformation - whether it’s focusing on a specific region, preserving aspect ratio, or aligning the image within the tensor. This flexibility is especially valuable when adapting to different model requirements or optimizing inference performance

The image below demonstrates how a specific region of the input image can be converted into a tensor.

<img alt="image_disposition" />

This example showcases how to convert a predefined region. While [qtivtransform](https://qualcomm-confluence.atlassian.net/wiki/spaces/VAISDK/pages/1669368059/qtivtransform) supports runtime crop window updates, this solution is not always scalable.

In more advanced use cases, **qtimlvconverter** can perform cropping on its own, without relying on [qtivtransform](https://qualcomm-confluence.atlassian.net/wiki/spaces/VAISDK/pages/1669368059/qtivtransform). For instance, if you have **two models working sequentially** - a detection model followed by a pose estimation model - **qtimlvconverter** can automatically crop and generate a tensor for each bounding box detected by the first stage.

# Normalization

Normalization of pixel values is performed automatically, tailored to the negotiated tensor data type. For floating-point tensor types **(FLOAT16, FLOAT32)**, pixel values are normalized to the range (0,1). For signed and unsigned integer types (such as **INT8, UINT8, INT16, UINT16**), normalization is applied according to the full value range of each type - for example, INT8 is normalized to \[−128 , 127], UINT8 to \[0 , 255], and INT16 to \[−32,768 , 32,767].

Beyond this automatic normalization, the plugin provides further customization through the **mean** and **sigma** properties, enabling per-channel normalization using the following formula:

```bash theme={null}
🛈 Normalization

normalized_value = (<Pixel Channel Value> - mean​) * sigma
```

```bash theme={null}
🛈 Example

An INT16 RGB image with values R=2000, G=-344, B=0 and property values
mean="<1280.0, 1560.0, -1240.0>" and sigma="<0.75, 0.34, 0.02>" would
have the following channel transformations:
R=(2000 - 1280.0) x 0.75, G=(-324 - 1560.0) x 0.34, B=(0 + 1240.0) x 0.02.
```

This custom normalization is applied subsequently, after the initial range-based normalization, allowing precise adjustment of pixel values to meet the requirements of specific neural network models or training regimes. This dual-stage normalization approach ensures that tensor data is both type-appropriate and optimally conditioned for inference, supporting robust and accurate model performance across diverse deployment scenarios.

# Batching

Batching is a key optimization technique in AI inference pipelines, allowing multiple frames to be processed simultaneously. This improves throughput and enables parallel execution on hardware accelerators.

In the context of **qtimlvconverter**, incoming video buffers may originate from:

* A single source (e.g., one camera or video file), where each **GstBuffer** contains a single **GstMemory** block.
* A multiplexed stream (e.g., multiple cameras or video files), where each **GstBuffer** contains multiple **GstMemory** blocks. This configuration is enabled via the **qtibatch** plugin, which aggregates frames from different sources into a single batched buffer.

Batching is especially useful when working with models trained to process multiple inputs simultaneously (e.g., batch size = 4). It allows the inference engine to utilize GPU resources more efficiently and reduces per-frame processing overhead.

**How it Works:**

* Each **GstMemory** block in the buffer represents one frame
* \**qtimlvconverter* iterates over all memory blocks and applies preprocessing steps (cropping, resizing, format conversion, normalization) to each frame individually.
* The processed frames are then packed into a single tensor with shape **\[N, H, W, C] or \[N, C, H, W]**, depending on the selected layout.
* The batch size **N** is inferred from the number of memory blocks in the buffer.

**Benefits**:

* Parallel Inference: Enables models to process multiple frames simultaneously, improving throughput and latency.
* Efficient GPU Utilization: Reduces per-frame overhead and maximizes hardware acceleration.
* Flexible Input Handling: Supports both single-stream and multi-stream scenarios without requiring manual configuration.

**Tensor Layouts**:

Depending on the model and configuration, the output tensor may use:

* **NHWC:** Interleaved layout (e.g., \[4, 480, 640, 3] for batch size 4, RGB).
* **NCHW:** Planar layout (e.g., \[4, 3, 480, 640]), often used in CNNs.
* **NDHWC:** For models requiring temporal depth (e.g., \[1, 4, 480, 640, 3]), where D represents history frames.
* **NDCHW:** Temporal depth with planar channels (e.g., \[1, 4, 3, 480, 640]).
* **HWC / CHW:** Single-image layouts (no batch dimension), e.g. \[480, 640, 3] or \[3, 480, 640], used by models that accept one frame at a time.

**qtimlvconverter** automatically selects the appropriate layout based on the model’s input signature and the number of frames in the batch.

# Temporal Batching

Temporal batching extends the concept of standard batching by grouping multiple frames from the same stream over time into a single tensor. This approach is essential for models that require temporal context, such as:

* Action recognition (e.g., detecting gestures or activities across consecutive frames).
* Object tracking (e.g., maintaining identity across frames).
* 3D CNNs or RNN-based vision models that process sequences rather than single images.

**How It Works:**

* Instead of aggregating frames from different sources, temporal batching collects D consecutive frames from the same stream.
* These frames are packed into a tensor with shape:
  * NDHWC: \[N, D, H, W, C] — interleaved channels, or NDCHW: \[N, D, C, H, W] — planar channels
    * N = batch size(number of sequences processed together)
    * D = depth(number of frames per sequence)
    * H,W = spatial dimensions
    * C = channels(e.g.,RGB)
* qtimlvconverter automatically handles:
  * Frame accumulation based on the configured depth (D).
  * Preprocessing for each frame (resize, normalization, format conversion).
  * Tensor assembly in the correct layout for the model.

**Benefits:**

* Preserves temporal continuity, enabling models to learn motion patterns.
* Improves inference accuracy for tasks that depend on frame-to-frame relationships.
* Fully utilizes GPU resources by processing sequences in parallel.

**Example:**

* For a model expecting 4-frame sequences:
  * Tensor shape: \[1, 4, 480, 640, 3] (batch size = 1, depth = 4, RGB).
  * Frames are normalized and resized individually, then stacked in temporal order.

# Multi-Stage Inference Pipelines with ROI-Based Processing

In advanced AI pipelines, inference is often performed in multiple stages - each consisting of preprocessing, inference, and postprocessing. These stages may operate on the full image or on specific regions of interest (ROIs) identified by earlier stages.

To support this, **qtimlvconverter** can process buffers containing multiple **GstVideoRegionOfInterestMeta** entries. These ROIs are typically generated by upstream ML components or custom plugins and represent targeted regions within the input frame that require further analysis.
By default, **qtimlvconverter** processes the entire image, ignoring any ROI metadata. However, when configured via the **mode** property, it can switch to ROI-based processing, enabling selective transformation of only the regions marked for further inference.

**ROI Processing Modes**

The **mode** property defines how input regions are handled and batched before being converted into tensors. There are two main categories:

* Image batch mode:
  * **image-batch-non-cumulative:** Processes each buffer immediately, regardless of batch size.
  * **image-batch-cumulative:** Accumulates full-frame inputs until batch size is met.
* ROI Batch Modes:
  * **roi-batch-non-cumulative:** Processes ROI metadata immediately, discarding excess entries.
  * **roi-batch-cumulative:** Accumulates ROI entries until batch size is met.

These modes allow fine-grained control over latency and resource utilization, depending on the model’s batch size (N) and the expected input rate.

**Submode Behaviour**

* **Non-Cumulative**:
  In non-cumulative submode, incoming buffers are processed immediately upon receipt, regardless of whether the number of image memory blocks or ROI metadata entries meets the model’s specified tensor batch size (N). This approach is recommended when the number of multiplexed streams and/or ROI metadata is not expected to exceed the batch size (N) of the model. Any ROI metadata or multiplexed GstMemory blocks exceeding the batch size (N) are discarded. A potential drawback of this submode is that if the batch is not fully populated - for example, if the batch size is set to N=4 but only three positions are filled - the **inference** plugin will still process the entire batch, resulting in resource inefficiency due to unutilized positions. However, the primary advantage of non-cumulative submode is the **elimination of processing latency:** buffers are not held back to accumulate additional inputs to fulfill the batch size requirement. This ensures prompt processing and rapid generation of prediction results, making it suitable for real-time applications where minimal delay is critical.
* **Cumulative:**
  In cumulative submode, incoming buffers are aggregated until the number of ROI metadata entries and/or multiplexed **GstMemory** blocks meets the model’s required tensor batch size (N). Accumulation may be interrupted prematurely if a GAP buffer is received, or, in ROI batch modes, if the incoming buffer contains no ROI metadata. This accumulation strategy introduces variable latency into the processing pipeline, influenced by factors such as frame production intervals and the number of ROIs or multiplexed images received per buffer. The principal advantage of cumulative submode is that all incoming ROI metadata and multiplexed **GstMemory** blocks are processed, ensuring that none are discarded due to batch size constraints. This mode is particularly recommended when the model’s batch size (N) exceeds one and inference processing time is sufficiently low, allowing for efficient utilization of resources and comprehensive processing of available input regions.

**Non-Cumulative:**

<img alt="non-cumulative" />

**Cumulative:**

<img alt="cumulative" />

# Preprocessing Metadata

In some cases, post-processing requires information about how each input frame was pre-processed - such as its placement within the input tensor, its dimensions, or which frames in a batch are valid. To support this, **qtimlvconverter** attaches metadata describing the preprocessing details to each tensor using **GstProtectionMeta**. The inference plugin then propagates this metadata from its input to its output, enabling the post-processing plugin to access the necessary information about how each frame was handled during preprocessing.

The **GstProtectionMeta** includes the following fields:

* **Input Tensor Dimensions:**
  * input-tensor-width \[G\_TYPE\_UINT] - Specifies the width (in pixels) of the tensor that the frame was mapped into. This is the final width after any resizing operations performed by qtimlvconverter.
  * input-tensor-height \[G\_TYPE\_UINT] - Specifies the height (in pixels) of the tensor after preprocessing. This value reflects the target model’s expected input dimensions.
* **Region Occupied by Actual Data within the Input Tensor:**
  * input-region-x \[G\_TYPE\_INT] - The X-coordinate (horizontal offset) of the region within the tensor where the actual image data is placed. Useful for determining padding or positioning when aspect ratio is preserved.
  * input-region-y \[G\_TYPE\_INT] - The Y-coordinate (vertical offset) of the region within the tensor where the image data starts.
  * input-region-width \[G\_TYPE\_INT] - The width of the actual image content inside the tensor. This may differ from input-tensor-width if padding was applied.
  * input-region-height \[G\_TYPE\_INT] - The height of the actual image content inside the tensor. Indicates how much of the tensor is occupied by real image data.
* **Batch Sequence Information:**
  * sequence-index \[G\_TYPE\_UINT] - The index of this entry within the current batch. For example, in a batch of size 4, valid values are 0–3.
  * sequence-num-entries \[G\_TYPE\_UINT] - The total number of entries in the batch. This helps post-processing plugins understand the batch context.
* **Timestamp of the buffer:**
  * timestamp \[G\_TYPE\_UINT64] - The timestamp of the buffer when it was processed by qtimlvconverter. Used for synchronization and latency measurements.
* **The stream ID from which this batch entry was produced:**
  * stream-id \[G\_TYPE\_INT] \[Optional] - Identifies the source stream that produced this batch entry. Useful in multi-stream pipelines for correlating inference results with their origin.
* **Timestamp of the stream buffer from which this batch entry was produced:**
  * stream-timestamp \[G\_TYPE\_UINT64] \[Optional] - The original timestamp of the frame from the source stream before preprocessing. Preserves temporal context for tracking or analytics.
* **The ID of the ROI meta from which this batch entry was produced:**
  * source-region-id \[G\_TYPE\_INT] \[Optional] - If the frame was derived from an ROI, this field contains the ID of the GstVideoRegionOfInterestMeta entry that defined the crop region. Enables downstream components to link inference results back to the original ROI.

# Guidelines for Writing an External Preprocessing Callback

External preprocessing lets you convert video into the model's input tensor **directly in your application**, instead of relying on one of the built-in engines (GLES, FastCV, OpenCV). You connect a single function — the *callback* — to the `qtimlvconverter` element, and from then on the SDK hands that function every frame and takes care of everything around it: caps negotiation, buffer pooling, batching, and ROI cropping. You only write the pixel transformation in the middle.

This is the way to implement a transformation that none of the built-in engines support — for example a custom color space conversion, a bespoke normalization curve, a non-typical pixel arrangement, or a proprietary algorithm. It can be done entirely in Python.

**To enable it, you only need to:**

1. Set the `engine` property to `none`, so no built-in engine claims the conversion.
2. Connect a handler to the `process` signal.

The element then invokes your handler once per frame. This works identically from both C/C++ and Python applications.

<Warning>If you connect the `process` signal but leave `engine` at its default backend, the built-in engine performs the conversion and your callback is **never** invoked. The `engine=none` property is required to route processing to your callback.</Warning>

### Callback Signature

Whatever the transformation, the callback always has the same shape — it receives the same arguments and returns a boolean. Only the pixel-processing logic in the middle changes. The handler looks like this:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    def preprocess(converter, vblits, mlsample):
        ...
        return True
    ```
  </Tab>

  <Tab title="C/C++">
    ```c theme={null}
    static gboolean
    preprocess (GstElement * converter, GstVideoBlits * blits,
        GstMLBundle * sample, gpointer userdata)
    {
      ...
      return TRUE;
    }
    ```
  </Tab>
</Tabs>

Each argument plays a specific role:

| Argument | Type (C / Python) | What it is |
| - | - | - |
| `converter` | `GstElement` / `qtimlvconverter` | The element that emitted the signal. Rarely needed inside the callback. |
| `blits` / `vblits` | `GstVideoBlits` | **The input.** The frame(s) you convert, each with its crop and placement rectangles. See [Video Blits](#video-blits). |
| `sample` / `mlsample` | `GstMLBundle` | **The output.** A container holding the output tensor you write into. See [ML Sample](#ml-sample). |
| `userdata` *(C only)* | `gpointer` | The pointer you passed to `g_signal_connect()`. Absent in Python. |
| *return value* | `gboolean` / `bool` | `TRUE` to let the pipeline continue, `FALSE` to abort it. |

And every callback follows the same four steps:

1. **Read** the input buffer(s) from `blits`.
2. **Transform** them — this is your custom conversion, normalization, or arrangement logic.
3. **Write** the result into the output tensor obtained from `sample`, respecting each blit's destination rectangle.
4. **Return** `TRUE`.

The two sections that follow describe the input (`blits`) and the output (`sample`) in detail, and the [Example](#example) at the end puts them together into a working handler.

### Video Blits

**This is the input to your callback** — the `blits` argument, of type `GstVideoBlits`. It holds one entry per input buffer to be converted — usually one, but potentially more when multiple sources are batched together.

| Field / accessor | Type (C / Python) | Description |
| - | - | - |
| number of entries | `gst_video_blits_size(blits)` / `vblits.size()` | How many blits the list holds. |
| individual entry | `gst_video_blits_entry(blits, i)` / `vblits.entry(i)` | The `i`-th blit. |

Each entry is a `GstVideoBlit` describing one input buffer and how it should be placed into the output tensor:

| Field | Type (C / Python) | Description |
| - | - | - |
| `buffer` | `GstBuffer` / `Gst.Buffer` | The input video buffer to read and convert. |
| `info` | `GstVideoInfo` / `GstVideo.VideoInfo` | Format, resolution, strides and plane offsets — needed to map `buffer`. |
| `source` | `GstVideoQuadrilateral` | The crop region in the input frame that should be read. |
| `destination` | `GstVideoRectangle` / `GstVideo.VideoRectangle` | The placement rectangle inside the output tensor that this blit fills. |
| `alpha` | `guint8` / `int` | Global alpha — `0` fully transparent, `255` fully opaque. |
| `rotate` | `GstVideoRotate` | Rotation to apply to the source content before placement. |

`source` is a quadrilateral (four corner points) rather than a plain rectangle, since a crop region can be an arbitrary quad in advanced scenarios. For the common case it is axis-aligned — convert it to a `GstVideoRectangle` with `gst_video_quadrilateral_to_rectangle()` in C, or `vblit.source.to_rectangle(rect)` in Python.

### ML Sample

**This is the output of your callback.** The `sample` argument, a `GstMLBundle`, wraps the output ML buffer (the tensor) together with its `GstMLInfo`. It starts out empty — writing your converted pixels into its tensor is what produces the plugin's output.

| Accessor | Description |
| - | - |
| `gst_ml_bundle_get_frame(sample, flags)` / `mlsample.get_frame(flags)` | Maps the sample's buffer and returns a `GstMLFrame` for reading or writing its tensor(s), depending on `flags`. |
| `gst_ml_bundle_get_buffer(sample)` / `mlsample.get_buffer()` | The underlying `GstBuffer`, unmapped. |
| `gst_ml_bundle_get_info(sample)` / `mlsample.get_info()` | The `GstMLInfo` describing the tensor shape and type. |

Map the frame with write access (`GST_MAP_WRITE` / `Gst.MapFlags.WRITE`) to get a `GstMLFrame` you can fill — in Python, `mlframe.get_tensor(0)` returns a writable, zero-copy NumPy array shaped to the tensor dimensions.

### ML Frame

**Obtained from `sample`.** `gst_ml_bundle_get_frame(sample, flags)` / `mlsample.get_frame(flags)` maps the sample's buffer and returns a `GstMLFrame`, which exposes the tensor(s) it wraps.

| Field / accessor | Type (C / Python) | Description |
| - | - | - |
| `info` | `GstMLInfo` | Describes the number of tensors and each tensor's type and dimensions. |
| `gst_ml_frame_get_tensor(mlframe, index)` *(C only)* | `GstMLTensor` | Wrapper struct giving direct pointer access to the `index`-th tensor's data and shape. |
| `mlframe.get_tensor(index)` *(Python only)* | `numpy.ndarray` | Zero-copy NumPy view of the `index`-th tensor's data, shaped to its dimensions. |

### ML Tensor

**Obtained from `mlframe`** via `gst_ml_frame_get_tensor()`. Only present in C/C++ — the Python bindings expose the equivalent data as a NumPy array through `mlframe.get_tensor()` instead, so there's no separate tensor struct to bind.

| Field | Type (C) | Description |
| - | - | - |
| `type` | `GstMLType` | Data type of the tensor elements, e.g. `UINT8`, `FLOAT32`. |
| `n_dimensions` | `guint` | Number of dimensions in the tensor, e.g. `4` for NHWC. |
| `dimensions` | `guint[GST_ML_TENSOR_MAX_DIMS]` | Size of each dimension, e.g. `[N, H, W, C]`. |
| `data` | `guint8 *` | Pointer to the raw, mapped tensor memory. |
| `size` | `gsize` | Size of `data` in bytes. |

### Example

The handler below implements the four steps for a single NV12 input buffer: it reads the buffer from `blits`, converts it to RGB with OpenCV, then resizes the source region into the destination region of the output tensor — reproducing the resize-and-place behavior of a built-in engine, but under full application control. The final lines show how the handler is connected to the element — note that `engine` is set to `none`.

<Tabs>
  <Tab title="Python application">
    In Python the signal is available through GObject introspection. The output tensor is exposed as a writable, zero-copy NumPy array through the `Frame.get_tensor()` helper provided by the `GstQtiML` overrides.

    ```python theme={null}
    import gi
    gi.require_version('Gst', '1.0')
    gi.require_version('GstVideo', '1.0')
    gi.require_version('GstQtiVideo', '1.0')
    gi.require_version('GstQtiML', '1.0')
    from gi.repository import Gst, GstVideo
    import numpy as np
    import cv2

    def preprocess(converter, vblits, mlsample):
        # Output tensor as a writable, zero-copy NumPy array.
        mlframe = mlsample.get_frame(Gst.MapFlags.WRITE)
        tensor = mlframe.get_tensor(0)

        # This example uses a single NV12 input buffer.
        vblit = vblits.entry(0)

        # Map the input buffer for reading.
        success, mapinfo = vblit.buffer.map(Gst.MapFlags.READ)
        if not success:
            raise RuntimeError("Could not map video buffer data!")

        # Wrap the NV12 luma plane as a zero-copy NumPy array.
        size = vblit.info.height * vblit.info.stride[0]
        luma = np.frombuffer(mapinfo.data, dtype='uint8', count=size)
        luma = luma.reshape(vblit.info.height, vblit.info.stride[0])

        # Wrap the NV12 chroma plane as a zero-copy NumPy array.
        size = (vblit.info.height // 2) * vblit.info.stride[1]
        chroma = np.frombuffer(mapinfo.data, dtype='uint8',
            offset=vblit.info.offset[1], count=size)
        chroma = chroma.reshape(vblit.info.height // 2, vblit.info.stride[1] // 2, 2)

        # Custom color conversion: NV12 -> RGB.
        rgb = cv2.cvtColorTwoPlane(luma, chroma, cv2.COLOR_YUV2RGB_NV12)

        # The source is a quadrilateral; convert it to a plain rectangle.
        source = GstVideo.VideoRectangle()
        vblit.source.to_rectangle(source)

        # Destination region inside the output tensor.
        left, right = vblit.destination.x, vblit.destination.x + vblit.destination.w
        top, bottom = vblit.destination.y, vblit.destination.y + vblit.destination.h

        # Crop the source region and resize it into the destination region.
        tensor[0, top:bottom, left:right] = cv2.resize(
            rgb[source.y:source.y + source.h, source.x:source.x + source.w],
            (vblit.destination.w, vblit.destination.h), cv2.INTER_AREA)

        vblit.buffer.unmap(mapinfo)
        return True  # return False to signal an error and stop the pipeline

    # ... after building the pipeline ...
    converter = pipeline.get_by_name("preprocess")
    converter.set_property("engine", "none")
    converter.connect("process", preprocess)
    ```
  </Tab>

  <Tab title="C/C++ application">
    In C/C++ connect a callback to the `process` signal, map the output ML frame for writing, and fill the tensor with your preprocessed data.

    ```cpp theme={null}
    #include <gst/ml/gstmlbundle.h>
    #include <gst/video/video-utils.h>
    #include <opencv2/opencv.hpp>

    static gboolean
    preprocess (GstElement * converter, GstVideoBlits * blits,
        GstMLBundle * sample, gpointer userdata)
    {
      // Map the output ML frame for writing and get its (first) tensor.
      GstMLFrame *mlframe = gst_ml_bundle_get_frame (sample, GST_MAP_WRITE);
      if (mlframe == NULL)
        return FALSE;

      GstMLTensor tensor = gst_ml_frame_get_tensor (mlframe, 0);

      // Tensor is NHWC: dimensions[1] = height, [2] = width, [3] = channels.
      gint dstheight = tensor.dimensions[1];
      gint dstwidth = tensor.dimensions[2];

      // This example uses a single NV12 input buffer.
      GstVideoBlit *blit = gst_video_blits_entry (blits, 0);

      // Map the input buffer for reading.
      GstMapInfo mapinfo;

      if (!gst_buffer_map (blit->buffer, &mapinfo, GST_MAP_READ)) {
        GST_ERROR ("Could not map video buffer data!");
        return FALSE;
      }

      gint srcheight = GST_VIDEO_INFO_HEIGHT (blit->info);
      gint lumastride = GST_VIDEO_INFO_PLANE_STRIDE (blit->info, 0);
      gint chromastride = GST_VIDEO_INFO_PLANE_STRIDE (blit->info, 1);
      gsize chromaoffset = GST_VIDEO_INFO_PLANE_OFFSET (blit->info, 1);

      // Wrap the NV12 luma and chroma planes as OpenCV matrices (zero copy).
      cv::Mat luma (srcheight, lumastride, CV_8UC1, mapinfo.data);
      cv::Mat chroma (srcheight / 2, chromastride / 2, CV_8UC2,
          mapinfo.data + chromaoffset);

      // Custom color conversion: NV12 -> RGB.
      cv::Mat rgb;
      cv::cvtColorTwoPlane (luma, chroma, rgb, cv::COLOR_YUV2RGB_NV12);

      // The source is a quadrilateral; convert it to a plain rectangle.
      GstVideoRectangle source;
      gst_video_quadrilateral_to_rectangle (&(blit->source), &source);

      // Wrap the output tensor as an RGB OpenCV matrix (zero copy).
      cv::Mat output (dstheight, dstwidth, CV_8UC3, tensor.data);

      // Crop the source region and resize it into the destination region.
      cv::Rect srcroi (source.x, source.y, source.w, source.h);
      cv::Rect dstroi (blit->destination.x, blit->destination.y,
          blit->destination.w, blit->destination.h);
      cv::resize (rgb (srcroi), output (dstroi),
          cv::Size (blit->destination.w, blit->destination.h), 0, 0, cv::INTER_AREA);

      gst_buffer_unmap (blit->buffer, &mapinfo);
      return TRUE; // return FALSE to signal an error and stop the pipeline
    }

    // ... after creating the element ...
    g_object_set (converter, "engine", "none", NULL);
    g_signal_connect (converter, "process", G_CALLBACK (preprocess), NULL);
    ```
  </Tab>
</Tabs>

# Usage

### Single Camera Stream — Save Tensors to File

Single camera stream with manually set UINT8 ML GstCaps and output tensors saved in separate files.

Common data types:

* `UINT8` — typical for quantized models, range 0–255
* `INT8` — used in signed quantized models, range −128 to 127
* `FLOAT16` / `FLOAT32` — for models requiring high precision, normalized to 0.0–1.0 or −1.0–1.0 (requires offset and scale)

<img alt="use" />

<Steps>
  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/media/output
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash Run the pipeline theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! \
    qtimlvconverter ! neural-network/tensors,type=UINT8,dimensions="<<1,300,300,3>>" ! multifilesink location=$HOME/Downloads/qimsdk_samples/media/output/tensor_u8_%d.bin
    ```
  </Step>
</Steps>

### Two-Stage Person Detection and Pose Estimation on Live Camera Stream

Demo pipeline with 2 ML stages running on a live camera stream. The first stage performs person detection and attaches the results to each frame. [`qtiobjtracker`](/plugin-reference/qtiobjtracker) then associates detected persons across frames and adds persistent tracking IDs. The second stage uses ROI-based preprocessing to crop each tracked person and runs pose estimation, producing skeleton keypoints overlaid on the display.

<img alt="use" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Person foot detection model | [Qualcomm AI Hub — Person Foot Detection](https://aihub.qualcomm.com/iot/models/foot_track_net) | `foot_track_net_w8a8.tflite` |
    | Person detection labels | <a href="../labels/foot_track_net.json">foot\_track\_net.json</a> | `foot_track_net.json` |
    | Foot track net settings | <a href="../labels/foot_track_net_settings.json">foot\_track\_net\_settings.json</a> | `foot_track_net_settings.json` |
    | HRNet pose model | [Qualcomm AI Hub — HRNet Pose](https://aihub.qualcomm.com/iot/models/hrnet_pose) | `hrnet_pose_w8a8.tflite` |
    | Pose labels | <a href="../labels/hrnet.json">hrnet.json</a> | `hrnet.json` |
    | HRNet settings | <a href="../labels/hrnet_settings.json">hrnet\_settings.json</a> | `hrnet_settings.json` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"
      scp foot_track_net_w8a8.tflite         <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp foot_track_net.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp foot_track_net_settings.json       <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp hrnet_pose_w8a8.tflite             <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp hrnet.json                         <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp hrnet_settings.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME_1=foot_track_net_w8a8.tflite
    export LABELS_NAME_1=foot_track_net.json
    export LABELS_NAME_2=foot_track_net_settings.json
    export MODEL_NAME_2=hrnet_pose_w8a8.tflite
    export LABELS_NAME_3=hrnet.json
    export LABELS_NAME_4=hrnet_settings.json
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash Run the pipeline theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qtimlvconverter name=stage_01_preproc mode=image-batch-non-cumulative \
    qtimltflite name=stage_01_inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
    external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" \
    model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_1 \
    qtimlpostprocess name=stage_01_postproc results=10 module=qpd labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_1 \
    settings=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_2 \
    qtimlvconverter name=stage_02_preproc image-disposition=centre mode=roi-batch-cumulative \
    qtimltflite name=stage_02_inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
    external-delegate-options="QNNExternalDelegate,backend_type=htp,htp_performance_mode=(string)2,log_level=(string)1;" \
    model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_2 \
    qtimlpostprocess name=stage_02_postproc results=1 module=hrnet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_3 \
    settings=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_4 \
    qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! tee name=t_split_1 \
    t_split_1. ! queue ! metamux_1. \
    t_split_1. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! stage_01_inference. stage_01_inference. ! queue ! \
    stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! metamux_1. \
    qtimetamux name=metamux_1 ! queue ! tee name=t_split_2 \
    t_split_2. ! queue ! metamux_2. \
    t_split_2. ! queue ! stage_02_preproc. stage_02_preproc. ! queue ! stage_02_inference. stage_02_inference. ! queue ! \
    stage_02_postproc. stage_02_postproc. ! text/x-raw ! queue ! metamux_2. \
    qtimetamux name=metamux_2 ! queue ! qtivoverlay ! queue ! waylandsink fullscreen=true sync=true async=true
    ```
  </Step>
</Steps>

### Four-Source Batched Object Detection with Compositor

Demo pipeline with batching of 4 sources. A detection inference is run and the results are overlaid via composer on the screen.

<img alt="use" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Yolov8 Detection W8A8 Batch 4 model | [Export from Qualcomm AI Hub](https://github.com/qualcomm/ai-hub-models/blob/v0.55.0/src/qai_hub_models/models/yolov8_det/README.md) | `yolov8_det_w8a8_batch_4.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp yolov8_det_w8a8_batch_4.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                           <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4             <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolov8_det_w8a8_batch_4.tflite
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME_1=ai_demo_sample.mp4
    export SRC_VIDEO_NAME_2=ai_demo_sample.mp4
    export SRC_VIDEO_NAME_3=ai_demo_sample.mp4
    export SRC_VIDEO_NAME_4=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash Run the pipeline theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qtimltflite name=inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,htp_performance_mode=(string)2;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME_1 ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw ! qtivtransform ! video/x-raw,format=NV12,width=640,height=360 ! queue ! tee name=tee_0 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME_2 ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw ! qtivtransform ! video/x-raw,format=NV12,width=640,height=360 ! queue ! tee name=tee_1 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME_3 ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw ! qtivtransform ! video/x-raw,format=NV12,width=640,height=360 ! queue ! tee name=tee_2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME_4 ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw ! qtivtransform ! video/x-raw,format=NV12,width=640,height=360 ! queue ! tee name=tee_3 \
    tee_0. ! video/x-raw,format=NV12 ! mixer. \
    tee_0. ! video/x-raw,format=NV12 ! batch. \
    tee_1. ! video/x-raw,format=NV12 ! mixer. \
    tee_1. ! video/x-raw,format=NV12 ! batch. \
    tee_2. ! video/x-raw,format=NV12 ! mixer. \
    tee_2. ! video/x-raw,format=NV12 ! batch. \
    tee_3. ! video/x-raw,format=NV12 ! mixer. \
    tee_3. ! video/x-raw,format=NV12 ! batch. \
    qtibatch name=batch ! queue ! qtimlvconverter ! queue ! inference. inference. ! queue ! qtimldemux name=mldemux \
    mldemux. ! queue ! qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! video/x-raw ! queue ! mixer. \
    mldemux. ! queue ! qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! video/x-raw ! queue ! mixer. \
    mldemux. ! queue ! qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! video/x-raw ! queue ! mixer. \
    mldemux. ! queue ! qtimlpostprocess results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! video/x-raw ! queue ! mixer. \
    qtivcomposer name=mixer \
    sink_0::position="<0, 0>" sink_0::dimensions="<960, 540>" \
    sink_1::position="<960,  0>" sink_1::dimensions="<960, 540>" \
    sink_2::position="<0, 540>" sink_2::dimensions="<960, 540>" \
    sink_3::position="<960, 540>" sink_3::dimensions="<960, 540>" \
    sink_4::position="<0, 0>" sink_4::dimensions="<960, 540>" \
    sink_5::position="<960, 0>" sink_5::dimensions="<960, 540>" \
    sink_6::position="<0, 540>" sink_6::dimensions="<960, 540>" \
    sink_7::position="<960, 540>" sink_7::dimensions="<960, 540>" \
    mixer. ! video/x-raw,format=NV12 ! queue ! waylandsink sync=true fullscreen=true
    ```
  </Step>
</Steps>


# qtimsgpub and qtimsgsub
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtimsg

 Message broker plugins

# Overview

The `gst-plugin-msgbroker` package consists of two GStreamer elements that implement standard Publish/Subscribe messaging semantics:

* `qtimsgpub`, which publishes messages to an external messaging backend
* `qtimsgsub`, which subsribes to messages from an external messaging backend

`gst-plugin-msgbroker` is a foundational messaging component in the Qualcomm QTI GStreamer plugin ecosystem. It enables seamless Pub/Sub communication by exposing a unified interface to the GStreamer pipeline.

At the architectural level, `msgbroker` functions as a high-level adapter that abstracts the underlying messaging protocol. Its extensible protocol adaptation layer is designed to support multiple Pub/Sub backends, allowing applications to integrate different messaging systems without changing the pipeline-facing interface.

By decoupling protocol-specific details from pipeline logic, `gst-plugin-msgbroker` allows GStreamer pipelines to operate cleanly as message producers or consumers while remaining portable across supported messaging implementations.

* `qtimsgpub`: **A publisher element that transmits pipeline data to an external message broker.**

```
gst-launch-1.0 -e --gst-debug=2 \
   [Source Plugin] ! \
   [Filter/Processing] ! \
   [Publisher Plugin] topic=<Target Topic> host=<Target Host> port=<Port> protocol=<Protocol>
```

* `qtimsgsub`: **A subscriber element that receives external messages from the broker and injects them into the media pipeline.**

**Supported protocols**

* MQTT
  * MQTT is a lightweight publish/subscribe messaging protocol optimized for machine-to-machine (M2M) telemetry in low-bandwidth environments. It uses a broker-based communication model that decouples clients from one another: publishers send messages to named topics, and subscribers receive messages from the topics to which they are subscribed.

    Built on the **libmosquitto** Mosquitto C/C++ client library, `gst-plugin-msgbroker` abstracts the underlying complexity of the **OASIS-standard MQTT protocol** and provides application developers with a clean, high-level interface for integrating reliable publish/subscribe (Pub/Sub) messaging into GStreamer-based applications and pipelines.

* KAFKA
  * Kafka is a distributed streaming platform designed for building real-time data pipelines and streaming applications. Kafka uses a distributed commit log architecture where messages are persisted to disk and replicated across multiple brokers for fault tolerance. Kafka organizes messages into topics, which are partitioned and distributed across a cluster of brokers. Producers publish messages to topics, and consumers subscribe to topics as part of consumer groups, enabling scalable message processing with automatic load balancing and fault tolerance.

    Built on the **librdkafka** C client library, the kafka protocol library abstracts the underlying complexity of the Apache Kafka protocol and provides application developers with a clean, high-level interface for integrating reliable publish/subscribe (Pub/Sub) messaging into GStreamer-based applications and pipelines.

<img alt="" />

```
# Message Construction
echo "Message in file" > /usr/local/message.txt

# Publisher pipeline
gst-launch-1.0 -e --gst-debug=2 filesrc location=/usr/local/message.txt ! qtimsgpub topic=file host=127.0.0.1 port=1883 protocol=mqtt config=/usr/local/config

# Subscriber pipeline
gst-launch-1.0 -e --gst-debug=2 qtimsgsub topic=file host=127.0.0.1 port=1883 protocol=mqtt config=/usr/local/config ! filesink location=/usr/local/sub_message.txt async=false
```

Publisher pipeline reads from disk and publishes the content via `file` topic to MQTT server. Subscriber pipeline receives messages from MQTT server via file topic and writes to disk.

# Hierarchy

* [GObject](https://docs.gtk.org/gobject/)<br />
     <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
        <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
           <Icon icon="arrow-turn-down-right" />[GstBaseSink](https://gstreamer.freedesktop.org/documentation/base/gstbasesink.html?gi-language=c)<br />
              <Icon icon="arrow-turn-down-right" />qtimsgpub

* [GObject](https://docs.gtk.org/gobject/)<br />
     <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
        <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
           <Icon icon="arrow-turn-down-right" />[GstBaseSrc](https://gstreamer.freedesktop.org/documentation/base/gstbasesrc.html?gi-language=c)<br />
              <Icon icon="arrow-turn-down-right" />qtimsgsub

# Pad Templates

* qtimsgpub plugin

| Capabilities | |
| - | - |
| `ANY` | `format: NA` |
| Availability: *Always* | |
| Direction: *sink* | |

* qtimsgsub plugin

| Capabilities | |
| - | - |
| `ANY` | `format: NA` |
| Availability: *Always* | |
| Direction: *src* | |

## Element Properties

| Property | Plugin | Description |
| - | - | - |
| `config` | qtimsgpub / qtimsgsub | Absolute path to the protocol configuration file.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `host` | qtimsgpub / qtimsgsub | Message broker server IP address.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `json` | qtimsgpub | Converts messages to JSON format before publishing. Ignored by qtimsgsub.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `message` | qtimsgpub | Message payload specified directly for publishing. Ignored by qtimsgsub.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable (changeable in NULL, READY, PAUSED, PLAYING states)` |
| `port` | qtimsgpub / qtimsgsub | Message broker server port.<br /><br />`Type: Integer`<br />`Default: 1883`<br />`Range: 0 - 2147483647`<br />`Flags: readable/writable` |
| `protocol` | qtimsgpub / qtimsgsub | Message protocol such as mqtt or kafka.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `topic` | qtimsgpub / qtimsgsub | Topic to publish to for qtimsgpub or subscribe to for qtimsgsub.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable (changeable in NULL, READY, PAUSED, PLAYING states)` |

## Signals

| Signals | Plugin | Description |
| - | - | - |
| `add-publish` | qtimsgpub | Signal to publish an additional topic and message at runtime.<br /><br />`Type: Signal (boolean return)`<br />`Parameters: (string topic, string message)` |

## MQTT Configuration File Options

This configuration file is provided to the element through the `config` property. The file uses a `key=value` format and supports comments starting with `#.`

| Option | Description |
| - | - |
| `clean_session` | Whether to clear historical sessions for the same client ID.<br /><br />`Type: bool`<br />`Default: TRUE` |
| `id` | MQTT client ID.<br /><br />`Type: string`<br />`Default: NULL (randomly assigned)` |
| `keepalive` | Heartbeat keep‑alive interval (seconds).<br /><br />`Type: int`<br />`Default: 60` |
| `max_inflight` | Maximum concurrent in‑flight messages for QoS 1/2.<br /><br />`Type: int`<br />`Default: 20` |
| `mqtt_version` | MQTT protocol version (MQTTV31 / MQTTV311 / MQTTV5).<br /><br />`Type: string`<br />`Default: MQTTV311` |
| `password` | Broker authentication password.<br /><br />`Type: string`<br />`Default: NULL` |
| `qos` | Quality of Service level (0 / 1 / 2).<br /><br />`Type: int`<br />`Default: 0` |
| `retain` | Whether messages are retained on the Broker.<br /><br />`Type: bool`<br />`Default: FALSE` |
| `tcp_nodelay` | Disable Nagle’s algorithm for lower latency.<br /><br />`Type: bool`<br />`Default: FALSE` |
| `username` | Broker authentication username.<br /><br />`Type: string`<br />`Default: NULL` |
| `will_payload` | Last Will message payload for unexpected disconnects.<br /><br />`Type: string`<br />`Default: NULL` |
| `will_qos` | QoS level for the Last Will message (0 / 1 / 2).<br /><br />`Type: int`<br />`Default: 0` |
| `will_retain` | Whether the Last Will message is retained on the Broker.<br /><br />`Type: bool`<br />`Default: FALSE` |
| `will_topic` | Last Will message topic for unexpected disconnects.<br /><br />`Type: string`<br />`Default: NULL` |

## Kafka Configuration File Options

This configuration file is provided to the element through the `config` property. The file uses an INI-style format with sections and supports comments starting with `#.`

The configuration file is organized into three sections:

### \[global-config]

Common configuration options that apply to both producers and consumers.

| Option | Description |
| - | - |
| `proto-cfg` | Protocol‑specific configuration string in key‑value format, separated by semicolons.<br /><br />`Type: string`<br />`Default: NULL (optional)`<br /> `Example: acks=all; compression.type=gzip; retries=3` |

### \[producer-config]

Configuration options specific to Kafka producers (qtimsgpub).

| Option | Description |
| - | - |
| `partition-key` | Key used for message partitioning. Messages with the same key are sent to the same partition.<br /><br />`Type: string`<br />`Default: NULL (required)` <br />  `Example: partition-key = "cars"` |
| `proto-cfg` | Producer‑specific protocol configuration string in key‑value format, separated by semicolons.<br /><br />`Type: string`<br />`Default: NULL (optional)`<br />Example: `batch.size=16384; linger.ms=10` |
| `timeout-ms` | Maximum time (in milliseconds) to wait for message delivery confirmation.<br /><br />`Type: int (milliseconds)`<br />`Default: NULL (required)`  <br />  `Example: timeout-ms = "5000"` |

### \[consumer-config]

Configuration options specific to Kafka consumers (qtimsgsub).

| Property | Description |
| - | - |
| `group-id` | Consumer group identifier. Consumers with the same group‑id share the load of consuming messages.<br /><br />`Type: string`<br />`Default: NULL (required)`   <br />  `Example: group-id = "tmp-group"` |
| `proto-cfg` | Consumer‑specific protocol configuration string in key‑value format, separated by semicolons.<br /><br />`Type: string`<br />`Default: NULL (optional)`<br />`Example: auto.offset.reset=earliest; enable.auto.commit=true` |

# Internal Architecture

The plugin follows a three-layer architecture that decouples the GStreamer element layer from the underlying messaging protocol: GStreamer Element Layer → Protocol Adaptor → Protocol Implementation. The adaptor layer dynamically loads protocol-specific implementations using `dlopen` and `dlsym`, and exposes a unified interface to the element layer.

<img alt="" />

## Protocol Adapter

`GstMsgProtocol`  is responsible for loading the protocol-specific shared library based on the configured protocol name (for example, `libgstqtimqttadaptor.so`), resolving the unified entry symbol `GST_PROTOCOL_CFUNC_SYMBOL`, and forwarding upper-layer API calls to the corresponding protocol implementation.

`GstProtocolCommonFunc` defines the common protocol interface, including `new, free, config, connect, disconnect, publish, and subscribe`.

```
struct _GstProtocolCommonFunc {
  GstProtocolNewFunction        new;        // Create protocol instance
  GstProtocolFreeFunction       free;       // Release protocol instance
  GstProtocolConfigFunction     config;     // Configure protocol parameters
  GstProtocolConnectFunction    connect;    // Connect to Broker
  GstProtocolDisconnectFunction disconnect; // Disconnect
  GstProtocolPublishFunction    publish;    // Publish message
  GstProtocolSubscribeFunction  subscribe;  // Subscribe to Topic
};
```

# MQTT Protocol Implementation (GstMqtt)

`GstMqtt (mqtt.c)` **implements the complete lifecycle of the Mosquitto clients and serves as the concrete MQTT protocol implementation behind the generic adaptor API.**

* **Dynamic Loading:**  The Mosquitto library is loaded at runtime using `dlopen("libmosquitto.so.1")`, avoiding a hard build-time dependency.
* **Dual-role support:** The implementation supports both publisher (`pub`) and subscriber (`sub`) roles. The role parameter determines the operating mode and the corresponding callbacks are registered accordingly.
* **Asynchronous event handling:** The Mosquitto network loop is started on a dedicated thread using `mosquitto_loop_start()`, enabling non-blocking network I/O.
* **Callback flow:** For subscribed messages, data is propagated through the callback chain `message_callback → gst_adaptor_sub_callback → gst_plugin_sub_callback`, and is ultimately pushed into GstDataQueue for consumption by the GStreamer pipeline thread.

## Message Delivery Flow

```
Publisher  │  Upstream element (GStreamer pipeline thread)
           │      │
           │      ▼
           │  GstBuffer
           │      │
           │      ▼
           │  gst_msg_pub_render()
           │      │
           │      ▼
           │  gst_msg_protocol_publish()
           │      │
           │      ▼
           │  gst_mqtt_publish()
           │      │  mosquitto_publish_v5()
           │      ▼
-----------+----------------------------------------------
Broker     │  ┌──────────────┐
           │  │  MQTT Broker           │
           │  └──────────────┘
-----------+----------------------------------------------
Subscriber │      │  (Mosquitto network thread)
           │      ▼
           │  message_callback()
           │      │
           │      ▼
           │  gst_adaptor_sub_callback()
           │      │
           │      ▼
           │  gst_plugin_sub_callback()
           │      │  gst_data_queue_push()
           │      ▼
           │  GstDataQueue (msg_queue)
           │      │  gst_data_queue_pop() (GStreamer pipeline thread)
           │      ▼
           │  gst_msg_sub_create()
           │      │
           │      ▼
           │  GstBuffer → downstream elements
```

## Publish Side Flow (qtimsgpub)

```
Pipeline PLAYING
    │
    ▼
gst_msg_pub_start()
    ├── gst_msg_protocol_new(protocol="mqtt", role="pub")
    │       └── dlopen("libgstqtimqttadaptor.so")
    │           └── gst_mqtt_new("pub")
    │               └── mosquitto_lib_init()
    ├── gst_msg_protocol_config(config_path)
    │       └── gst_mqtt_config()
    │           ├── Parse configuration file (extract_prop_from_file)
    │           ├── mosquitto_new(id, clean_session)
    │           ├── Set protocol version, maximum inflight messages, will message, authentication, etc.
    │           └── Register connect/disconnect/publish callbacks
    └── gst_msg_protocol_connect(host, port)
            └── gst_mqtt_connect()
                ├── mosquitto_connect_bind_v5()
                └── mosquitto_loop_start()  ← Start asynchronous network thread
data arrival (gst_msg_pub_render)
    ├── [Optional] Publish CLI message from message_cmd
    ├── [Optional] Convert GstBuffer data to JSON
    └── gst_msg_protocol_publish(topic, message)
            └── gst_mqtt_publish()
                └── mosquitto_publish_v5()
Pipeline STOP
    └── gst_msg_pub_stop()
            └── gst_msg_protocol_disconnect()
                    └── gst_mqtt_disconnect()
                        ├── mosquitto_disconnect_v5()
                        └── mosquitto_loop_stop()
```

## Subscribe Side Flow (qtimsgsub)

```
Pipeline PLAYING
    │
    ▼
gst_msg_sub_start()
    ├── gst_msg_protocol_new(protocol="mqtt", role="sub")
    │       └── dlopen("libgstqtimqttadaptor.so")
    │           └── gst_mqtt_new("sub")
    ├── gst_msg_protocol_config(config_path)
    │       └── gst_mqtt_config()
    │           ├── Parse configuration file
    │           ├── mosquitto_new()
    │           └── Register connect/disconnect/subscribe/message callbacks
    ├── gst_msg_protocol_connect(host, port)
    │       └── mosquitto_connect_bind_v5() + mosquitto_loop_start()
    └── gst_msg_protocol_subscribe(topic, msg_queue, callback)
            └── gst_mqtt_subscribe()
                └── mosquitto_subscribe_v5(topic)
Message arrival (network thread)
    └── message_callback()
            ├── Verify Topic match
            ├── Wrap into GstBuffer (gst_buffer_new_memdup)
            └── Push into GstDataQueue
GStreamer thread requests data
    └── gst_msg_sub_create()
            ├── gst_data_queue_pop()
            └── Return GstBuffer
Pipeline STOP
    └── gst_msg_sub_stop()
            └── gst_msg_protocol_disconnect()
```

# Kafka Protocol Implementation (GstKafka)

`GstKafka` (kafka.c) implements the complete lifecycle of the librdkafka client and serves as the concrete Kafka protocol implementation behind the generic adaptor API.

* **Dual-role support:** The implementation supports both producer (pub) and subscriber (sub) roles. The role parameter determines the operating mode, and the corresponding callbacks are registered accordingly.

* **Asynchronous event handling:**
  * For producers, the delivery callback (`gst_kafka_dr_msg_cb`) is invoked asynchronously when messages are acknowledged by the broker. The producer uses `rd_kafka_poll()` to service delivery callbacks.
  * For consumers, a dedicated `GstTask` thread continuously polls the broker using `rd_kafka_consumer_poll()` for new messages, enabling non-blocking network I/O.

* **Callback flow:** For subscribed messages, data is propagated through the callback chain: `gst_kafka_consume_message → gst_adaptor_sub_callback → gst_plugin_sub_callback`, and is ultimately pushed into GstDataQueue for consumption by the GStreamer pipeline thread.

* **Message delivery tracking:** The producer implementation uses a mutex-protected status variable to track message delivery state (submitted, success, or failure), ensuring synchronous confirmation of message delivery.

## Message Delivery Flow

```
Publisher  │  Upstream element (GStreamer pipeline thread)
           │      │
           │      ▼
           │  GstBuffer
           │      │
           │      ▼
           │  gst_msg_pub_render()
           │      │
           │      ▼
           │  gst_msg_protocol_publish()
           │      │
           │      ▼
           │  gst_kafka_publish()
           │      │  rd_kafka_producev()
           │      │  rd_kafka_poll() (wait for delivery callback)
           │      ▼
-----------+----------------------------------------------
Broker     │  ┌──────────────────┐
           │  │  Kafka Cluster                │
           │  │  (Partitioned  Topics)        │
           │  │                               │
           │  └──────────────────┘
-----------+----------------------------------------------
Subscriber │      │  (Consumer task thread - GstTask)
           │      ▼
           │  gst_kafka_consume_message()
           │      │  rd_kafka_consumer_poll()
           │      ▼
           │  message received
           │      │
           │      ▼
           │  gst_adaptor_sub_callback()
           │      │
           │      ▼
           │  gst_plugin_sub_callback()
           │      │  gst_data_queue_push()
           │      ▼
           │  GstDataQueue (msg_queue)
           │      │  gst_data_queue_pop() (GStreamer pipeline thread)
           │      ▼
           │  gst_msg_sub_create()
           │      │
           │      ▼
           │  GstBuffer → downstream elements
```

## Publisher Side Flow(qtimsgpub)

```
Pipeline PLAYING
│
▼
gst_msg_pub_start()
├── gst_msg_protocol_new(protocol="kafka", role="pub")
│       └── dlopen("libgstqtikafkaadaptor.so")
│           └── gst_kafka_new("pub")
│               └── Initialize GstKafka structure
├── gst_msg_protocol_config(config_path)
│       └── gst_kafka_config()
│           ├── Parse configuration file (gst_fetch_config_value)
│           ├── Parse proto-cfg strings (gst_kafka_parse_proto_cfg)
│           │   └── Set key=value pairs via rd_kafka_conf_set()
│           └── Create rd_kafka_conf_t configuration object
└── gst_msg_protocol_connect(host, port)
    └── gst_kafka_connect()
        ├── Set bootstrap.servers = host:port
        ├── Register delivery callback (gst_kafka_dr_msg_cb)
        └── rd_kafka_new(RD_KAFKA_PRODUCER, conf)

Data arrival (gst_msg_pub_render)
├── [Optional] Publish CLI message from message_cmd
├── [Optional] Convert GstBuffer data to JSON
└── gst_msg_protocol_publish(topic, message)
    └── gst_kafka_publish()
        ├── Set msgstatus = GST_KAFKA_MSG_SUBMITTED
        ├── rd_kafka_producev()
        │   ├── RD_KAFKA_V_TOPIC(topic)
        │   ├── RD_KAFKA_V_VALUE(payload)
        │   ├── RD_KAFKA_V_KEY(partition_key)
        │   └── RD_KAFKA_V_OPAQUE(self) for callback context
        ├── rd_kafka_poll(timeout_ms) ← Wait for delivery callback
        └── Check msgstatus == GST_KAFKA_MSG_DELIVERY_SUCCESS

Delivery callback (network thread)
└── gst_kafka_dr_msg_cb()
    ├── Check rkmessage->err
    ├── Set msgstatus = SUCCESS or FAIL
    └── rd_kafka_yield() ← Return control to rd_kafka_poll()

Pipeline STOP
└── gst_msg_pub_stop()
    └── gst_msg_protocol_disconnect()
        └── gst_kafka_disconnect()
            ├── rd_kafka_flush(10000) ← Flush pending messages
            └── rd_kafka_destroy(producer)
```

## Subscribe Side Flow(qtimsgsub)

```
Pipeline PLAYING
│
▼
gst_msg_sub_start()
├── gst_msg_protocol_new(protocol="kafka", role="sub")
│       └── dlopen("libgstqtikafkaadaptor.so")
│           └── gst_kafka_new("sub")
├── gst_msg_protocol_config(config_path)
│       └── gst_kafka_config()
│           ├── Parse configuration file
│           ├── Parse proto-cfg strings
│           └── Set group.id via rd_kafka_conf_set()
├── gst_msg_protocol_connect(host, port)
│       └── gst_kafka_connect()
│           ├── Set bootstrap.servers = host:port
│           └── rd_kafka_new(RD_KAFKA_CONSUMER, conf)
└── gst_msg_protocol_subscribe(topic, msg_queue, callback)
    └── gst_kafka_subscribe()
        ├── rd_kafka_topic_partition_list_new()
        ├── rd_kafka_topic_partition_list_add(topic)
        ├── rd_kafka_subscribe(subscription)
        ├── Create GstTask (gst_kafka_consume_message)
        └── gst_task_start() ← Start consumer polling thread

Consumer task loop (dedicated thread)
└── gst_kafka_consume_message()
    ├── rd_kafka_consumer_poll(100ms)
    ├── Check rkm->err
    ├── Extract payload and topic
    ├── Create GstBuffer from payload
    └── Invoke callback chain
        └── gst_adaptor_sub_callback()
            └── Push to GstDataQueue

GStreamer thread requests data
└── gst_msg_sub_create()
    ├── gst_data_queue_pop()
    └── Return GstBuffer

Pipeline STOP
└── gst_msg_sub_stop()
    └── gst_msg_protocol_disconnect()
        └── gst_kafka_disconnect()
            ├── gst_task_stop() ← Stop consumer task
            ├── rd_kafka_consumer_close()
            ├── rd_kafka_destroy(consumer)
            └── gst_task_join() ← Wait for task completion
```

# Prerequisites & Assumptions

### MQTT

* Mosquitto server installed and configured with optional username/password.
  * Steps to run Mosquitto server:
    * Installation steps refer to [Download | Eclipse Mosquitto](https://mosquitto.org/download/). (Simply: `sudo apt-get install mosquitto`)

<Callout type="note">
  The location of the configuration file `mosquitto.conf` is displayed after this installation.
</Callout>

* `mosquitto_passwd -c /etc/mosquitto/pwfile <username>`, set password following the instructions on shell
* Config `mosquitto.conf` with below settings:

```
password_file /etc/mosquitto/pwfile
allow_anonymous false
listener 1883
```

* `systemctl daemon-reload && systemctl restart mosquitto`
* Client has a valid plugin configuration file on device (/usr/local/config) with mqtt\_version/username/password as needed.

```
# MQTT Client Configuration
mqtt_version = MQTTV311
username = <username registered on server>
password = <password registered on server>
...
```

* Network connectivity: client and server can ping each other; broker listens on 1883.

### KAFKA

* Step 1: Get Kafka server running on a host machine.
  * [Download](https://dlcdn.apache.org/kafka/) the latest Kafka release and extract it.
  ```
  $ tar -xzf kafka_<ver>.tgz
  $ cd kafka_<ver>
  ```

<Note>
  This has been tested with `v2.13-4.3.0`
</Note>

* Step 2: Start the Kafka environment
  * NOTE: Your local environment must have Java 17+ installed.
  * Kafka can be run using local scripts and downloaded files or the docker image.
  * Using downloaded files:

1. Update config/server.properties.

```
# Update the "advertised.listeners" with the IP address of the machine.
advertised.listeners=PLAINTEXT://<IP ADDR of HOST>:9092,CONTROLLER://localhost:9093
```

2. Generate a Cluster UUID

```
$ KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
```

3. Format Log Directories

```
$ bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties
```

4. Start the Kafka Server

```
$ bin/kafka-server-start.sh config/server.properties
```

Once the Kafka server has successfully launched, you will have a basic Kafka environment running and ready to use.

5. Create topics with partitions based on your use case.
   This topic will be used to publish and consume messages later.

```
# From a different shell
$ bin/kafka-topics.sh --create --topic tmp --bootstrap-server localhost:9092
```

<Note>
  A topic has to be created before running any testcases.
</Note>

* Step 3: Connect the device to the same netowork as the server. Server and client(qtimsgpub/qtimsgsub) should be able to ping each other.

# Use Cases

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp yolox_w8a8.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/models $HOME/Downloads/qimsdk_samples/labels $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/media/output
    export MODEL_NAME=yolox_w8a8.tflite
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>
</Steps>

<Note>
  Subscriber pipeline should be run first and then launch the publisher pipeline from a different shell.
</Note>

### Publish CLI message -> Subscribe to file

This example demonstrates how a message provided through the message property is transmitted from the publisher pipeline to the subscriber pipeline.

<img alt="" />

In this pipeline (publisher), `fakesrc` is used as a dummy source to establish the pipeline structure. The `qtimsgpub` element then publishes the literal message (`"Command Message"`) specified through its `message` property to the `cmd_topic` topic on the configured MQTT or KAFKA broker.

**Publisher Pipeline:**

Use following for **MQTT**:

```
Publisher (one-shot):
gst-launch-1.0 -e --gst-debug=2 fakesrc ! qtimsgpub topic=cmd_topic message="Command Message" host=<mosquitto server server ip> port=<port advertised by the server> protocol=mqtt config=<path to mqtt config file>
```

for **Kafka**:

```
# Create topic on host
bin/kafka-topics.sh --create --topic cmd_topic --bootstrap-server localhost:9092
```

```
#Example configuration file
[producer-config]
partition-key = "objects"
timeout-ms = "5000"

[consumer-config]
group-id = "tmp-group"
```

```
Publisher (one-shot):
gst-launch-1.0 -e --gst-debug=2 fakesrc ! qtimsgpub topic=cmd_topic message="Command Message" host=<kafka server ip> port=<port advertised by the server> protocol=kafka config=<path to kafka config file>
```

**Subscriber Pipeline:**

This pipeline (subscriber) uses `qtimsgsub` to subscribe to messages published on the `cmd_topic` topic of the specified MQTT or KAFKA broker. The received messages are forwarded directly to `filesink`, which writes them to `sub_message.txt`.

Use following for **MQTT:**

```
Subscriber (press Ctrl+C to stop):
gst-launch-1.0 -e --gst-debug=2 qtimsgsub topic=cmd_topic host=<mosquitto server> port=<port advertised by the server> protocol=mqtt config=<path to mqtt config file> ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/sub_message.txt async=false
```

for **Kafka**

```
Subscriber (press Ctrl+C to stop):
gst-launch-1.0 -e --gst-debug=2 qtimsgsub topic=cmd_topic host=<kafka server ip> port=<port advertised by the server> protocol=kafka config=<path to kafka config file> ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/sub_message.txt async=false
```

### Publish file contents → Subscribe to file

This example demonstrates the transmission of a text-file-based message from the publisher pipeline to the subscriber pipeline. The file `message.txt` is first created with the content `"Message in file"`, and the publisher pipeline uses `filesrc` to read the file contents. The `qtimsgpub` element then publishes the data as a message to the `file_topic` topic on the configured MQTT/KAFKA broker. Similar to Use Case 1, the subscriber pipeline uses `qtimsgsub` to subscribe to the `file_topic` topic on the MQTT or KAFKA broker, and the received messages are written to `sub_message.txt` using `filesink`.

<img alt="" />

**Publisher Pipeline:**

Use following for **MQTT**:

```
Publisher (prepare and send):
echo "Message in file" > $HOME/Downloads/qimsdk_samples/media/message.txt
gst-launch-1.0 -e --gst-debug=2 filesrc location=$HOME/Downloads/qimsdk_samples/media/message.txt ! qtimsgpub topic=file_topic host=<mosquitto server> port=<port advertised by the server> protocol=mqtt config=<path to mqtt config file>
```

For **Kafka**:

```
# Create topic on host
bin/kafka-topics.sh --create --topic file_topic --bootstrap-server localhost:9092
```

```
Publisher (prepare and send):
echo "Message in file" > $HOME/Downloads/qimsdk_samples/media/message.txt
gst-launch-1.0 -e --gst-debug=2 filesrc location=$HOME/Downloads/qimsdk_samples/media/message.txt ! qtimsgpub topic=file_topic host=<kafka server ip> port=<port advertised by the server> protocol=kafka config=<path to kafka config file>
```

**Subscriber Pipeline:**

Use following for **MQTT**:

```
Subscriber (press Ctrl+C to stop):
gst-launch-1.0 -e --gst-debug=2 qtimsgsub topic=file_topic host=<mosquitto server> port=<port advertised by the server> protocol=mqtt config=<path to mqtt config file> ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/sub_message.txt async=false
```

For **Kafka**:

```
Subscriber (press Ctrl+C to stop):
gst-launch-1.0 -e --gst-debug=2 qtimsgsub topic=file_topic host=<kafka server ip> port=<port advertised by the server> protocol=kafka config=<path to kafka config file> ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/sub_message.txt async=false
```

### Publish AI Metadata packaged in JSON → Subscribe to AI Metadata

This example demonstrates how AI metadata produced after model inference can be transmitted from the publisher pipeline running the AI model to the subscriber pipeline. The subscriber pipeline can then implement custom logic to handle the AI event and trigger the appropriate action.

<img alt="" />

This AI pipeline performs object detection on incoming video frames, and `qtimsgpub` publishes the resulting classification metadata to the MQTT or KAFKA broker in JSON format.

**Publisher Pipeline:**

Use following for **MQTT:**

```
Publisher (AI pipeline → JSON):
gst-launch-1.0 -e --gst-debug=2 filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! queue ! qtimlvconverter ! queue ! qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME ! queue ! qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue !  qtimsgpub topic=detection_topic host=<mosquitto server> port=<port advertised by the server> protocol=mqtt config=<path to mqtt config file> json=true async=false
```

For **Kafka**

```
# Create topic on host
bin/kafka-topics.sh --create --topic detection_topic --bootstrap-server localhost:9092
```

```
Publisher (AI pipeline → JSON):
gst-launch-1.0 -e --gst-debug=2 filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! queue ! qtimlvconverter ! queue ! qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME ! queue ! qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue !  qtimsgpub topic=detection_topic host=<kafka server ip> port=<port advertised by the server> protocol=kafka config=<path to kafka config file> json=true async=false
```

**Subscriber Pipeline:**

This subscriber pipeline uses `qtimsgsub` to subscribe to the `detection_topic` MQTT or KAFKA topic. The received data, expected to be in JSON format, is then written directly to `sub.json` through `filesink`.

Use following for **MQTT**:

```
Subscriber (press Ctrl+C to stop):
gst-launch-1.0 -e --gst-debug=2 qtimsgsub topic=detection_topic host=<mosquitto server> port=<port advertised by the server> protocol=mqtt config=<path to mqtt config file> ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/sub.json async=false
```

For **Kafka**:

```
Subscriber (press Ctrl+C to stop):
gst-launch-1.0 -e --gst-debug=2 qtimsgsub topic=detection_topic host=<kafka server ip> port=<port advertised by the server> protocol=kafka config=<path to kafka config file> ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/sub.json async=false
```


# qtiobjtracker
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtiobjtracker

Real time multi object tracking plugin

# Overview

`qtiobjtracker` is a GStreamer plugin that provides **real-time multi-object tracking** by associating detected objects across consecutive video frames and assigning a persistent **tracking ID** to each object.

The plugin operates on object detection metadata produced by upstream inference or post-processing elements. For each detected object, it analyzes temporal continuity across frames and updates the metadata with tracking information, allowing the same object to be identified consistently over time.

### Key Responsibilities

The primary purpose of `qtiobjtracker` is to:

* maintain stable object identities across frames using persistent track IDs
* track object motion over time based on detection results
* improve the temporal consistency of object-level analytics
* enable downstream components to perform higher-level video analytics, event processing, and behavior analysis.

`qtiobjtracker` does **not** perform object detection itself. It depends on upstream pipeline elements to generate object detections and associated metadata. The tracker consumes that metadata, performs frame-to-frame association, and augments the object metadata with tracking IDs for downstream use.

<img alt="" />

## Example Pipeline

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp yolox_w8a8.tflite           <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolox_w8a8.tflite
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    tee name=t ! qtimetamux name=obj_mux ! queue ! qtiobjtracker algo=bytetrack ! queue ! qtivoverlay ! waylandsink fullscreen=true sync=true \
    t. ! queue ! qtimlvconverter ! queue ! \
    qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
      external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
    qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME \
      settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! obj_mux.
    ```
  </Step>
</Steps>

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />qtiobjtracker

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-raw` | `format: ANY` |
| `text/x-raw` | `format: utf8` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `video/x-raw` | `format: ANY` |
| `text/x-raw` | `format: utf8` |
| Availability: *Always* | |
| Direction: *source* | |

# Element Properties

| Property | Description |
| - | - |
| `algo` | Algorithm name used for the object tracker.<br /><br />`Type: Enum` <br />`Default: 0, "bytetrack"`<br />`Flags: readable/writable (changeable in NULL, READY, PAUSED, PLAYING)` <br /> `Example: algo="bytetrack" (or) algo=0` |
| `parameters` | Parameters used by the chosen object tracker algorithm in GstStructure string format. Applicable only for some algorithms.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |

# Internal Architecture Details

### Pluggable Tracking Backend Architecture

`qtiobjtracker` is designed with a modular tracking architecture that separates the GStreamer plugin framework from the underlying tracking algorithm implementation. The plugin exposes a common tracking interface while allowing different tracking algorithms to be implemented, selected, and maintained independently of the core element.

Each tracking algorithm is packaged as a separate shared library, referred to as a **tracking backend**. The `qtiobjtracker` element is responsible for:

* managing the GStreamer element lifecycle
* integrating with the pipeline
* receiving and forwarding detection metadata
* loading and interfacing with the selected tracking backend

The tracking logic itself is implemented entirely within the backend library.

**Runtime Algorithm Selection**

The tracking algorithm is selected at runtime through the `algo` property. Based on the configured value, `qtiobjtracker` dynamically loads the corresponding backend library and initializes the selected implementation.

This design provides several benefits:

* **runtime flexibility** — tracking behavior can be selected per pipeline or use case
* **separation of concerns** — algorithm implementation remains independent of the plugin core
* **maintainability** — tracking backends can be developed and updated independently
* **extensibility** — new tracking algorithms can be added without changing the public plugin interface

Only the backend associated with the selected algorithm is loaded and executed.

**Backend Responsibilities**

Each tracking backend implements a common interface and is responsible for:

* associating detections across consecutive frames
* creating, updating, and terminating tracks
* applying motion prediction and/or spatial matching
* maintaining internal tracking state

Backends operate exclusively on detection metadata, such as bounding boxes, class labels, and confidence scores. They do not perform object detection.

The backend-based architecture allows `qtiobjtracker` to support multiple tracking strategies within a consistent plugin interface. This makes it easier to tune tracking behavior for different workloads, evaluate alternative algorithms, and optimize implementations for specific hardware or application requirements.

### Input and Output Formats

`qtiobjtracker` operates entirely on object detection metadata and associated coordinates. It does not inspect, analyze, or modify pixel data from video frames. Tracking decisions are based only on the detection metadata received from upstream elements.

For this reason, `qtiobjtracker` must be placed downstream of one or more elements that generate object detections and attach the corresponding metadata.

### Supported Detection Metadata Formats

`qtiobjtracker` supports two input formats for detected objects. Both are commonly used in GStreamer-based AI pipelines.

**1. Structured Text Metadata (`text/x-raw`)**

In this mode, detection results are transmitted separately from video buffers as structured text data.

* buffer caps: `text/x-raw`
* detection results are stored in the buffer payload
* the payload contains a structured description of detected objects
* the text representation can be converted to and from a `GstStructure`
* bounding box coordinates are normalized in the range `[0.0, 1.0]`
* coordinates are resolution-independent

This format allows the same detection data to be reused across streams of different resolutions, including resized or scaled video branches.

**2. ROI Metadata on Video Buffers (`GstROIMeta`)**

In this mode, detection results are attached directly to video buffers as ROI metadata.

* detection results are carried as `GstROIMeta` metadata attached to the original video buffer
* each ROI entry represents one detected object
* bounding box coordinates are expressed in the coordinate space of the video frame (absolute, resolution-dependent)

# Tracking Behavior and Format Handling

`qtiobjtracker` is independent of the underlying video content and relies only on detection metadata for tracking. It supports both structured text metadata and ROI metadata without requiring conversion between the two formats.

The plugin preserves the input metadata representation throughout processing. The output format always matches the input format:

* if the input is `text/x-raw`, the output remains `text/x-raw`
* if the input uses `GstROI` metadata, the output remains ROI metadata attached to the same video buffer

`qtiobjtracker` does not convert between text-based metadata and ROI-based metadata.

### Output Tracking Information

`qtiobjtracker` preserves all input detection metadata and adds a single tracking attribute to each detected object:

* **Unique Track ID** — a persistent identifier used to associate the same object across consecutive frames.

All existing detection attributes, including bounding boxes, class labels, confidence scores, and coordinate representation, are passed through unchanged. The plugin does not modify or extend any other object properties.

The output metadata format always matches the input format. If detections are received as `text/x-raw`, the tracked results are emitted in the same format. If detections are provided as ROI metadata on video buffers, the updated tracking information is attached to the same metadata representation.

# Usage

### Attach Tracking ID to Each Detected Object

This example demonstrates real-time tracking of objects detected by an AI inference pipeline running on a live camera stream. The inference results are attached to each `GstBuffer` as `MLMeta`, after which `qtiobjtracker` tracks the detected objects across frames and adds persistent tracking IDs to the metadata. The resulting AI metadata, including the tracking information, is then serialized into JSON using [`qtimlmetaparser`](/plugin-reference/qtimetaparser) and published to a Redis server through the [`qtiredissink`](/plugin-reference/qtiredissink) plugin.

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"
      scp yolox_w8a8.tflite   <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json         <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolox_w8a8.tflite
    export LABELS_NAME=yolov8.json
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash Run the pipeline theme={null}
    gst-launch-1.0 --gst-debug=2 \
    qtimlvconverter name=stage_01_preproc \
    qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
      external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" name=stage_01_inference \
    qtimlpostprocess name=stage_01_postproc results=10 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME \
    qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! tee name=t \
    t. ! queue ! metamux. \
    t. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! stage_01_inference. stage_01_inference. ! queue ! stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! metamux. \
    qtimetamux name=metamux ! queue ! qtiobjtracker algo=bytetrack ! queue ! qtimlmetaparser module=json ! queue ! qtiredissink host=127.0.0.1 port=6379 channel=ml_results

    #Listen to the published data with Redis CLI from another shell:
    redis-cli SUBSCRIBE ml_results
    ```
  </Step>
</Steps>

### Attach Tracking ID and Propagate to Next Stage AI Inference

This example demonstrates a real-time, multi-stage AI pipeline running on a live camera stream. The first inference stage performs object detection and attaches the results to each `GstBuffer` as `MLMeta`. `qtiobjtracker` then associates the detected objects across frames and adds persistent tracking IDs to the metadata. The video frames, together with the enriched metadata, are passed to a subsequent pose-estimation stage for further inference. Finally, [`qtimetamux`](/plugin-reference/qtimetamux) merges the metadata from all stages, and the overlay stage renders the combined results — including bounding boxes, tracking IDs, and estimated poses — for live display.

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Person/foot detection model | [Qualcomm AI Hub — Person Foot Detection](https://aihub.qualcomm.com/iot/models/foot_track_net) | `foot_track_net_w8a8.tflite` |
    | Person detection labels | <a href="../labels/foot_track_net.json">foot\_track\_net.json</a> | `foot_track_net.json` |
    | Foot track net settings | <a href="../labels/foot_track_net_settings.json">foot\_track\_net\_settings.json</a> | `foot_track_net_settings.json` |
    | HRNet pose model | [Qualcomm AI Hub — HRNet Pose](https://aihub.qualcomm.com/iot/models/hrnet_pose) | `hrnet_pose_w8a8.tflite` |
    | Pose labels | <a href="../labels/hrnet.json">hrnet.json</a> | `hrnet.json` |
    | HRNet settings | <a href="../labels/hrnet_settings.json">hrnet\_settings.json</a> | `hrnet_settings.json` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"
      scp foot_track_net_w8a8.tflite         <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp foot_track_net.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp foot_track_net_settings.json       <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp hrnet_pose_w8a8.tflite             <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp hrnet.json                         <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp hrnet_settings.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME_1=foot_track_net_w8a8.tflite
    export LABELS_NAME_1=foot_track_net.json
    export LABELS_NAME_2=foot_track_net_settings.json
    export MODEL_NAME_2=hrnet_pose_w8a8.tflite
    export LABELS_NAME_3=hrnet.json
    export LABELS_NAME_4=hrnet_settings.json
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash Run the pipeline theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qtimlvconverter name=stage_01_preproc mode=image-batch-non-cumulative \
    qtimltflite name=stage_01_inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_1 \
    qtimlpostprocess name=stage_01_postproc results=10 module=qpd labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_1 settings=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_2 \
    qtimlvconverter name=stage_02_preproc image-disposition=centre mode=roi-batch-cumulative \
    qtimltflite name=stage_02_inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,htp_performance_mode=(string)2;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_2 \
    qtimlpostprocess name=stage_02_postproc results=1 module=hrnet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_2 settings=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_4 \
    qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! tee name=t_split_1 \
    t_split_1. ! queue ! metamux_1. \
    t_split_1. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! stage_01_inference. stage_01_inference. ! queue ! stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! metamux_1. \
    qtimetamux name=metamux_1 ! queue ! qtiobjtracker algo=bytetrack ! queue ! tee name=t_split_2 \
    t_split_2. ! queue ! metamux_2. \
    t_split_2. ! queue ! stage_02_preproc. stage_02_preproc. ! queue ! stage_02_inference. stage_02_inference. ! queue ! stage_02_postproc. stage_02_postproc. ! text/x-raw ! queue ! metamux_2. \
    qtimetamux name=metamux_2 ! queue ! qtivoverlay ! queue ! waylandsink fullscreen=true sync=true async=true
    ```
  </Step>
</Steps>


# qtiredissink
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtiredissink

Redis Pub/Sub sink plugin for real-time text-based metadata publishing

# Overview

The **qtiredissink** is a GStreamer sink element that publishes text-based pipeline output to a Redis server in real time via Redis Pub/Sub. It accepts `text/x-raw` buffers from upstream elements and forwards each buffer as a message to a configured Redis channel.

It is designed for media and AI pipelines where machine learning results, analytics output, or serialized metadata must be delivered to backend systems, dashboards, or distributed consumers with minimal integration effort.

**Supported payload types:**

* Machine learning metadata — detection/classification results, tracking information
* Application-generated messages or serialized custom data
* JSON or other serialized text payloads

**qtiredissink** manages the Redis connection and publishing internally, reducing the application logic needed to bridge a GStreamer pipeline with Redis messaging.

**Core capabilities:**

* **Redis connectivity** — connects to a Redis server using configurable host and port, with optional authentication
* **Channel-based publishing** — sends each incoming buffer to a selected Redis Pub/Sub channel
* **Live data forwarding** — publishes metadata or text output as soon as it arrives

<Tip>
  - `qtiredissink` is a sink element — it terminates the pipeline branch.
  - It accepts only `text/x-raw` input; it is not intended for raw audio or video transport.
  - Messages are delivered only to currently connected subscribers and are **not retained** for later delivery.
  - A valid `channel` must be configured before publishing.
</Tip>

<img alt="qtiredissink_arch" />

## Example Pipeline

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp yolox_w8a8.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=yolox_w8a8.tflite
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    qtimlvconverter ! queue ! \
    qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
      external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
    qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME \
      settings="{\"confidence\": 51.0}" ! text/x-raw ! qtiredissink host=127.0.0.1 port=6379 channel=ml_results

    #Listen to the published data with Redis CLI from another shell:
    redis-cli SUBSCRIBE ml_results
    ```
  </Step>
</Steps>

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />[GstBaseSink](https://gstreamer.freedesktop.org/documentation/base/gstbasesink.html?gi-language=c)<br />
            <Icon icon="arrow-turn-down-right" />qtiredissink

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `text/x-raw` | `format: NA` |
| Availability: *Always* | |
| Direction: *sink* | |

# Element Properties

| Property | Description |
| - | - |
| `channel` | Redis Pub/Sub channel to publish messages to.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `host` | Redis server hostname or IP address.<br /><br />`Type: String`<br />`Default: "127.0.0.1"`<br />`Flags: readable/writable` |
| `password` | Password for Redis authentication.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `port` | Redis server TCP port.<br /><br />`Type: Unsigned Integer`<br />`Default: 6379`<br /> `Range: 0 - 4294967295` <br /> `Flags: readable/writable` |
| `username` | Username for Redis authentication.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |

# Internal Architecture and Messaging Model

**qtiredissink** connects to the configured Redis host and port (with optional authentication), reads each incoming `text/x-raw` buffer, and publishes its contents to the specified channel using the Redis `PUBLISH` command.

Because it uses Pub/Sub, messages are delivered only in real time to currently connected subscribers and are not retained for later delivery. This makes the element suitable for live metadata and event distribution rather than persistent storage.

### Custom Payload Support

**qtiredissink** can transport any custom textual or serialized payload as long as it is `text/x-raw`. This is useful for:

* Sensor or telemetry data serialized as text
* Application-specific metadata such as JSON
* Event notifications and status messages
* Any AI metadata not natively supported by QIM SDK

***

# Usage

<Note>
  Ensure you have followed the [prerequisites](/plugin-reference/qtiredissink#example-pipeline) before continuing
</Note>

### AI pipeline with video overlay and Redis metadata streaming

This pipeline performs object detection on offline video. Inference results are attached to each frame as metadata by [`qtimetamux`](/plugin-reference/qtimetamux), converted to JSON by [`qtimlmetaparser`](/plugin-reference/qtimetaparser), and published to a Redis channel by **qtiredissink**. In parallel, the video stream continues to the display path where detection overlays are rendered on screen. This separates metadata distribution from video rendering, allowing external applications to consume real-time inference results through Redis independently of the displayed video.

<img alt="qtiredissink_arch" />

```bash theme={null}
gst-launch-1.0 filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! tee name=t t. ! queue ! qtimlvconverter ! queue ! qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME ! queue ! qtimlpostprocess module=yolov8 results=6 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! queue ! text/x-raw ! metamux. t. ! qtimetamux name=metamux ! qtivoverlay ! tee name=t_split_4 t_split_4. ! queue ! waylandsink sync=true fullscreen=true t_split_4. ! queue ! qtimlmetaparser module=json ! qtiredissink sync=false async=false channel=ml_results host=127.0.0.1 port=6379
```

Listen to the published data with Redis CLI:

```bash theme={null}
redis-cli SUBSCRIBE ml_results
```

***

### AI pipeline with direct metadata streaming to Redis

This pipeline performs object detection on offline video. The decoded frames pass through the inference and post-processing stages, which produce detection results as structured text. That output is consumed directly by **qtiredissink**, which publishes the inference metadata to a Redis channel — without any video overlay or display path.

<img alt="qtiredissink_arch" />

```bash theme={null}
gst-launch-1.0 -e --gst-debug=2 filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! qtimlvconverter ! queue ! qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME ! queue ! qtimlpostprocess results=5 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! text/x-raw ! queue ! qtiredissink host=127.0.0.1 port=6379 channel=detections
```

Listen to the published data with Redis CLI:

```bash theme={null}
redis-cli SUBSCRIBE detections
```


# qtirtspbin
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtirtspbin

A GstBin-derived sink element that encapsulates RTSP streaming functionality by integrating GStreamer RTSP Server APIs.

# Overview

`qtirtspbin` is a GStreamer bin element that simplifies the construction and management of **RTSP streaming pipelines** for applications that need to publish media over a network. It operates as an **RTSP server**, exposing a local streaming endpoint that RTSP-compatible clients, such as VLC, can connect to in order to receive live or processed media streams.

Rather than functioning as a client that consumes an external RTSP source, `qtirtspbin` provides a server-side streaming interface that allows an application to distribute its output to one or more downstream consumers. This makes it well suited for use cases where media, analytics results, or other generated data must be shared across the network in real time.

The element supports a variety of payload types, including:

* **Encoded video streams** such as H.264 and H.265
* **Audio streams**
* **Machine learning metadata**
* **Serialized custom data**

`qtirtspbin`  can handle multiple input types and automatically assembles the internal pipeline required for stream packetization, transport, and RTSP session handling. By managing these lower-level details internally, it reduces the amount of application-side logic needed to build a complete streaming solution.

In addition to basic RTSP transmission, `qtirtspbin` includes built-in mechanisms for:

* **Session Management**: supports multiple simultaneous client connections and manages each RTSP session
* **Stream synchronization**: preserves consistent timing across video, audio, and metadata streams
* **Custom payload support**: enables transmission of non-standard data types by encapsulating them in RTP payloads

Overall, `qtirtspbin` provides a flexible and developer-friendly abstraction for building RTSP-based streaming applications that need to deliver synchronized media and auxiliary data to network clients.

<img alt="" />

## Example Pipeline

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      scp yolox_w8a8.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    export MODEL_NAME=yolox_w8a8.tflite
    export LABELS_NAME=yolov8.json
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
    v4l2h264enc capture-io-mode=4 output-io-mode=5 ! queue ! \
    h264parse config-interval=1 ! queue ! qtirtspbin address=<ip addr of device>

    Connect to the stream with VLC: rtsp://<ip addr of device>:8900/live
    ```
  </Step>
</Steps>

<Note>
  Ensure that the device has to be connected to network to try out these example pipelines.
</Note>

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />[GstBin](https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c)<br />
            <Icon icon="arrow-turn-down-right" />qtirtspbin

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-h264` | `format: NA` |
| `video/x-h265` | `format: NA` |
| `audio/mpeg` | `format: NA` |
| `text/x-raw` | `format: NA` |
| Availability: *On request* | |
| Direction: *sink* | |
| Pad Name: `sink_%u` | |

# Element Properties

| Property | Description |
| - | - |
| `address` | IP address of the server.<br /><br />`Type: String`<br />`Default: "127.0.0.1"`<br />`Flags: readable/writable` |
| `mode` | Operational mode.<br /><br />`Type: Enum `<br />`Default: 0, "async"`<br />`Range:`<br />    `(0): async - Return buffers immediately if no client is connected`<br />    `(1): sync - Restricts the number of input buffers`<br />`Flags: readable/writable (changeable in NULL, READY)` |
| `mpoint` | Mounting point.<br /><br />`Type: String`<br />`Default: "/live"`<br />`Flags: readable/writable` |
| `port` | Port to listen on.<br /><br />`Type: String`<br />`Default: "8900"`<br />`Flags: readable/writable` |

# Internal Architecture and Streaming model:

`qtirtspbin` is a GStreamer bin element that internally manages the components required to transmit media streams over RTSP. Unlike an RTSP client, it operates as an RTSP server and exposes a mount point that external clients can connect to for receiving the stream

## Core components

Internally, `qtirtspbin` dynamically constructs a pipeline that includes the following elements:

### Input pads

The plugin accepts multiple types of input through its pads, including:

* **Encoded video**, such as H.264 and H.265
* **Audio Streams**
* **ML metadata**, such as JSON
* **Custom serialized data**

### RTP Packetizer

Incoming media and data are converted into RTP packets using the appropriate payloaders, such as:

* `rtph264pay` and `rtph265pay` for video
* `rtpmp4apay` for audio
* `rtpgstpay` for metadata or other structured custom data

### RTSP Server Interface

`qtirtspbin` launches an internal RTSP server, typically on `127.0.0.1:8900`, and registers one or more mount points such as `/live` or `/stream`. Through this interface, it manages:

* Client sessions
* Stream negotiation
* Reconnection behavior

## Processing flow

1. **Media Input**
   * External elements push encoded media or data into `qtirtspbin` through request pads

2. **Stream Identification and Routing**
   * Each incoming stream is identified
   * The stream is routed to the appropriate RTP payloader.

3. **Packetization**
   * Media is packetized into RTP format.
   * Synchronization across streams is handled internally.

4. **Session Management**
   * The plugin registers the stream with the RTSP server.
   * Clients connect using RTSP and receive the live stream.

5. **Client Handling**
   * Multiple clients can connect at the same time.
   * The plugin manages buffering, latency, and session state.

## Custom payload support

In addition to standard media types such as video and audio, `qtirtspbin` can also transport **custom data streams** over RTSP by encapsulating them in RTP payloads.

This is useful for applications that need to deliver non-standard or structured information, such as:

* **Machine learning inference results** like bounding boxes, labels, and scores
* **Sensor data** such as telemetry or depth information
* **Application-specific metadata** such as JSON or other serialized formats

For custom payloads, the flow is typically:

**Input**

Custom data is pushed into `qtirtspbin` through a request pad.

**Payloading**

The plugin uses `rtpgstpay` to encapsulate the incoming buffer into RTP packets.

**Transmission**

The RTP stream is published through the internal RTSP server on the configured mount point.

**Client Consumption**

Clients such as VLC, FFmpeg, or custom RTSP receivers can consume the stream and deserialize the payload based on the agreed format.

# Usage

<Note>
  Ensure you have followed the [prerequisites](/plugin-reference/qtirtspbin#example-pipeline) before continuing
</Note>

### Single stream from camera to the RTSP

Demonstrates real-time RTSP stream reception from live camera source.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e --gst-debug=2 \
qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
v4l2h264enc capture-io-mode=4 output-io-mode=5 ! queue ! \
h264parse config-interval=1 ! queue ! qtirtspbin address=<ip addr of device>

Connect to the stream with VLC: rtsp://<ip addr of device>:8900/live
```

### Two streams from camera to RTSP

Demonstrates usage of rtspbin to stream two different live feeds. Each `qtirtspbin` has internal RTSP server and the client should connect to it. When creating an unique port should be set to each instance of the plugin.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! tee name=camsrc ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! queue ! h264parse config-interval=1 ! queue ! qtirtspbin address=<ip addr of device> port=8900 \
camsrc. ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! queue ! h264parse config-interval=1 ! queue ! qtirtspbin address=<ip addr of device> port=8901

Connect to the stream with VLC: rtsp://<ip addr of device>:8900/live
Connect to the stream with VLC: rtsp://<ip addr of device>:8901/live
```

### Single stream from camera to the RTSP with ML detection

Demonstrates streaming of live camera stream and ML detection metadata on two request pads of the `qtirtspbin`. The video and ML metadata are received on the other pipeline and muxed together to overlay the objects detected and display the bounding boxes.

<img alt="" />

```bash theme={null}
#Console 1:
gst-launch-1.0 -e qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
queue ! tee name=split ! \
  v4l2h264enc capture-io-mode=4 output-io-mode=5 ! queue ! \
  h264parse config-interval=-1 ! queue ! qtirtspbin name=rtsp address=<ip addr of device> \
split. ! queue ! qtimlvconverter ! queue ! \
  qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME ! queue ! \
  qtimlpostprocess results=5 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 75.0}" ! text/x-raw ! queue ! rtsp.

#Console 2:
gst-launch-1.0 -e rtspsrc location=rtsp://<ip addr of device>:8900/live ! rtph264depay ! video/x-h264,colorimetry=bt709 ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! qtimetamux name=metamux ! queue ! qtivoverlay ! waylandsink fullscreen=true async=true sync=true rtspsrc location=rtsp://<ip addr of device>:8900/live ! rtpgstdepay ! text/x-raw ! queue ! metamux.
```


# qtisocketsink
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtisocketsink

Socket Sink plugin

# Overview

The `qtisocketsink`  is a GStreamer sink element that transmits `GstBuffers` between processes over UNIX domain sockets. It is designed for zero-copy buffer sharing between tightly coupled components, such as a media pipeline and an AI inference engine running in a separate process or container.

The element sends FD-backed buffers, such as DMA buffers, to a receiving process that uses qtisocketsrc as first element in pipeline to receive FD-backed buffers. This avoids memory copies, reduces CPU overhead, and improves overall performance.

To use `qtisocketsink`, provide a valid UNIX socket path through the `socket` property. The socket file must use the .sock extension. Buffer lifetime is managed through reference counting mechanism when buffers are sent and returned back.

Supported buffer types include:

* `video/x-raw` for raw video frames
* `neural-network/tensors` for ML tensor data
* `text/x-raw` for textual or structured metadata

`qtisocketsink` is well suited for modular and containerized pipelines where capture, transform, inference, encoding, or visualization run in separate processes. Common use cases include:

* Sending raw camera or decode video frames from capture pipeline running in separate process to another process for encoding/ live streaming, live display, or AI inference.
* Sending AI tensor data `neural-network/tensors` between the pipelines running in separate processes.
* Streaming AI metadata, such as bounding boxes, using `text/x-raw`

By using FD-backed buffers without memory copies, qtisocketsink enables efficient real-time communication between GStreamer components and is well suited for embedded systems, edge AI, and scalable media pipelines.

As illustrated in the diagram, `qtisocketsink` enables the transmission of buffers from a camera source `qticamsrc`, to a separate process or container, where [`qtisocketsrc`](/plugin-reference/qtisocketsrc) receives the buffers for subsequent processing, including encoding, muxing, or storage. This architecture facilitates a modular pipeline design, supports containerized deployment, and enables efficient zero-copy buffer sharing.

<img alt="" />

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />[GstBaseSink](https://gstreamer.freedesktop.org/documentation/base/gstbasesink.html?gi-language=c)<br />
            <Icon icon="arrow-turn-down-right" />qtisocketsink

# Pad Templates

### sink

\| Capabilities            |

| Capabilities | |
| - | - |
| `video/x-raw` | `format: NA` |
| `text/x-raw` | `format: NA` |
| `neural-network/tensors` | `format: NA` |
| Availability: *Always* | |
| Direction: *sink* | |

# Element Properties

| Property | Description |
| - | - |
| `socket` | Location of the Unix Domain Socket.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable (changeable only in NULL or READY state)` |

## Example Pipeline

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    qtisocketsink socket=/tmp/qimsdk.sock

    #To receive data with above pipeline and display, run the Example Pipeline from qtisocketsrc in a different terminal
    ```
  </Step>
</Steps>

# Internal Architecture Details

## Core Components

### Sink Pad

Receives `GstBuffer` objects from upstream pipeline elements.

### Socket Manager

Establishes and manages a UNIX domain socket connection to the consumer process.<br />Transfers buffer file descriptors over the socket.

### Buffer Tracker

Tracks buffer ownership and reference counts after transmission.<br />Prevents buffer release until the receiving process returns ownership.

### Timing Controller

Applies timing-based drop decisions using max-lateness.<br />Throttles outgoing data rate using max-bitrate.

### Async Transition Handler

When `async=false`, the element completes the transition to `PAUSED` synchronously instead of asynchronously.

## Data Flow

### Buffer reception

Buffers are received on the sink pad from upstream elements.

To be transferable over a UNIX domain socket, buffers must be backed by a valid file descriptor, such as DMA-backed memory.

### Socket Transmission

For each eligible buffer, the element sends the associated file descriptor to the receiver process over the UNIX domain socket.

Any required metadata is transmitted either together with the file descriptor or through a separate channel, depending on implementation.

### Buffer Reference Count Management

When a buffer is sent to the receiver, the plugin increments its internal reference tracking.

Once the receiver returns the buffer, the plugin decrements the corresponding reference count.

A buffer is released only after all outstanding references have been returned.

### Timing and Drop Control

If `async=false`, the element performs a synchronous state transition to `PAUSED`.

`max-lateness` defines the maximum allowed delay for a buffer relative to its expected presentation time. Buffers exceeding this threshold are dropped.

`max-bitrate` limits the transmission rate by capping the total number of bits sent per second.

# Usage

<Note>
  Ensure you have followed the [prerequisites](/plugin-reference/qtisocketsink#example-pipeline) before continuing
</Note>

### Encode raw frames in a secondary process to a file

This example demonstrates a camera pipeline executing in one process and transferring raw frames to a second process, which receives the frames, performs encoding, and muxes the resulting stream into an output file.

<img alt="" />

**Start 2 separate shell consoles**

**In 1st console:**

```
gst-launch-1.0 -e --gst-debug=2 qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
qtisocketsink socket=/tmp/docker/socket.sock
```

**In 2nd console:**

```
gst-launch-1.0 -e --gst-debug=2 qtisocketsrc socket=/tmp/docker/socket.sock ! video/x-raw,format=NV12,width=1920,height=1080 ! \
queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! queue ! h264parse ! mp4mux ! queue ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/video.mp4
```

### Render raw frames from a secondary process to display

This example demonstrates a camera pipeline executing in one process and transferring raw frames to a second process, which receives the frames, performs AI inference, Overlay and rendering the composed out to a display.

<img alt="" />

**Start 2 separate shell consoles**

**In 1st console:**

```
gst-launch-1.0 -e --gst-debug=2 rtspsrc location=rtsp://127.0.0.1:8900/live ! rtph264depay ! video/x-h264,colorimetry=bt709 ! h264parse ! \
v4l2h264dec capture-io-mode=4 output-io-mode=4 ! queue ! qtisocketsink socket=/tmp/socket.sock
```

**In 2nd console:**

```
gst-launch-1.0 -e --gst-debug=2 qtisocketsrc socket=/tmp/socket.sock ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! tee name=t \
t. ! queue ! qtimlvconverter ! queue ! \
qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/yolov8_det_w8a8.tflite ! \
qtimlpostprocess module=yolov8 results=5 labels=$HOME/Downloads/qimsdk_samples/labels/yolov8.json settings="{\"confidence\": 70.0}" ! text/x-raw ! metamux. \
t. ! qtimetamux name=metamux ! queue ! qtivoverlay ! waylandsink async=true sync=true fullscreen=true
```


# qtisocketsrc
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtisocketsrc

Socket Source plugin

# Overview

`qtisocketsrc` is a GStreamer source element designed to receive `GstBuffer` objects from another process over a UNIX domain socket.

It is typically used together with [`qtisocketsink`](/plugin-reference/qtisocketsink), forming a pair that enables efficient inter-process communication between separate GStreamer pipelines running in separate processes.

A key feature of `qtisocketsrc` is its support for zero-copy data transfer using file descriptor-backed buffers, such as DMA buffers. Instead of copying media data from one process to another, the buffer ownership is passed through the socket using file descriptors. This significantly reduces memory bandwidth usage and CPU overhead, while improving overall performance and latency!

The element listens on a UNIX socket file, usually with a `.sock` extension, which is specified through the `socket` property. When [`qtisocketsink`](/plugin-reference/qtisocketsink) sends a buffer, the source receives the associated file descriptor, reconstructs it into a `GstBuffer`, and injects it into the downstream GStreamer pipeline. From that point onward, the buffer behaves like any other standard GStreamer buffer and can be consumed by elements such as encoders, inference engines, converters, or display sinks.

This design is especially useful in modular media architectures where different stages of a pipeline are separated into independent processes or containers. For example, a camera capture pipeline can run in one process, while an AI inference pipeline or video encoding pipeline runs in another. `qtisocketsrc` acts as the bridge between them, allowing media to flow efficiently without requiring shared memory copies or tight coupling between components.

The element supports multiple data types, making it flexible for different workloads:

* `video/x-raw` — for raw video frames
* `neural-network/tensors` — for tensor data used in machine learning workflows
* `text/x-raw` — for textual metadata or structured annotations.

Because of this flexibility, `qtisocketsrc` can be used not only for video transport, but also for AI and metadata pipelines. It can receive raw camera frames for further processing, ingest tensor outputs from an inference stage for post-processing, or consume metadata such as bounding boxes, labels, or classifications for overlay and logging.

In practical deployments, this makes `qtisocketsrc` valuable in embedded systems, edge AI applications, and containerized media platforms where scalability, isolation, and performance are important. By combining zero-copy transport with process isolation, it enables high-throughput, low-latency communication between GStreamer components while preserving a clean and modular pipeline design.

As shown in the diagram below [`qtisocketsink`](/plugin-reference/qtisocketsink) can be used to transmit buffers from a camera source such as qtqmmfsrc to another process or container. On the receiving side, qtisocketsrc ingests these buffers and continues pipeline processing, such as encoding, muxing, or saving the output to storage. This approach enables a modular pipeline architecture, supports containerized deployment, and provides efficient zero-copy buffer sharing between isolated components.

<img alt="Usecase 1" />

## Example Pipeline

<img alt="" />

<Steps>
  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qtisocketsrc socket=/tmp/qimsdk.sock ! video/x-raw,width=1920,height=1080,format=NV12 ! queue ! \
    waylandsink fullscreen=true sync=true

    #To receive data with above pipeline and display, run the Example Pipeline from qtisocketsink in a different terminal
    ```
  </Step>
</Steps>

# Hieararchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />[GstBaseSrc](https://gstreamer.freedesktop.org/documentation/base/gstbasesrc.html?gi-language=c)<br />
            <Icon icon="arrow-turn-down-right" />[GstPushSrc](https://gstreamer.freedesktop.org/documentation/base/gstpushsrc.html?gi-language=c)<br />
               <Icon icon="arrow-turn-down-right" />qtisocketsrc

# Pad Templates

### src

| Capabilities | |
| - | - |
| `video/x-raw` | `format: NA` |
| `text/x-raw` | `format: NA` |
| `neural-network/tensors` | `format: NA` |
| Availability: *Always* | |
| Direction: *source* | |

# Element Properties

| Property | Description |
| - | - |
| `socket` | Location of the Unix Domain Socket.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable (changeable only in NULL or READY state)` |
| `timeout` | Socket post timeout.<br /><br />`Type: Unsigned Integer64`<br />`Default: 1000`<br />`Range: 0 - 18446744073709551615`<br />`Flags: readable/writable (changeable only in NULL or READY state)` |

# Internal Architecture

The internal architecture of `qtisocketsrc` is built around a lightweight but efficient mechanism for receiving media buffers from another process and delivering them into a GStreamer pipeline. Its design focuses on preserving performance, minimizing memory copies, and keeping the transfer of data between processes seamless and reliable.

## Core Components

### Source Pad

The source pad is the output interface of `qtisocketsrc`. Once a buffer is received and reconstructed, it is pushed out through this pad so that downstream GStreamer elements can consume it. In practice, the source pad serves as the handoff point between the socket-receiving logic and the rest of the media pipeline.

### Socket Listener

The socket listener is responsible for opening, monitoring, and maintaining the UNIX domain socket specified by the `socket` property. It waits for incoming connections from the sending side, typically [`qtisocketsink`](/plugin-reference/qtisocketsink), and listens for buffer transmissions over this local IPC channel.

This component acts as the communication entry point for the plugin. It ensures that the source element can continuously receive data from a peer process without requiring shared memory copying or network-based transport.

### Buffer Importer

The buffer importer takes the received file descriptors and converts them into `GstBuffer` objects that can be used by GStreamer. Since the incoming data is FD-backed, the importer does not need to duplicate the actual media payload. Instead, it wraps the descriptor and makes the buffer available to the pipeline in a form that downstream elements understand.

If additional information such as caps, timestamps, or other metadata is available, the importer can attach that information to the buffer so the receiving pipeline maintains proper media context and synchronization.

### Buffer Lifecycle Manager

The buffer lifecycle manager is responsible for tracking buffer ownership, usage, and release. Because buffers may be shared across process boundaries, careful management is needed to ensure that resources are not leaked and are released at the correct time.

In some cases, the lifecycle manager may also support returning buffers to the sender once downstream processing is complete, depending on the buffer-handling strategy used by the pipeline.

## Data Flow

The data flow in `qtisocketsrc` follows a clear sequence from socket setup to buffer delivery. Each step is designed to keep the transfer efficient while preserving the semantics of the original media stream.

### Socket Initialization

At startup, the plugin opens the UNIX socket specified through the `socket` property. It then begins waiting for incoming connections from the remote process.

This stage establishes the IPC channel that will be used for media exchange. Once the connection is ready, the element is prepared to receive file descriptors and any associated control information sent by the peer.

### Buffer Reception

When the sender process transmits data through [`qtisocketsink`](/plugin-reference/qtisocketsink), `qtisocketsrc` receives the FD-backed buffers over the socket. These buffers may represent:

* raw video frames
* ML tensor data
* structured metadata or text-based annotations

At this stage, the source element is only receiving the transport representation of the buffer, not copying the underlying memory contents.

### Buffer Wrapping

After reception, each file descriptor is wrapped into a `GstBuffer`. This step transforms the incoming IPC payload into a native GStreamer object that downstream plugins can process.

If available, the plugin also applies buffer metadata such as caps, timestamps, or other stream attributes. This ensures that the received data remains properly formatted and synchronized within the pipeline.

### Buffer Output

Once the `GstBuffer` is prepared, it is pushed out through the source pad into the downstream portion of the pipeline.

From this point onward, the data can be consumed by elements such as:

* encoders
* inference models
* converters
* display sinks
* post-processing components

This allows the receiving pipeline to continue processing the media stream as if the data had originated locally.

# Usage

<Note>
  Ensure you have followed the [prerequisites](/plugin-reference/qtisocketsrc#example-pipeline) before continuing
</Note>

### Transmitting camera stream through a Docker container for encoding

<img alt="" />

<Steps>
  <Step title="Console 1 — Receive from container and encode to file">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 qtisocketsrc socket=/tmp/docker/output.sock ! \
    video/x-raw,format=NV12,width=1920,height=1080 ! \ 
    v4l2h264enc output-io-mode=dmabuf-import capture-io-mode=dmabuf ! \
    h264parse ! mp4mux ! queue ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/video.mp4
    ```
  </Step>

  <Step title="Console 2 — Run Docker container to bridge sockets">
    ```bash theme={null}
    docker run -v /tmp/docker:/tmp/docker -ti gst-image /bin/bash -c \
    "gst-launch-1.0 -e --gst-debug=2 qtisocketsrc socket=/tmp/docker/input.sock ! \ 
    qtisocketsink socket=/tmp/docker/output.sock"
    ```
  </Step>

  <Step title="Console 3 — Capture camera and send to container">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 \
    ! qtisocketsink socket=/tmp/docker/input.sock
    ```
  </Step>
</Steps>


# qtivcomposer
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtivcomposer

Video composition pipelines plugin

# Overview

**qtivcomposer** is a video composition element that combines multiple video streams into a single output frame using GPU hardware acceleration.

At a fundamental level, video composition involves taking multiple independent video frames and arranging them into one output frame according to defined spatial rules. These rules determine the position of each input frame and, when necessary, resize it to fit its destination window in the output frame. Resizing may involve either downscaling or upscaling, depending on the source and destination rectangles.

**qtivcomposer** is commonly used in scenarios such as:

* Multi-camera systems, such as surround-view and security grid layouts
* Picture-in-picture compositions
* Video wall applications
* Overlaying auxiliary video streams onto a primary video feed

The element is optimized for real-time, low-latency pipelines and is designed to operate efficiently on platforms that provide GPU or dedicated hardware composition engines.

Unlike software-based compositors, which rely on CPU memory copies and pixel-by-pixel processing, **qtivcomposer** leverages hardware acceleration to reduce memory bandwidth usage and lower end-to-end latency.

By enforcing strict buffer requirements and separating layout definition from execution, the plugin provides a hardware-accelerated, deterministic, and scalable solution for video composition in GStreamer pipelines.

<img alt="" />

## Example Pipeline

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/media/output
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! tee name=t \
    t. ! queue ! qtivcomposer name=mixer sink_0::dimensions="<960,540>" sink_1::dimensions="<960,540>" ! \
    queue ! waylandsink fullscreen=true sync=true \
    t. ! queue ! mixer.
    ```
  </Step>
</Steps>

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />[GstAggregator](https://gstreamer.freedesktop.org/documentation/base/gstaggregator.html?gi-language=c#GstAggregator)<br />
            <Icon icon="arrow-turn-down-right" />[GstVideoAggregator](https://gstreamer.freedesktop.org/documentation/video/gstvideoaggregator.html?gi-language=c#GstVideoAggregator)<br />
               <Icon icon="arrow-turn-down-right" />qtivcomposer

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { NV12, NV21, UYVY, YUY2, P010_10LE, RGBA, BGRA, ARGB, ABGR, RGBx, BGRx, xRGB, xBGR, RGB, BGR, GRAY8, NV12_Q08C }` <br /> `width: [1, 32767]` <br /> `height: [1, 32767]` <br /> `framerate: [0/1, 255/1]` |
| Availability: *On request* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { NV12, NV21, UYVY, YUY2, P010_10LE, RGBA, BGRA, ARGB, ABGR, RGBx, BGRx, xRGB, xBGR, RGB, BGR, GRAY8, NV12_Q08C }` <br /> `width: [1, 32767]` <br /> `height: [1, 32767]` <br /> `framerate: [0/1, 255/1]` |
| Presence: *Always* | |
| Direction: *source* | |

# Pad Properties

### sink

| Property | Description |
| - | - |
| `alpha` | Alpha channel value.<br /><br />`Type: Double`<br />`Default: 1`<br />`Range: 0 - 1`<br />`Flags: readable/writable` |
| `crop` | The crop rectangle in the format `<X, Y, WIDTH, HEIGHT>`.<br /><br />`Type: GstValueArray of type gint`<br />`Default: "<0, 0, 0, 0>"`<br />`Flags: readable/writable` <br /> `Example: crop="<0,0,1280,720>"` |
| `dimensions` | The destination rectangle width and height. If not set, it remains the same as input dimensions in the format `<WIDTH, HEIGHT>`.<br /><br />`Type: GstValueArray of type gint`<br />`Default: "<   >"`<br />`Flags: readable/writable` <br /> `Example: crop="<1280,720>"` |
| `flip-horizontal` | Flips the video frame horizontally.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `flip-vertical` | Flips the video frame vertically.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `position` | The X and Y coordinates of the destination rectangle's top-left corner in the format `<X, Y>`.<br /><br />`Type: GstValueArray of type gint`<br />`Default: "<   >"`<br />`Flags: readable/writable` <br /> `Example: position="<10,20>"` |
| `rotate` | Specifies rotation to apply to the video.<br /><br />`Type: Enum `<br />`Default: 0, "none"`<br />`Range:`<br />    `(0): none`<br />    `(1): 90CW - Rotate 90 degrees clockwise`<br />    `(2): 90CCW - Rotate 90 degrees counter-clockwise`<br />    `(3): 180 - Rotate 180 degrees`<br />`Flags: readable/writable` <br /> `Example: rotate="90CW" (or) rotate=1` |
| `zorder` | Z-axis order. By default, follows the order of creation.<br /><br />`Type: Integer`<br />`Default: -1`<br />`Range: -1 - 2147483647`<br />`Flags: readable/writable` |

# Element Properties

| Property | Description |
| - | - |
| `background` | Background color.<br /><br />`Type: Unsigned Integer`<br />`Default: 4286611584`<br />`Range: 0 - 4294967295`<br />`Flags: readable/writable` |
| `engine` | Engine backend used for the conversion operations.<br /><br />`Type: Enum `<br />`Default: 2, "gles"`<br />`Range:`<br />    `(0): none - No backend used`<br />    `(2): gles - Use OpenGLES based video converter`<br />    `(3): fcv - Use FastCV based video converter`<br />`Flags: readable/writable`   <br /> `Example: engine="gles" (or) engine=2` |

# Architecture, Processing, and Role in Vision AI

**qtivcomposer** is a multi-sink GStreamer element with one source pad. Each sink pad accepts one input stream, typically through dynamically requested pads such as `sink_0`, `sink_1`, and so on. The source pad produces the final composed frame.

Internally, the element includes four main parts: a pad manager that tracks active inputs, a layout configuration that defines where and how each input appears in the output, a hardware composition backend that performs the composition, and a buffer manager that enforces DMA-compatible memory usage.

At setup time, caps negotiation defines the output frame, including resolution and color format. Input streams must be compatible with the compositor. When needed, hardware-supported scaling and format conversion are applied during composition.

Each sink pad is configured with layout metadata that specifies:

* Output position `(x, y)`
* Output size `(width, height)`
* Z-order

These settings are typically provided through element or per-pad properties. The properties are parsed into an internal layout structure that describes the final output frame independently of buffer arrival order. This separation keeps composition deterministic and reproducible.

During runtime, each sink pad receives buffers independently. To generate a valid output frame, **qtivcomposer** synchronizes inputs by presentation time and ensures that all required frames are available. If a frame is missing or delayed, the element may reuse the previous frame, drop the output frame, or stall, depending on configuration.

**qtivcomposer** operates only on DMA-backed buffers. This allows the hardware backend to access memory directly without CPU copies, enabling zero-copy operation, hardware interoperability, and predictable latency. If upstream does not provide suitable buffers, the element advertises its own buffer pool through GStreamer allocation negotiation. If compatible allocation cannot be established, pipeline setup fails rather than falling back to software memory.

Once the required input buffers are ready, the hardware backend reads each frame, applies scaling, positioning, and any supported blending, and draws the inputs onto the output surface in Z-order. The composed frame is then pushed downstream to encoders, display sinks, or network outputs with a consistent format and timeline.

In multi-stream vision AI pipelines, **qtivcomposer** is not only a display component but also a performance optimization stage. It combines multiple AI-processed streams into a single output surface, reducing memory bandwidth pressure, CPU overhead, GPU context switching, and cross-stream synchronization cost. In a typical QIM SDK pipeline, each stream is decoded, preprocessed, inferred, and post-processed independently, with overlays such as boxes or labels applied per stream. The streams are merged only at the final stage by **qtivcomposer**, which preserves parallel processing while making final output efficient.

# Usage

### Side-by-side composition of two camera streams

This pipeline uses `qtivcomposer` to combine two NV12 camera streams into a single output frame arranged side by side, with the resulting output displayed on the screen.

<img alt="" />

```
gst-launch-1.0 \
  qtivcomposer name=mixer \
  sink_0::position="<0, 0>" sink_0::dimensions="<960, 540>" \
  sink_1::position="<960, 0>" sink_1::dimensions="<960, 540>" \
  ! waylandsink fullscreen=true async=true sync=true \
  qticamsrc camera=0 ! video/x-raw,format=NV12,width=1280,height=720 ! mixer. \
  qticamsrc camera=1 ! video/x-raw,format=NV12,width=1280,height=720 ! mixer.
```

### Picture-in-picture (PiP) composition of two camera streams

This example uses qtivcomposer to overlay a secondary video stream on top of a primary stream, creating a PiP layout in a single output frame.

<img alt="" />

```
gst-launch-1.0 \
  qtivcomposer name=mixer \
  sink_0::position="<0, 0>" sink_0::dimensions="<1920, 1080>" \
  sink_1::position="<1180, 620>" sink_1::dimensions="<640, 360>" \
  ! waylandsink fullscreen=true async=true sync=true \
  qticamsrc camera=0 ! video/x-raw,format=NV12,width=1920,height=1080 ! mixer. \
  qticamsrc camera=1 ! video/x-raw,format=NV12,width=1920,height=1080 ! mixer.
```

### ML result overlay

This example demonstrates how qtivcomposer can be used to overlay an image mask or other ML-generated visual output on top of a video stream. The pipeline splits the input video into two branches: one branch preserves the original video, while the second branch generates a visualization of the ML inference results. qtivcomposer then blends the two streams using alpha compositing, producing a single output stream that combines the original content with the ML overlay.
This approach provides an efficient way to visualize inference results directly in the video pipeline and can be used as an alternative to solutions based on qtimetamux and qtivoverlay when the ML output is available as a video frame or image mask.

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | DeepLabV3-Plus-MobileNet W8A8 model | [Qualcomm AI Hub — DeepLabV3-Plus-MobileNet](https://aihub.qualcomm.com/iot/models/deeplabv3_plus_mobilenet?searchTerm=DeepLabV3-Plus-MobileNet) | `deeplabv3_plus_mobilenet_w8a8.tflite` |
    | Detection labels | <a href="../labels/dv3-argmax.json">dv3-argmax.json</a> | `dv3-argmax.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp deeplabv3_plus_mobilenet_w8a8.tflite   <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp dv3-argmax.json                        <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4                     <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export MODEL_NAME=deeplabv3_plus_mobilenet_w8a8.tflite
    export LABELS_NAME=dv3-argmax.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! tee name=split \
    split. ! queue ! qtivcomposer name=mixer sink_1::alpha=0.5 ! video/x-raw,format=NV12 ! queue ! waylandsink sync=true fullscreen=true \
    split. ! queue ! qtimlvconverter ! queue ! \
    qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME ! queue ! \
    qtimlpostprocess module=deeplab-argmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw ! queue ! mixer.
    ```
  </Step>
</Steps>

### Multi‑Stream Vision AI Video Wall

This example demonstrates how `qtivcomposer` serves as the final aggregation point for building an video wall, where each stream runs its own AI inference pipeline, applies AI Metadata as overlay, and is then composed into a single output to create the final video wall.

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLO model | [Qualcomm Model](https://aihub.qualcomm.com/iot/models/yolox) | `yolov8_det_w8a8.tflite` |
    | YOLO labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"
      scp yolov8_det_w8a8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/models $HOME/Downloads/qimsdk_samples/labels
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -v qtivcomposer background=0x000000FF name=mix \
    sink_0::position="<0,     0>" sink_0::dimensions="<640,  360>" \
    sink_1::position="<640,   0>" sink_1::dimensions="<640,  360>" \
    sink_2::position="<1280,  0>" sink_2::dimensions="<640,  360>" \
    sink_3::position="<0,    360>" sink_3::dimensions="<640, 360>" \
    sink_4::position="<640,  360>" sink_4::dimensions="<640, 360>" \
    sink_5::position="<1280, 360>" sink_5::dimensions="<640, 360>" \
    sink_6::position="<0,    720>" sink_6::dimensions="<640, 360>" \
    sink_7::position="<640,  720>" sink_7::dimensions="<640, 360>" \
    sink_8::position="<1280, 720>" sink_8::dimensions="<640, 360>" \
    sink_9::position="<0,      0>" sink_9::dimensions="<640,  360>" \
    sink_10::position="<640,   0>"  sink_10::dimensions="<640, 360>" \
    sink_11::position="<1280,  0>"  sink_11::dimensions="<640, 360>" \
    sink_12::position="<0,    360>" sink_12::dimensions="<640, 360>" \
    sink_13::position="<640,  360>" sink_13::dimensions="<640, 360>" \
    sink_14::position="<1280, 360>" sink_14::dimensions="<640, 360>" \
    sink_15::position="<0,    720>" sink_15::dimensions="<640, 360>" \
    sink_16::position="<640,  720>" sink_16::dimensions="<640, 360>" \
    sink_17::position="<1280, 720>" sink_17::dimensions="<640, 360>" \
    mix. ! queue ! fpsdisplaysink text-overlay=false sync=true video-sink="waylandsink fullscreen=true async=true sync=true" \
    rtspsrc location=rtsp://admin:qualcomm1@192.168.0.14:554/Streaming/Channels/101 ! queue ! rtpptdemux ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw, format=NV12 ! queue ! tee name=t0  ! queue ! mix. \
    rtspsrc location=rtsp://admin:qualcomm1@192.168.0.14:554/Streaming/Channels/101 ! queue ! rtpptdemux ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw, format=NV12 ! queue ! tee name=t1  ! queue ! mix. \
    rtspsrc location=rtsp://admin:qualcomm1@192.168.0.14:554/Streaming/Channels/101 ! queue ! rtpptdemux ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw, format=NV12 ! queue ! tee name=t2  ! queue ! mix. \
    rtspsrc location=rtsp://admin:qualcomm1@192.168.0.14:554/Streaming/Channels/101 ! queue ! rtpptdemux ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw, format=NV12 ! queue ! tee name=t3  ! queue ! mix. \
    rtspsrc location=rtsp://admin:qualcomm1@192.168.0.14:554/Streaming/Channels/101 ! queue ! rtpptdemux ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw, format=NV12 ! queue ! tee name=t4  ! queue ! mix. \
    rtspsrc location=rtsp://admin:qualcomm1@192.168.0.14:554/Streaming/Channels/101 ! queue ! rtpptdemux ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw, format=NV12 ! queue ! tee name=t5  ! queue ! mix. \
    rtspsrc location=rtsp://admin:qualcomm1@192.168.0.14:554/Streaming/Channels/101 ! queue ! rtpptdemux ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw, format=NV12 ! queue ! tee name=t6  ! queue ! mix. \
    rtspsrc location=rtsp://admin:qualcomm1@192.168.0.14:554/Streaming/Channels/101 ! queue ! rtpptdemux ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw, format=NV12 ! queue ! tee name=t7  ! queue ! mix. \
    rtspsrc location=rtsp://admin:qualcomm1@192.168.0.14:554/Streaming/Channels/101 ! queue ! rtpptdemux ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw, format=NV12 ! queue ! tee name=t8  ! queue ! mix. \
    split0.  ! videorate drop-only=true ! video/x-raw,framerate=15/1 ! queue ! qtimlvconverter ! queue ! qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/yolov8_det_w8a8.tflite ! queue ! qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/yolov8.json settings="{\"confidence\": 50.0}" ! queue ! mix. \
    split1.  ! videorate drop-only=true ! video/x-raw,framerate=15/1 ! queue ! qtimlvconverter ! queue ! qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/yolov8_det_w8a8.tflite ! queue ! qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/yolov8.json settings="{\"confidence\": 50.0}" ! queue ! mix. \
    split2.  ! videorate drop-only=true ! video/x-raw,framerate=15/1 ! queue ! qtimlvconverter ! queue ! qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/yolov8_det_w8a8.tflite ! queue ! qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/yolov8.json settings="{\"confidence\": 50.0}" ! queue ! mix. \
    split3.  ! videorate drop-only=true ! video/x-raw,framerate=15/1 ! queue ! qtimlvconverter ! queue ! qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/yolov8_det_w8a8.tflite ! queue ! qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/yolov8.json settings="{\"confidence\": 50.0}" ! queue ! mix. \
    split4.  ! videorate drop-only=true ! video/x-raw,framerate=15/1 ! queue ! qtimlvconverter ! queue ! qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/yolov8_det_w8a8.tflite ! queue ! qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/yolov8.json settings="{\"confidence\": 50.0}" ! queue ! mix. \
    split5.  ! videorate drop-only=true ! video/x-raw,framerate=15/1 ! queue ! qtimlvconverter ! queue ! qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/yolov8_det_w8a8.tflite ! queue ! qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/yolov8.json settings="{\"confidence\": 50.0}" ! queue ! mix. \
    split6.  ! videorate drop-only=true ! video/x-raw,framerate=15/1 ! queue ! qtimlvconverter ! queue ! qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/yolov8_det_w8a8.tflite ! queue ! qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/yolov8.json settings="{\"confidence\": 50.0}" ! queue ! mix. \
    split7.  ! videorate drop-only=true ! video/x-raw,framerate=15/1 ! queue ! qtimlvconverter ! queue ! qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/yolov8_det_w8a8.tflite ! queue ! qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/yolov8.json settings="{\"confidence\": 50.0}" ! queue ! mix. \
    split8.  ! videorate drop-only=true ! video/x-raw,framerate=15/1 ! queue ! qtimlvconverter ! queue ! qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/yolov8_det_w8a8.tflite ! queue ! qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/yolov8.json settings="{\"confidence\": 50.0}" ! queue ! mix.
    ```
  </Step>
</Steps>


# qtivoverlay
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtivoverlay

Hardware-accelerated in-place video overlay plugin

# Overview

The **qtivoverlay** element is a hardware-accelerated, in-place drawing and blitting plugin designed to render visual overlay objects directly on top of incoming video frames, such as YUV or RGB buffers. By compositing overlays directly onto the frame data, it enables the processed video to be displayed on a screen or forwarded for encoding with minimal additional overhead. This makes it well suited for real-time video applications where performance and low latency are important.

The element supports a wide range of overlay content, including user-defined graphics and annotations such as logos, bounding boxes, custom labels, date and time, buffer timestamps, privacy masks, and other visual indicators. In addition, **qtivoverlay** can render dynamic overlay information carried in **GstMeta**, which is commonly used to attach AI or analytics results to each frame for post-processing and visualization.

To achieve efficient overlay rendering, **qtivoverlay** combines CPU-based drawing with GPU-based blending:

* **CPU rendering with Cairo**: Overlay content is first drawn using the open-source Cairo graphics library into compact, memory-efficient overlay buffers.
* **GPU hardware blending**: These rendered overlay buffers are then blended with the main video frame using GPU hardware acceleration.

This hybrid approach helps reduce memory usage and improves overall performance, especially in pipelines handling high-resolution or high-frame-rate video streams.

The overlays supported by the element can be described in two ways:

**Through element properties**

These are overlays configured manually by the user, including:

* Static images or logos
* Bounding boxes
* Date and/or time
* Buffer timestamps
* Custom text
* Privacy masks

**Through buffer metadata**

These overlays are attached to each input frame as metadata, and are typically added by the **qtimetamux** plugin. This metadata is used to draw machine-learning-related overlays such as:

* Detection overlays, including bounding boxes
* Segmentation overlays, such as semantic masks or mask images
* Classification overlays, such as labels or user text
* Pose graph overlays

<Tip>In summary, qtivoverlay is a hardware-accelerated, in-place image drawing and blitting plugin for overlaying visual annotations on video frames. It supports both manually configured overlays and metadata-driven overlays, making it suitable for use cases such as video analytics, AI inference visualization, and privacy masking.</Tip>

<img alt="qtivoverlay_arch" />

## Example Pipeline

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp yolox_w8a8.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/models $HOME/Downloads/qimsdk_samples/labels $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/media/output
    export MODEL_NAME=yolox_w8a8.tflite
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
    t. ! queue ! qtimlvconverter ! queue ! \
    qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
      external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
    qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME \
      settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! obj_mux.
    ```
  </Step>
</Steps>

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />[GstBaseTransform](https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c)<br />
            <Icon icon="arrow-turn-down-right" />qtivoverlay

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { NV12, NV21, YUY2, RGBA, BGRA, ARGB, ABGR, RGBx, BGRx, xRGB, xBGR, RGB, BGR, NV12_Q08 }` <br /> `width: [1, 2147483647]` <br /> `height: [1, 2147483647]` <br /> `framerate: [0/1, 2147483647/1]` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { NV12, NV21, YUY2, RGBA, BGRA, ARGB, ABGR, RGBx, BGRx, xRGB, xBGR, RGB, BGR, NV12_Q08 }` <br /> `width: [1, 2147483647]` <br /> `height: [1, 2147483647]` <br /> `framerate: [0/1, 2147483647/1]` |
| Availability: *Always* | |
| Direction: *source* | |

# Element Properties

| Property | Description |
| - | - |
| `bboxes` | Manually set multiple custom bounding boxes as a list of GstStructures with a unique name and three parameters: `position`, `dimensions`, and `color`. The `position` and `dimensions` are mandatory for new entries.<br /><br />`Type: String`<br />`Default: "{ }"`<br />`Flags: readable/writable`<br />`Example:`<br />`{(structure)\"Box1,position=<100,100>,dimensions=<640,480>;\", (structure)\"Box2,position=<1000,100>,dimensions=<300,300>,color=0xFF0000FF;\"}` |
| `images` | Manually set multiple custom BGRA images as a list of GstStructures with a unique name and three mandatory parameters: `path`, `resolution`, and `destination`.<br /><br />`Type: String`<br />`Default: "{ }"`<br />`Flags: readable/writable`<br />`Example:`<br />`{(structure)\"Image1,path=/data/image1.bgra,resolution=<480,360>,destination=<0,0,640,480>;\", (structure)\"Image2,path=/data/image2.bgra,resolution=<240,180>,destination=<100,100,480,360>;\"}` |
| `masks` | Manually set multiple masks as a list of GstStructures with a unique name and parameters `color` and either `circle=<X, Y, RADIUS>` or `rectangle=<X, Y, WIDTH, HEIGHT>` (polygon also supported).<br /><br />`Type: String`<br />`Default: "{ }"`<br />`Flags: readable/writable`<br />`Example:`<br />`{(structure)\"Mask1,color=0xRRGGBBAA,circle=<400,400,200>;\",(structure)\"Mask2,color=0xRRGGBBAA,rectangle=<0,0,20,10>;\",(structure)\"Mask3,color=0xRRGGBBAA,polygon=<<2,2>,<2,4>,<4,4>>;\"}` |
| `strings` | Manually set multiple custom strings as a list of GstStructures with a unique name and four parameters: `contents`, `fontsize`, `position`, and `color`. The `contents` field is mandatory for new entries.<br /><br />`Type: String`<br />`Default: "{ }"`<br />`Flags: readable/writable`<br />`Example:`<br />`{(structure)\"Text1,contents=\\\"Example\ 1\\\",fontsize=12,position=<0,0>,color=0xRRGGBBAA;\"}` |
| `timestamps` | Manually set timestamps using GstStructures. Use `Date/Time` for displaying formatted date and time with optional parameters `format`, `fontsize`, `position`, and `color`. Use `PTS/DTS` for displaying buffer timestamps with optional parameters `fontsize`, `position`, and `color`.<br /><br />`Type: String`<br />`Default: "{ }"`<br />`Flags: readable/writable`<br />`Example:`<br />`{(structure)\"Date/Time,format=\\\"%d/%m/%Y\ %H:%M:%S\\\",fontsize=12,position=<0,0>,color=0xRRGGBBAA;\", (structure)\"PTS/DTS,fontsize=12,position=<0,0>,color=0xRRGGBBAA;\}` |

# Metadata (AI Post-Process Information)

Incoming video buffers may contain additional metadata in the form of **GstMeta**. The element inspects each buffer for supported metadata types and renders any detected metadata directly on top of the corresponding image frame. Each supported metadata type is processed and displayed according to its own visual representation.

## Supported Metadata Types

### GstVideoRegionOfInterestMeta

This metadata describes a region of interest (ROI) within the frame, typically representing an object such as a person, vehicle, keyboard, or any other detected item.

A rectangle is drawn around the ROI using the specified X/Y coordinates, width, and height. The rectangle is rendered with a visible border and a transparent interior so that the object inside the region remains visible.

<img alt="GstVideoRegionOfInterestMeta" />

### GstVideoLandmarksMeta

This metadata describes a collection of landmark points that may represent human pose key-points, facial features, hand joints, or other connected points of interest.

Both the individual points and the lines connecting related points are drawn on the image, allowing the landmark structure to be visualized clearly.

<img alt="GstVideoLandmarksMeta" />

### GstVideoClassificationMeta

This metadata contains a list of possible labels or classifications for the entire image.

All associated labels are rendered in the top-left corner of the frame, making the classification results easy to view at a glance.

<img alt="GstVideoClassificationMeta" />

### GstCvOptclFlowMeta

This metadata represents optical flow motion vectors, which describe the movement between pixels across two sequential video frames.

The vectors indicate motion direction and magnitude, making it useful for visualizing frame-to-frame movement in the video stream.

<img alt="GstCvOptclFlowMeta" />

# User-Defined Overlay Objects via Properties

In addition to buffer metadata, the plugin supports user-defined overlay objects that can be configured through properties. These overlays can be added, updated, or removed during runtime. Once configured, each overlay remains visible on every incoming video frame until the user explicitly removes it.

## Supported Property Types

### Static Images

Static images such as logos, icons, or watermarks can be added using the **images** property.
All images provided through this property must be in raw **RGBA** format.

The property payload follows the GStreamer structure layout and requires the following mandatory parameters when first set:

| Parameter | Description |
| - | - |
| `path` | URL or path to the image file |
| `resolution` | Width and height of the image in pixels |
| `destination` | Top-left X/Y coordinates where the image should be placed, along with its final rendered size |

```bash theme={null}
"{(structure)"Image1,path=/data/logo1.rgba,resolution=<480,360>,destination=<0,0,640,480>;", (structure)"Image2,path=/data/logo2.rgba,resolution=<240,180>,destination=<100,100,480,360>;"}"
```

| | Description |
| - | - |
| **Image1** | Resolution 480×360, destination coordinates X=0, Y=0, rendered size 640×480 |
| **Image2** | Resolution 240×180, destination coordinates X=100, Y=100, rendered size 480×360 |

<img alt="StaticImage" />

### Custom Text

Text overlays such as captions, titles, annotations, or labels can be set using the **strings** property.

The property payload follows the GStreamer structure layout and requires the following parameters when first set:

| Parameter | Description |
| - | - |
| `contents` | Text content to be displayed |
| `fontsize` | Font size of the text |
| `position` | Top-left X/Y coordinates where the text should appear |
| `color` | Text color in RGBA hex format |

```bash theme={null}
"{(structure)"Text1,contents=\"Example 1\",fontsize=48,position=<50,50>,color=0x0000FFFF;", (structure)"Text2,contents=\"Example 2\",fontsize=64,position=<1620,520>,color=0xFF0000FF;"}"
```

| | Description |
| - | - |
| **Text1** | Font size 48, position X=50, Y=50, blue color |
| **Text2** | Font size 64, position X=1620, Y=520, red color |

<img alt="CustomText" />

### Timestamps

Timestamps such as the current date/time or buffer timestamps can be displayed using the **timestamps** property.

The property payload follows the GStreamer structure layout and requires the following parameters when first set:

| Parameter | Description |
| - | - |
| `format` | Date/time formatting string. Not applicable for `PTS/DTS` |
| `fontsize` | Font size of the timestamp |
| `position` | Top-left X/Y coordinates where the timestamp should be placed |
| `color` | Timestamp color in RGBA hex format |

```bash theme={null}
"{(structure)"Date/Time,format=\"%d/%m/%Y %H:%M:%S\",fontsize=48,position=<50,50>,color=0xFFFFFFFF;", (structure)"PTS/DTS,fontsize=64,position=<50,920>,color=0xFF0000FF;"}"
```

| | Description |
| - | - |
| **Date/Time** | Format DAY/MONTH/YEAR HOUR:MIN:SEC, font size 48, position X=50, Y=50, white color |
| **PTS/DTS** | Buffer timestamp, font size 64, position X=50, Y=920, red color |

<img alt="Timestamp" />

### Privacy Masks

Privacy masks can be used to obscure specific portions of the image using the **masks** property.

The property payload follows the GStreamer structure layout and requires a unique name along with a `color` value and one of the supported shape definitions:

| Shape | Syntax | Description |
| - | - | - |
| `circle` | `circle=<X, Y, RADIUS>` | Draws a circular mask centered at the specified X/Y position with the given radius |
| `rectangle` | `rectangle=<X, Y, WIDTH, HEIGHT>` | Draws a rectangular mask using the specified coordinates and dimensions |
| `polygon` | `polygon=<X1, Y1, X2, Y2, X3, Y3, ...>` | Draws a polygonal mask defined by multiple coordinate points |

Additional parameter:

| Parameter | Description |
| - | - |
| `color` | Mask color in RGBA hex format |

```bash theme={null}
"{(structure)"Mask2,color=0xFF0000FF,circle=<1520,730,50>;", (structure)"Mask1,color=0x0000FFFF,rectangle=<1280,860,100,150>;", (structure)"Mask3,color=0x00FF00FF,polygon=<<910,510>,<860,740>,<1140,760>,<1160,520>>;"}"
```

| | Description |
| - | - |
| **Mask2** | Red circle mask — X=1520, Y=730, R=50 |
| **Mask1** | Blue rectangle mask — X=1280, Y=860, Width=100, Height=150 |
| **Mask3** | Green polygon mask — points: (910,510), (860,740), (1140,760), (1160,520) |

<img alt="Privacy" />

### Bounding Boxes

Rectangles used to highlight a region or object within the frame can be configured using the **bboxes** property.

The property payload follows the GStreamer structure layout and requires the following parameters when first set:

| Parameter | Description |
| - | - |
| `position` | Top-left X/Y coordinates where the rectangle should be placed |
| `dimensions` | Width and height of the rectangle |
| `color` | Rectangle color in RGBA hex format |

```bash theme={null}
"{(structure)"Box1,position=<100,100>,dimensions=<640,480>;", (structure)"Box2,position=<1000,100>,dimensions=<300,300>,color=0xFF0000FF;"}"
```

| | Description |
| - | - |
| **Box1** | Position X=100, Y=100, size 640×480 |
| **Box2** | Position X=1000, Y=100, size 300×300, red color |

<img alt="BoundingBox" />

# Usage

### User-Defined Overlays with Live Camera Preview

Sample pipeline displaying user-defined text positioned at the **top-left** corner of the video frames (live preview from camera), along with a circular privacy mask applied to conceal a selected region of the image.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e --gst-debug=2 \
qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! \
qtivoverlay \
masks='{(structure)"Mask1,color=0xff0011ff,inverse=false,circle=<500,500,50>;"}' \
strings='{(structure)"Text1,contents=\"Example\ 1\",fontsize=40,position=<0,0>,color=0xFF00FFFF;"}' \
! queue ! waylandsink fullscreen=true sync=true
```

### Single-Stage AI Inference with Overlay

Sample pipeline containing a single stage AI inference that performs object detection on the video frame. The AI stage generates ROI metadata in string format and attaches it to the main frame. This metadata is then rendered as an overlay using the **qtivoverlay** plugin and displayed on the output video.

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
    | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"
      scp yolox_w8a8.tflite   <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json         <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/models $HOME/Downloads/qimsdk_samples/labels
    export MODEL_NAME=yolox_w8a8.tflite
    export LABELS_NAME=yolov8.json
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! tee name=t_split \
    t_split. ! queue ! metamux. \
    t_split. ! queue ! qtimlvconverter ! queue ! qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! qtimlpostprocess results=6 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! text/x-raw ! queue ! metamux. \
    qtimetamux name=metamux ! queue ! qtivoverlay ! queue ! waylandsink fullscreen=true sync=true
    ```
  </Step>
</Steps>

### Applying a Static Image as Overlay

This pipeline demonstrates how to blit a static image (e.g., a company logo or watermark) onto a live video stream using the `images` property of **qtivoverlay**. The image is loaded once at pipeline start and composited onto every frame at the specified position and size.

<Note>
  The image file must be in raw **BGRA** format. Save your image as `logo.bgra` before running this pipeline.
</Note>

<img alt="" />

<Steps>
  <Step title="Prepare Required Files">
    | File | Description | Save as |
    | - | - | - |
    | Logo image | Raw BGRA format image file (e.g., your company logo or watermark) | `logo.bgra` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media"
      scp logo.bgra  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/media
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
    qtivoverlay \
    images="{(structure)\"Image1,path=$HOME/Downloads/qimsdk_samples/media/logo.bgra,resolution=<480,360>,destination=<0,0,640,480>;\"}" \
    ! waylandsink fullscreen=true sync=true
    ```

    | Parameter | Value | Notes |
    | - | - | - |
    | `path` | `$HOME/Downloads/qimsdk_samples/media/logo.bgra` | Absolute path to the raw BGRA image file |
    | `resolution` | `<480,360>` | Original width × height of the source image |
    | `destination` | `<0,0,640,480>` | X, Y position and rendered width × height |
  </Step>
</Steps>

### Applying Timestamp — DD/MM/YY and Time as Text Overlay

This pipeline demonstrates how to render the current date in **DD/MM/YYYY** format and the current time as a live text overlay on each video frame using the `timestamps` property of **qtivoverlay**. The timestamp is updated automatically on every buffer.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e \
qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
qtivoverlay \
timestamps='{(structure)"Date/Time,format=\"%d/%m/%Y\ %H:%M:%S\",fontsize=60,position=<200,200>,color=0xFFFFFFFF;"}' \
! waylandsink sync=true fullscreen=true
```

| Parameter | Value | Notes |
| - | - | - |
| `format` | `%d/%m/%Y %H:%M:%S` | Day/Month/Year Hour:Min:Sec — e.g., `16/04/2025 14:35:07` |
| `fontsize` | `60` | Font size in points |
| `position` | `<200,200>` | X, Y pixel coordinates on the frame |
| `color` | `0xFFFFFFFF` | White text in RGBA hex format |

***

### Applying Circular and Rhombus Privacy Masks

This pipeline demonstrates how to apply multiple privacy masks of different shapes — a **circle** and a **rhombus** (polygon) — to obscure sensitive regions of the video frame using the `masks` property of **qtivoverlay**. Both masks are rendered simultaneously on every frame.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e --gst-debug=2 \
qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
qtivoverlay \
masks='{(structure)"Mask1,color=0xff0011ff,inverse=false,circle=<500,500,50>;",(structure)"Mask2,color=0xff0011f0,polygon=<<100,50>,<150,100>,<100,150>,<50,100>>;"}' \
! waylandsink sync=true fullscreen=true
```

| Mask | Shape | Parameters | `color` | Notes |
| - | - | - | - | - |
| Mask 1 | `circle` | `circle=<500,500,50>`, `inverse=false` | `0xff0011ff` | Circle at center X=500, Y=500, R=50 |
| Mask 2 | `polygon` | `polygon=<<100,50>,<150,100>,<100,150>,<50,100>>` | `0xff0011f0` | Rhombus defined by four corner points |

<Tip>Multiple masks can be combined in a single `masks` property by listing multiple GStreamer structures separated by commas. Each mask is rendered independently and can use a different shape, position, and color.</Tip>

***


# qtivsplit
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtivsplit

qtivsplit is a GStreamer plugin that creates new video buffers from an input video frame by applying spatial transformations such as cropping, resizing, and color conversion.

# Overview

`qtivsplit` is a GStreamer plugin that creates new video buffers from an input video frame by applying spatial transformations such as **cropping, resizing, and color conversion**. It can use either the **entire input frame** or **regions of interest (ROIs)** attached as metadata as the source for these transformations. This allows a single input stream to be repurposed into multiple output streams, each tailored to the requirements of different downstream elements.

When used in **full-frame mode**, `qtivsplit` treats the entire input frame as the source image and generates one or more output buffers from it. Each output can be configured independently with its own: **resolution, pixel format, color space**

This is useful when the same video stream must be prepared differently for multiple downstream paths, such as: **display, video encoding, network streaming, and AI inference**.

`qtivsplit` generates a separate output buffer for each branch, so each downstream path receives its own independent buffer. This prevents one branch from affecting another. If one of the requested outputs already matches the input frame format and resolution, the plugin may forward the original input buffer to that branch instead of creating a duplicate, while still generating separate buffers for the remaining outputs.

When used in **ROI mode**, `qtivsplit` operates on metadata-defined regions inside the input frame rather than on the full frame. For each ROI, the plugin extracts the corresponding image region and generates a new output buffer after applying the required: **crop, resize, and format conversion**.

This makes it possible to move from **frame-based processing** to **object-based processing**. Instead of sending the full frame downstream, the pipeline can send individual object images derived from the detected ROIs. This is useful in workflows where downstream elements need to process each detected object independently.

ROI-derived outputs can be delivered in two ways:

* **sequentially on a single output pad**, for example when generating a stream of thumbnails for all detected objects
* **across multiple output pads in parallel**, for example when several downstream inference branches process different objects at the same time

In practice, `qtivsplit` is useful whenever a pipeline needs to transform a single input frame into multiple derived outputs—either as multiple full-frame variants or as separate object-level buffers. This makes it a flexible element for building pipelines that combine video rendering, encoding, and AI processing in the same workflow.

<img alt="" />

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />qtivsplit

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { NV12, NV21, UYVY, YUY2, RGBA, BGRA, ARGB, ABGR, RGBx, BGRx, xRGB, xBGR, RGB, BGR, GRAY8, NV12_Q08C }` <br /> `width: [1, 32767]` <br /> `height: [1, 32767]` <br /> `framerate: [0/1, 255/1]` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { NV12, NV21, UYVY, YUY2, RGBA, BGRA, ARGB, ABGR, RGBx, BGRx, xRGB, xBGR, RGB, BGR, GRAY8, NV12_Q08C }` <br /> `width: [1, 32767]` <br /> `height: [1, 32767]` <br /> `framerate: [0/1, 255/1]` |
| Availability: *On request* | |
| Direction: *source* | |

# Pad Properties

### src

| Property | Description |
| - | - |
| `mode` | Source pad operational mode.<br /><br />`Type: Enum `<br />`Default: 0, "none"`<br />`Range:`<br />    `(0): none - The input buffer is resized and color-converted as needed to match the negotiated output caps. If the input and output caps already match, the original input buffer is forwarded directly to the output pad by increasing its reference count`<br />    `(1): force-transform - The input buffer is resized and color-converted to match the negotiated output caps. Unlike none, this mode always produces a new output buffer, even when the input and output caps are identical`<br />    `(2): single-roi-meta - The input buffer is checked for ROI metadata. If an ROI entry corresponding to this pad is found, the plugin crops that region from the input frame, applies the required resize and color conversion, and pushes the transformed buffer downstream. If no matching ROI is found, the pad produces a GAP buffer`<br />    `(3): batch-roi-meta - The input buffer is checked for ROI metadata. For every ROI entry found, the plugin crops the corresponding region from the input frame, applies the required resize and color conversion, and produces a separate output buffer. If no ROI metadata is present, the pad produces a GAP buffer`<br />`Flags: readable/writable (changeable only in NULL or READY state)` <br /> `Example: mode="force-transform" (or) mode=1` |

# Element Properties

| Property | Description |
| - | - |
| `engine` | Engine backend used for the conversion operations.<br /><br />`Type: Enum `<br />`Default: 2, "gles"`<br />`Range:`<br />    `(0): none - No backend used`<br />    `(2): gles - OpenGLES based engine is used`<br />    `(3): fcv - FastCV based engine is used`<br />`Flags: readable/writable` <br /> `Example: engine="gles" (or) engine=2` |

# Architecture Details

`qtivsplit` uses a single sink pad and a dynamically created set of source pads (`src_%u`). Each source pad has its own internal output context, which stores the pad's operating mode, negotiated output format and resolution, dedicated buffer pool, and an internal queue for staging generated buffers.

Internally, the element uses an asynchronous, queue-based processing model. When an input frame arrives, `qtivsplit` quickly determines the required output operations for each active source pad based on its configuration. These operations may include cropping, resizing, and color conversion, and are submitted to the underlying hardware conversion engine. The input frame and its associated output work are then placed into an internal request queue.

Worker tasks process this queue asynchronously. As conversion operations complete, the generated output buffers are delivered to the corresponding source pads. This design separates input-frame handling from output-buffer production, which helps maintain high throughput even when different output branches run at different speeds.

Each source pad uses its own image buffer pool, created after output caps negotiation. The pool is configured using the negotiated resolution, pixel format, and alignment requirements, so the generated buffers are ready for downstream use without additional allocation or format conversion.

This architecture allows `qtivsplit` to generate multiple independent outputs from the same input frame, whether those outputs are derived from the full frame or from ROI metadata. It also keeps output branches isolated from one another and provides predictable behavior under varying downstream load conditions.

# Output Operation Modes

<img alt="" />

Each output pad of **qtivsplit** operates in a specific mode that defines how output video buffers are produced from the input stream. The selected mode controls whether the output is derived from the full input frame or from regions of interest (ROIs), and whether output buffers are reused or always newly created.

In **full‑frame modes**, the entire input frame is used to generate exactly one output frame per input frame.

## Example Pipeline

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/media/output
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    qtivsplit name=vsplit \
    vsplit. ! video/x-raw,format=NV12 ! queue ! waylandsink fullscreen=true sync=true \
    vsplit. ! video/x-raw,format=NV12 ! queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! h264parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/split_output.mp4
    ```
  </Step>
</Steps>

## Frame-based (reuse when possible) (`mode=none`)

For each input frame, a full-frame output is produced. If the input and output formats already
match, the input buffer may be forwarded directly to the output to avoid extra processing. If
the formats differ, the frame is resized and/or color‑converted to match the configured output
format.

## Frame-based (forced transform) (`mode=force-transform`)

This mode also produces one full-frame output per input frame, but always creates a new
output buffer, even when the input format already matches. This is useful when downstream
elements require a guaranteed independent buffer.

In ROI‑based modes, qtivsplit uses ROI metadata attached to the input frame to generate
output buffers from specific regions.

## Single‑ROI output (`mode=single-roi-meta`)

Each input frame is checked for ROI metadata. If an ROI is present, the output buffer is
created by cropping the frame to that region, then applying the configured resize and color
conversion. If no ROI is available, an empty (GAP‑like) frame is produced. This mode is
typically used when ROIs are processed as a serialized stream.

## Multi‑ROI output (`mode=batch-roi-meta`)

When multiple ROIs are present on a frame, this mode generates one output buffer per ROI.
All buffers are emitted sequentially on the same output pad, forming a single serialized output
stream. Each output is cropped from the corresponding ROI and then resized and
color‑converted according to the output configuration.

# Processing Flow

## Output pads and configuration

`qtivsplit` exposes a single sink pad and a dynamically created set of output pads named `src_%u`. Output pads are created on demand.

Each output pad is configured with a **mode** property (default: `none`) that defines how output buffers are generated from incoming frames. The mode must be set before streaming starts (it is mutable in the READY state). Attempts to change the mode while streaming are rejected and reported as warnings.

## Output format negotiation

Each output pad performs caps negotiation independently with its downstream peer.

During negotiation:

* Downstream‑accepted caps are queried and fixated. The element prefers formats that minimize destructive conversion, such as:
  * Matching pixel formats and base types
  * Preserving alpha channels where applicable
* Width, height, and pixel aspect ratio are fixated to preserve display aspect ratio whenever possible.
* The output frame rate must match the input frame rate. Negotiation fails if they differ.
* In `batch-roi-meta` mode, `multiview-mode` is set to a frame‑by‑frame mode to signal that multiple output frames may be produced from a single input frame.

## Buffer pool setup

After caps negotiation, an allocation query is issued downstream. Based on the response:

* A buffer pool is created per output pad.
* The pool is configured with the appropriate allocator and GPU alignment requirements derived from the negotiated output format.
* The pool is activated and retained for subsequent buffer acquisition.

## Input frame handling

For each incoming input frame:

* A request object is created, with one slot per configured output pad.
* A reference to the input frame is stored along with a timestamp for internal tracking.
* A set of conversion compositions is prepared. This determines:
  * Which outputs will produce frames
  * Whether each output is generated from the full frame or from ROIs
  * The required spatial transforms, scaling, and color conversion

All compositions for the input frame are submitted together in a single conversion engine call to allow efficient scheduling and resource reuse.

## Output generation modes

The output generation behavior depends on the configured mode of each output pad.

`mode = none`

* Each input frame produces one full‑frame output.
* If the input already matches the negotiated output format and dimensions, the buffer may be forwarded by reference (passthrough).
* Otherwise, a new buffer is generated using resize and/or color conversion.

`mode = force-transform`

* Each input frame produces one full‑frame output.
* A new output buffer is always generated, even if the input already matches the output configuration.
* This mode guarantees a distinct buffer instance for downstream processing.

`mode = single-roi-meta`

* Each input frame is inspected for ROI metadata.
* Only top‑level ROIs (`parent_id == -1`) are considered.
* If an applicable ROI is found:
  * The frame is cropped to the ROI
  * The result is resized and color‑converted to the configured output format
  * Aspect ratio is preserved and content is centered where applicable
  * An `ImageRegion` meta is attached to describe the filled region
* If no applicable ROI is found, an empty (GAP‑like) buffer is produced for that output.

`mode = batch-roi-meta`

* Each input frame is inspected for ROI metadata.
* One output buffer is generated per ROI using the same crop, resize, and conversion behavior as `single-roi-meta`.
* All generated buffers are emitted sequentially on the same output pad, forming a single serialized stream.
* If no ROI metadata is present, a GAP buffer is emitted to indicate that no result is available for the current frame and to maintain downstream synchronization.

## Request Submission and synchronization

When conversion work is required:

* All prepared compositions are submitted to the hardware conversion engine as a batch.
* A synchronization fence is returned and stored with the corresponding request.
* Worker tasks monitor the request queue, wait for conversion completion, and push the resulting buffers to their respective output pads.

## Output delivery

* In passthrough cases, the input buffer is forwarded by reference.
* If no output buffer was generated for an output pad, an empty buffer is emitted as a placeholder.
* Otherwise, produced buffers are dequeued from the request and pushed downstream in order.

## State transitions and flushing

When leaving the streaming state:

* Worker tasks are stopped.
* The conversion engine is flushed.
* All pending requests are cleared.

This ensures a clean shutdown and prevents incomplete or stale conversions from carrying across state transitions.

## Chained scaling optimization

When multiple full‑frame outputs are configured at different resolutions, `qtivsplit` applies a chained scaling strategy to reduce hardware load.

Instead of scaling each output directly from the input resolution:

* The highest‑resolution output is generated first from the input frame.
* Each subsequent lower‑resolution output is derived from the closest preceding intermediate resolution.

For example, with an 8K input and outputs at 4K, Full HD, HD, and 640×360:

| Step | Input | Output |
| - | - | - |
| 1 | 8K | 4K |
| 2 | 4K | Full HD |
| 3 | Full HD | HD |
| 4 | HD | 640×360 |

This significantly reduces the total cost of scaling.

Scaling and color conversion are treated as separate operations. When outputs require different pixel formats, scaling results may still be reused, and color conversion is applied only where needed after the appropriate spatial resolution has been produced. This optimization is transparent to downstream elements and preserves per‑output guarantees for buffer independence and format correctness, while improving overall hardware efficiency.

# Usage

### Camera Stream Usecase

This example demonstrates how a live 1080p camera stream can be split into three output branches using the default `qtivsplit` mode (`none`). One branch preserves the original input resolution and format and renders the stream using [`waylandsink`](/plugin-reference/waylandsink). The second branch performs color conversion from `NV12` to `RGB` and delivers the output to `fakesink`. The third branch scales the video to 720p and stores the result as an MP4 file using `filesink`.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e --gst-debug=2 \
qticamsrc ! video/x-raw,width=1920,height=1080,framerate=30/1,format=NV12 ! queue ! \
qtivsplit name=vsplit \
vsplit. ! video/x-raw,width=1920,height=1080,format=NV12 ! queue ! waylandsink fullscreen=true sync=true \
vsplit. ! video/x-raw,width=1920,height=1080,format=RGB ! queue ! fakesink \
vsplit. ! video/x-raw,width=1280,height=720,format=NV12 ! queue ! \
v4l2h264enc capture-io-mode=4 output-io-mode=5 ! h264parse ! mp4mux ! queue ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/output_720p.mp4
```

### Single ROI Meta

This example demonstrates the `single-roi-meta` mode of `qtivsplit` in a live camera pipeline with object detection. The camera stream is processed by a YOLOv8 detection branch, and the resulting ROI metadata is attached to the video buffer using [`qtimetamux`](/plugin-reference/qtimetamux). qtivsplit then reads the ROI metadata for each detected objects and generates up to four ROI-derived outputs, one per source pad. These outputs are combined into a 2×2 layout by [`qtivcomposer`](/plugin-reference/qtivcomposer) and rendered for display using [`waylandsink`](/plugin-reference/waylandsink).

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLO model | [Qualcomm AI hub model](https://aihub.qualcomm.com/iot/models/yolox) | `yolov8_det_w8a8.tflite` |
    | YOLO labels | \<a href="../labels/yolov8.json" download="yolov8.json">yolov8.json\</a> | `yolov8.json` |
    | Input video | [Input video](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/) | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
      scp yolov8_det_w8a8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4              <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/models $HOME/Downloads/qimsdk_samples/labels $HOME/Downloads/qimsdk_samples/media
    export MODEL_NAME=yolov8_det_w8a8.tflite
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qtivsplit name=vsplit src_0::mode=batch-roi-meta \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    tee name=t_split ! queue ! qtimetamux name=metamux ! queue ! vsplit. \
    t_split. ! queue ! qtimlvconverter ! queue ! \
    qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME ! queue ! \
    qtimlpostprocess results=5 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! text/x-raw ! metamux. \
    vsplit. ! video/x-raw,width=640,height=360,format=NV12 ! multifilesink enable-last-sample=false location=$HOME/Downloads/qimsdk_samples/media/frame%d.yuv
    ```
  </Step>
</Steps>

### Batch ROI Meta

This example demonstrates the `batch-roi-meta` mode of qtivsplit in a file-based object-detection pipeline. The input video is decoded and processed by a YOLOv8 detection branch, and the resulting ROI metadata is attached to the video buffer using [`qtimetamux`](/plugin-reference/qtimetamux). qtivsplit then reads all ROI metadata entries and generates a separate ROI-derived output buffer for each detected object on the same source pad. The resulting outputs are written to individual YUV files using `multifilesink`.

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLO model | [Qualcomm AI Hub](https://aihub.qualcomm.com/iot/models/yolox) | `yolov8_det_w8a8.tflite` |
    | YOLO labels | \<a href="../labels/yolov8.json" download="yolov8.json">yolov8.json\</a> | `yolov8.json` |
    | Input video | [Input Video](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/) | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
      scp yolov8_det_w8a8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
      scp yolov8.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
      scp ai_demo_sample.mp4              <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/models $HOME/Downloads/qimsdk_samples/labels $HOME/Downloads/qimsdk_samples/media
    export MODEL_NAME=yolov8_det_w8a8.tflite
    export LABELS_NAME=yolov8.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    qtivsplit name=vsplit src_0::mode=batch-roi-meta \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    tee name=t_split ! queue ! qtimetamux name=metamux ! queue ! vsplit. \
    t_split. ! queue ! qtimlvconverter ! queue ! \
    qtimltflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME ! queue ! \
    qtimlpostprocess results=5 module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! text/x-raw ! metamux. \
    vsplit. ! video/x-raw,width=640,height=360,format=NV12 ! multifilesink enable-last-sample=false location=$HOME/Downloads/qimsdk_samples/media/frame%d.yuv
    ```
  </Step>
</Steps>


# qtivtransform
Source: https://imsdkdocs.qualcomm.com/plugin-reference/qtivtransform

AI pre-processing plugin

# Overview

The **qtivtransform** plugin is a GStreamer video transformation element designed for real-time, hardware-accelerated manipulation of video frames.
It plays a critical role in video and vision processing pipelines where performance, flexibility, and low latency are essential.

<img alt="" />

This plugin enables developers to apply a variety of transformations - such as **resizing**, **rotating**, **flipping**, **cropping**, and **color conversion** - all while maintaining high throughput.

Modern multimedia often require video frames to be transformed before further processing. For example:

* **Resizing**: A video stream may need to be resized to fit the native resolution of a screen (e.g., resizing 1920×1080 to 1280×720 for a smaller display).
* **Rotating**: Mobile devices or webcams may produce rotated frames depending on how the device is held. Rotating the frame ensures proper display alignment.
* **Flipping/Mirroring**: Front cameras often produce mirrored images. Horizontal flipping corrects this for natural viewing.
* **Cropping**: Cropping can help convert non-standard aspect ratios (e.g., 4:3 or 21:9) to standard ones like 16:9 for compatibility with downstream encoders or players.
* **Color conversion**: Тransform between formats (e.g., RGB to NV12) for compatibility with downstream components.

**qtivtransform** addresses these needs efficiently by leveraging GPU acceleration, making it suitable for a wide range of platforms - from low-power embedded systems to high-performance multimedia setups.

## Example Pipeline

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/media/output
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    decodebin ! \
    videoconvert ! \
    video/x-raw,format=NV12 ! \
    qtivtransform ! \
    videoconvert ! \
    waylandsink sync=true
    ```
  </Step>
</Steps>

# Hiearchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)<br />
         <Icon icon="arrow-turn-down-right" />[GstBaseTransform](https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c)<br />
            <Icon icon="arrow-turn-down-right" />qtivtransform

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { NV12, NV21, YUY2, P010_10LE, NV12_10LE32, RGBA, BGRA, ARGB, ABGR, RGBx, BGRx, xRGB, xBGR, RGB, BGR, GRAY8, NV12_Q08C }` <br /> `width: [1, 32767]` <br /> `height: [1, 32767]` <br /> `framerate: [0/1, 255/1]` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { NV12, NV21, YUY2, P010_10LE, RGBA, BGRA, ARGB, ABGR, RGBx, BGRx, xRGB, xBGR, RGB, BGR, RGBP, BGRP, GRAY8, NV12_Q08C }` <br /> `width: [1, 32767]` <br /> `height: [1, 32767]` <br /> `framerate: [0/1, 255/1]` |
| Availability: *Always* | |
| Direction: *source* | |

# Element Properties

| Property | Description |
| - | - |
| `background` | Defines the background color used when the destination rectangle does not fill the entire output frame.<br /><br />`Type: Unsigned Integer`<br />`Default: 4286611584`<br />`Range: 0 - 4294967295`<br />`Flags: readable/writable` |
| `crop` | Defines the crop rectangle on the input frame in the format `<X, Y, WIDTH, HEIGHT>`. Cropping is applied immediately upon frame reception and cannot be time-synchronized.<br /><br /> `Type: GstValueArray of type gint` <br /> `Default: Default: "<0, 0, 0, 0 >"` <br /> `Flags: readable/writable` <br /> `Example: crop="<0,0,1280,720>"` |
| `destination` | Specifies the destination rectangle within the output frame in the format `<X, Y, WIDTH, HEIGHT>`. Useful for positioning the transformed content within a larger output frame.<br /><br />`Type: GstValueArray of type gint` <br /> `Default: Default: "<0, 0, 0, 0 >"` <br /> `Flags: readable/writable` <br /> `Example: destination="<0,0,1280,720>"` |
| `engine` | Engine backend used for the transformation operations.<br /><br />`Type: Enum` <br /> `Default: 2, "gles"` <br /> `Range:`<br />    `(0): none - No backend used`<br />    `(2): gles - Use OpenGLES based video converter`<br />    `(3): fcv - Use FastCV-based video converter` <br /> `Flags: readable/writable` <br /> `Example: engine="gles" (or) engine=2` |
| `engine-param` | Additional parameters for the selected engine backend.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `flip-horizontal` | If set to true, the video frame is flipped horizontally (mirrored). Commonly used for correcting front-facing camera output.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `flip-vertical` | If set to true, the video frame is flipped vertically. Useful for correcting upside-down camera feeds.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `rotate` | Specifies the rotation to apply to the video frame.<br /><br />`Type: Enum` <br /> `Default: 0, "none"` <br /> `Range:`<br />    `(0): none - No rotation`<br />    `(1): 90CW - Rotate 90 degrees clockwise`<br />    `(2): 90CCW - Rotate 90 degrees counter-clockwise` <br />    `(3): 180 - Rotate 180 degrees`<br /> `Flags: readable/writable` <br /> `Example: rotate="90CW" (or) rotate=1` |

# Cropping Behavior

Cropping in qtivtransform is applied directly to the input frame, before any other transformation operations. This means the crop region is extracted from the original frame as it arrives at the input pad, ensuring that subsequent steps - such as scaling or color conversion - operate only on the cropped region.
This approach improves performance by reducing the number of pixels processed downstream and ensures that transformations are applied precisely to the intended area of the frame.
The diagram below illustrates this behavior: the crop region is selected from the full input frame, and only that region is passed forward for further processing.

<img alt="Cropping example" />

# Internal Architecture

The qtivtransform plugin operates as a GStreamer element with a straightforward but efficient internal architecture. It consists of two pads:

* Input Pad: Receives video frames from upstream elements (e.g. camera source, decoder).
* Output Pad: Sends transformed frames downstream (e.g. display, encoder).
  <img alt="qtivtransform architecture diagram" />

## Processing Flow

### Caps negotiation

* The output format and resolution in qtivtransform is determined by the negotiated output caps during the GStreamer pipeline setup.
* These caps define the resolution, color format, and other properties that downstream elements expect, and qtivtransform adapts its output accordingly.

### Property Parsing and Configuration

* Transformation parameters (e.g., rotation angle, crop region, resize dimensions) are parsed from the element’s properties.
* These parameters are stored in an internal configuration structure that guides the transformation logic.

### Frame Reception

* Incoming video frames are received through the sink pad (input pad).
* The plugin validates the frame format and dimensions.

### Buffer Allocation and Pool Negotiation

* The plugin expects DMA-backed buffers for zero-copy performance and GPU compatibility.
* If the upstream element does not provide DMA buffers:
  * qtivtransform offers its own buffer pool via GStreamer’s allocation mechanism.
  * If the upstream accepts the pool, compatible buffers are allocated.
  * If not, the plugin fails during negotiation and reports an error.
* This ensures buffer memory layout is optimized for the selected engine.

### Transformation Execution

* The GPU engine performs the requested operations (e.g., rotation angle, crop region, resize dimensions).
* The scaling is performed before color conversion to minimize GPU memory bandwidth usage.

### Frame Output

* The transformed frame is pushed to the source pad (output pad).
* Downstream elements (e.g. encoders, displays) receive the processed frame for further handling.

# Buffer Management and Pool Requirements

qtivtransform is designed to operate efficiently with DMA-allocated buffers, which are essential for zero-copy performance and hardware acceleration. To ensure compatibility and optimal throughput, the plugin enforces specific buffer handling requirements:

* DMA Buffer Requirement
  * The plugin expects incoming buffers to be DMA-backed.
  * This is crucial for interoperability with GPU-based backends and for minimizing memory copies during transformation.
* Buffer Pool Negotiation
  * If the upstream element does not provide DMA buffers, qtivtransform can offer its own buffer pool to the upstream element via the standard GStreamer allocation mechanism.
  * This allows the upstream plugin to allocate compatible buffers from qtivtransform's pool.
* Fallback Behavior
  * If the upstream element does not support buffer pool negotiation (i.e., cannot accept a pool from qtivtransform), the plugin will not function correctly.
  * In such cases, pipeline setup will fail, and an error will be reported during negotiation.

This behavior ensures that buffer allocation is tightly controlled and optimized for the transformation backend.

# Usage

### Downscale a YUV video stream to RGB format

* This pipeline demonstrates how qtivtransform can be used to convert and downscale a YUV (NV12) video stream to RGB format.
* This is useful in scenarios where:
  * You need to dump RGB frames to disk for use in a custom image processing or computer vision algorithm that expects raw RGB input.
  * A downstream plugin or application requires RGB format instead of NV12.
    <img alt="Usecase 1" />

```bash theme={null}
gst-launch-1.0 -e --gst-debug=2 qticamsrc ! video/x-raw,width=1920,height=1080,format=NV12,framerate=5/1 ! queue ! qtivtransform ! video/x-raw,width=1280,height=720,format=RGB ! multifilesink location=$HOME/Downloads/qimsdk_samples/media/frame_%d.rgb
```

### Horizontal flip

* This pipeline captures video frames from a Qualcomm camera source (qticamsrc) at 1920×1080 resolution in NV12 format. The frames are passed to the qtivtransform, which applies a 90-degree clockwise rotation and horizontal flip.
  After transformation, the frames are encoded using the hardware-accelerated H.264 encoder (v4l2h264enc). The encoded stream is parsed (h264parse), multiplexed into an MP4 container (mp4mux), and written to disk via filesink.
  <img alt="Usecase 2" />

```bash theme={null}
gst-launch-1.0 -e --gst-debug=2 qticamsrc ! video/x-raw,width=1920,height=1080,format=NV12,framerate=30/1 ! queue ! qtivtransform rotate=90CW flip-horizontal=true ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! h264parse ! mp4mux ! queue ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/video.mp4
```

### Cropping and Scaling

* The following GStreamer pipeline demonstrates how qtivtransform performs cropping and scaling before encoding the video:

  * Input: 1280×720 NV12 video stream from qticamsrc.
  * Cropping: qtivtransform crops a 640×360 region starting at (320,180) from the input frame. This is done immediately upon receiving the frame.
  * Scaling: The cropped region is then scaled up to 1920×1080 as specified by the output caps.
  * Encoding: The transformed frame is encoded using v4l2h264enc and saved as an MP4 file.
    <img alt="Usecase 3" />

  ```bash theme={null}
  gst-launch-1.0 -e --gst-debug=2 qticamsrc ! video/x-raw,width=1280,height=720,format=NV12,framerate=30/1 ! queue ! qtivtransform crop="<320,180,640,360>" ! video/x-raw,width=1920,height=1080 ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! h264parse ! mp4mux ! queue ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/video.mp4
  ```


# v4l2av1dec
Source: https://imsdkdocs.qualcomm.com/plugin-reference/v4l2av1dec

Hardware-accelerated AV1 video decoder using the V4L2 stateful decoder API on Qualcomm hardware.

# Overview

`v4l2av1dec` is a hardware-accelerated video decoder that uses the Video4Linux2 (V4L2) stateful decoder API to offload AV1 video decoding to the Qualcomm Video Processing Unit (VPU).

This plugin is provided and maintained by the GStreamer community. This document focuses on its usage in conjunction with Qualcomm-specific QIM SDK GStreamer plugins, along with relevant use cases and internal architectural considerations.

`v4l2av1dec` is typically used to decode AV1 streams provided by:

* File sources (WebM, Matroska/MKV, MP4/ISOBMFF containers)
* RTSP sources
* HLS/HTTP streaming sources (e.g., YouTube AV1, streaming platforms)

The element is responsible **only for decoding** video frames. It does not demux or parse streams on its own — those operations must be handled by peer GStreamer elements upstream (e.g., `matroskademux` and `av1parse`).

## Example Pipeline

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    av1\_1080p.webm file is used here as an example.
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media"
      scp av1_1080p.webm <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export SRC_VIDEO_NAME=av1_1080p.webm
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e \
      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
      matroskademux ! queue ! av1parse ! \
      v4l2av1dec capture-io-mode=4 output-io-mode=4 ! \
      video/x-raw,format=NV12 ! queue ! \
      waylandsink fullscreen=true sync=true
    ```
  </Step>
</Steps>

# Key Responsibilities

`v4l2av1dec` is responsible for:

* **Hardware acceleration** — offloads AV1 decoding to the dedicated VPU
* **V4L2 state management** — manages the V4L2 stateful decoder state machine (device open, buffer allocation, stream on/off)
* **Buffer I/O management** — handles buffer exchange between GStreamer and the V4L2 driver using DMABuf, MMAP, or UserPtr modes
* **Format negotiation** — negotiates raw output formats supported by the hardware, including UBWC-compressed formats (`NV12_Q08C` for 8-bit SDR, `NV12_Q10LE32C` for 10-bit HDR) for reduced memory bandwidth
* **Multi-stream support** — supports multiple concurrent decoder instances (subject to hardware resource limits)
* **Error handling** — supports decoder error controls such as `max-errors` and corrupted-frame discard behavior

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html)<br />
         <Icon icon="arrow-turn-down-right" />GstVideoDecoder<br />
            <Icon icon="arrow-turn-down-right" />GstV4l2VideoDec<br />
               <Icon icon="arrow-turn-down-right" />v4l2av1dec

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-av1` | `format: NA` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { NV12_Q08C, NV12_Q10LE32C, NV12, NV21 }` <br /> `width: [1, 32768]` <br /> `height: [1, 32768]` |
| Availability: *Always* | |
| Direction: *source* | |

<Note>
  AV1 does not perform profile or level probing via V4L2 control enumeration in the current implementation. The sink caps are therefore unconstrained beyond the `video/x-av1` media type. Hardware capabilities are implicitly constrained by what the V4L2 driver accepts at runtime — if the driver does not support a particular AV1 profile or level, it will return an error during format negotiation (`VIDIOC_S_FMT`).
</Note>

# Element Properties

| Property | Description |
| - | - |
| `automatic-request-sync-point-flags` | Flags used when automatically requesting sync points.<br /><br />`Type: Flags`<br />`Default: 0x3 (corrupt-output+discard-input)`<br />`Flags: readable/writable` |
| `automatic-request-sync-points` | Automatically requests sync points such as keyframes when useful, for example after decoding errors.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `capture-io-mode` | I/O mode for the capture queue (src pad). Controls how decoded frame buffers are allocated and transferred.<br /><br />`Type: Enum (GstV4l2IOMode)`<br />`Default: 0, "auto"`<br />`Flags: readable/writable` |
| `device` | The V4L2 device node path. Set automatically at registration time and read-only after element creation.<br /><br />`Type: String`<br />`Default: "/dev/video32"`<br />`Flags: readable` |
| `device-fd` | File descriptor of the opened V4L2 device. A value of `-1` indicates that the device is not open.<br /><br />`Type: Integer`<br />`Default: -1`<br />`Flags: readable` |
| `device-name` | Human-readable name of the V4L2 device as reported by the driver via VIDIOC\_QUERYCAP.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable` |
| `discard-corrupted-frames` | When enabled, frames marked as corrupted by the driver are dropped instead of passed downstream.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `extra-controls` | Extra V4L2 controls (CIDs) to set on the device, specified as a GstStructure. Applied via VIDIOC\_S\_EXT\_CTRLS.<br /><br />`Type: GstStructure`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `max-errors` | Maximum number of consecutive decoder errors before the element returns a flow error. A value of `-1` means unlimited.<br /><br />`Type: Integer`<br />`Default: -1`<br />`Flags: readable/writable` |
| `min-force-key-unit-interval` | Minimum interval in nanoseconds between force-keyunit requests sent upstream. A value of `0` means no minimum interval.<br /><br />`Type: Unsigned Integer64`<br />`Default: 0`<br />`Flags: readable/writable` |
| `output-io-mode` | I/O mode for the output queue (sink pad). Controls how compressed input buffers are submitted to the driver.<br /><br />`Type: Enum (GstV4l2IOMode)`<br />`Default: 0, "auto"`<br />`Flags: readable/writable` |
| `qos` | Handles Quality-of-Service events from downstream. When enabled, the decoder may drop frames to maintain real-time playback.<br /><br />`Type: Boolean`<br />`Default: true`<br />`Flags: readable/writable` |

## I/O Mode Values

Both `capture-io-mode` and `output-io-mode` accept the same `GstV4l2IOMode` enumeration:

| Value | Integer | Description |
| - | - | - |
| `auto` | 0 | Automatically select the best I/O mode. Prefers DMABuf export for capture, MMAP for output. |
| `rw` | 1 | Use read/write system calls. Rarely used for M2M devices. |
| `mmap` | 2 | Use kernel memory-mapped buffers (`V4L2_MEMORY_MMAP`). |
| `userptr` | 3 | Use user-space pointer buffers (`V4L2_MEMORY_USERPTR`). |
| `dmabuf` | 4 | Export buffers as DMABuf file descriptors (`V4L2_MEMORY_DMABUF`). Enables zero-copy sharing with downstream GPU/display elements. |
| `dmabuf-import` | 5 | Import DMABuf file descriptors from an external allocator into the V4L2 queue. |

# Internal Architecture

`v4l2av1dec` operates using two V4L2 queue objects internally:

* **Output queue** (`V4L2_BUF_TYPE_VIDEO_OUTPUT`) — receives compressed AV1 OBU buffers from upstream
* **Capture queue** (`V4L2_BUF_TYPE_VIDEO_CAPTURE`) — produces decoded raw video frames for downstream

<img alt="" />

# State Transitions

| Transition | Decoder Behavior | Notes |
| - | - | - |
| `NULL → READY` | Element becomes ready. Static configuration available. | Configure `capture-io-mode` and `output-io-mode` before active decoding. |
| `READY → PAUSED` | Decoder activates. V4L2 session setup begins. | Caps negotiation: sink is `video/x-av1`, src is `video/x-raw` (NV12/NV12\_Q08C). AV1 OBU-aligned frames are expected — use `av1parse` upstream to ensure correct framing. |
| `PAUSED → PLAYING` | Continuous decode starts. | With `capture-io-mode=4` and `output-io-mode=4`, pipeline is configured for DMABuf zero-copy. |
| `PLAYING → PAUSED` | Decoding paused at GStreamer scheduling level. | No full decoder teardown. Resume continues from current position. |
| `PLAYING/PAUSED → READY` | Decoder stops. V4L2 streaming stopped, buffers released. | Key cleanup transition. |
| `READY → NULL` | Element fully shut down. All resources released. | |
| EOS while PLAYING | Decoder drains pending input, outputs remaining frames, then forwards EOS. | |
| FLUSH | Current decode interrupted. Queued buffers discarded. | Common during seek or pipeline reset. Decoder waits for new AV1 intra-only (keyframe) after flush. |

# Dynamic Resolution Change

`v4l2av1dec` supports mid-stream dynamic resolution changes without requiring a pipeline restart, handled through the V4L2 source change event mechanism:

1. At initialization, the decoder subscribes to `V4L2_EVENT_SOURCE_CHANGE` events on the capture queue.
2. When the driver detects a resolution change in the bitstream (e.g., a new AV1 sequence header with different frame dimensions), it signals the event.
3. The decoder stops the capture queue, discards the existing buffer pool, re-negotiates format via `VIDIOC_G_FMT`, sets a new output state with updated dimensions, and reallocates the capture buffer pool.

This allows seamless decoding of adaptive bitrate AV1 streams and content with embedded resolution changes (e.g., AV1 streams with multiple sequence headers).

# Memory and Buffer Management

## DMABuf Usage

Setting `capture-io-mode=4` exports decoded frame buffers as DMABuf file descriptors. These can be imported directly by `waylandsink` or `qtivcomposer` without CPU memory copies, enabling zero-copy pipelines.

## Alignment Requirements

Decoded buffers follow Qualcomm hardware alignment requirements (e.g., 128-byte stride alignment).

## Format Support

Standard NV12 is common. UBWC (Universal Bandwidth Compression) is supported through:

* `NV12_Q08C` — 8-bit SDR content
* `NV12_Q10LE32C` — 10-bit HDR content (HDR10, HDR10+, HLG)

For 10-bit AV1 content, the decoder automatically selects `NV12_Q10LE32C` when the downstream element supports it.

## Codec Header Initialization

Before the first frame is decoded, the decoder sends the AV1 sequence header OBU (Open Bitstream Unit) to the output queue. If the input caps contain `codec_data` (e.g., from the `av1C` configuration record in MP4/ISOBMFF containers), that data is sent first. Otherwise, the first input buffer itself is used as the initialization data.

## Latency

The decoder computes and reports pipeline latency based on the minimum number of capture buffers required by the driver and the frame duration:

```
latency = min_buffers × frame_duration
```

## Drain and Flush

* **Drain** — sends `V4L2_DEC_CMD_STOP` to signal end-of-stream; waits for all remaining frames to be produced
* **Flush** — stops both queues, resets buffer pools, and restarts streaming; used during seek operations

# Usage

<Note>
  Ensure you have followed the [prerequisites](/plugin-reference/v4l2av1dec#example-pipeline) before continuing
</Note>

### Decode AV1 from a WebM File and Display

Demonstrates hardware-accelerated AV1 decoding from a WebM file with DMABuf zero-copy for Wayland preview.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
  matroskademux ! queue ! av1parse ! \
  v4l2av1dec output-io-mode=4 capture-io-mode=4 ! \
  video/x-raw,format=NV12 ! queue ! \
  waylandsink fullscreen=true
```

### Four-Stream Side-by-Side Composition

Demonstrates decoding four AV1 input streams using four hardware decoder instances, arranging them in a 2×2 grid using `qtivcomposer`, and displaying the composed frame.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e --gst-debug=2 \
  qtivcomposer name=mixer \
    sink_0::position="<0, 0>"     sink_0::dimensions="<960, 540>" \
    sink_1::position="<960, 0>"   sink_1::dimensions="<960, 540>" \
    sink_2::position="<0, 540>"   sink_2::dimensions="<960, 540>" \
    sink_3::position="<960, 540>" sink_3::dimensions="<960, 540>" \
  ! queue ! waylandsink fullscreen=true sync=true enable-last-sample=false \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    matroskademux ! queue ! av1parse ! v4l2av1dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    matroskademux ! queue ! av1parse ! v4l2av1dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    matroskademux ! queue ! av1parse ! v4l2av1dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    matroskademux ! queue ! av1parse ! v4l2av1dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer.
```


# v4l2h264dec
Source: https://imsdkdocs.qualcomm.com/plugin-reference/v4l2h264dec

Hardware-accelerated H.264 video decoder using the V4L2 stateful decoder API on Qualcomm hardware.

# Overview

`v4l2h264dec` is a hardware-accelerated video decoder that uses the Video4Linux2 (V4L2) stateful decoder API to offload H.264 (AVC) video decoding to the Qualcomm Video Processing Unit (VPU).

This plugin is provided and maintained by the GStreamer community. This document focuses on its usage in conjunction with Qualcomm-specific QIM SDK GStreamer plugins, along with relevant use cases and internal architectural considerations.

`v4l2h264dec` is typically used to decode H.264 streams provided by:

* File sources
* RTSP sources
* HLS/HTTP streaming sources

The element is responsible **only for decoding** video frames. It does not demux or parse streams on its own — those operations must be handled by peer GStreamer elements upstream (e.g., `qtdemux` and `h264parse`).

## Example Pipeline

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media"
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    waylandsink fullscreen=true sync=true
    ```
  </Step>
</Steps>

# Key Responsibilities

`v4l2h264dec` is responsible for:

* **Hardware acceleration** — offloads H.264 decoding to the dedicated VPU
* **V4L2 state management** — manages the V4L2 stateful decoder state machine (device open, buffer allocation, stream on/off)
* **Buffer I/O management** — handles buffer exchange between GStreamer and the V4L2 driver using DMABuf, MMAP, or UserPtr modes
* **Format negotiation** — negotiates raw output formats supported by the hardware
* **Multi-stream support** — supports multiple concurrent decoder instances (subject to hardware resource limits)
* **Error handling** — supports decoder error controls such as `max-errors` and corrupted-frame discard behavior

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html)<br />
         <Icon icon="arrow-turn-down-right" />GstVideoDecoder<br />
            <Icon icon="arrow-turn-down-right" />GstV4l2VideoDec<br />
               <Icon icon="arrow-turn-down-right" />v4l2h264dec

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-h264` | `stream-format: byte-stream` <br /> `alignment: au` <br /> `profiles: baseline, main, high, high-10, high-4:2:2, and more` <br /> `levels: 1–6.2` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { NV12_Q08C, NV12_Q10LE32C, NV12, NV21 }` <br /> `width: [1, 32768]` <br /> `height: [1, 32768]` |
| Availability: *Always* | |
| Direction: *source* | |

<Note>
  The exact set of profiles and levels reported depends on what the underlying V4L2 driver enumerates via `V4L2_CID_MPEG_VIDEO_H264_PROFILE` and `V4L2_CID_MPEG_VIDEO_H264_LEVEL`.
</Note>

# Element Properties

| Property | Description |
| - | - |
| `automatic-request-sync-point-flags` | Flags used when automatically requesting sync points.<br /><br />`Type: Flags`<br />`Default: 0x3 (corrupt-output+discard-input)`<br />`Flags: readable/writable` |
| `automatic-request-sync-points` | Automatically requests sync points such as keyframes when useful, for example after decoding errors.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `capture-io-mode` | I/O mode for the capture queue (src pad). Controls how decoded frame buffers are allocated and transferred.<br /><br />`Type: Enum `<br />`Default: 0, "auto"`<br />`Flags: readable/writable` |
| `device` | The V4L2 device node path. Set automatically at registration time and read-only after element creation.<br /><br />`Type: String`<br />`Default: "/dev/video32"`<br />`Flags: readable` |
| `device-fd` | File descriptor of the opened V4L2 device. A value of `-1` indicates that the device is not open.<br /><br />`Type: Integer`<br />`Default: -1`<br />`Flags: readable` |
| `device-name` | Human-readable name of the V4L2 device as reported by the driver via VIDIOC\_QUERYCAP.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable` |
| `discard-corrupted-frames` | When enabled, frames marked as corrupted by the driver are dropped instead of passed downstream.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `extra-controls` | Extra V4L2 controls (CIDs) to set on the device, specified as a GstStructure. Applied via VIDIOC\_S\_EXT\_CTRLS.<br /><br />`Type: GstStructure`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `max-errors` | Maximum number of consecutive decoder errors before the element returns a flow error. A value of `-1` means unlimited.<br /><br />`Type: Integer`<br />`Default: -1`<br />`Flags: readable/writable` |
| `min-force-key-unit-interval` | Minimum interval in nanoseconds between force-keyunit requests sent upstream. A value of `0` means no minimum interval.<br /><br />`Type: Unsigned Integer64`<br />`Default: 0`<br />`Flags: readable/writable` |
| `output-io-mode` | I/O mode for the output queue (sink pad). Controls how compressed input buffers are submitted to the driver.<br /><br />`Type: Enum `<br />`Default: 0, "auto"`<br />`Flags: readable/writable` |
| `qos` | Handles Quality-of-Service events from downstream. When enabled, the decoder may drop frames to maintain real-time playback.<br /><br />`Type: Boolean`<br />`Default: true`<br />`Flags: readable/writable` |

## I/O Mode Values

Both `capture-io-mode` and `output-io-mode` accept the same `GstV4l2IOMode` enumeration:

| Value | Integer | Description |
| - | - | - |
| `auto` | 0 | Automatically select the best I/O mode. Prefers DMABuf export for capture, MMAP for output. |
| `rw` | 1 | Use read/write system calls. Rarely used for M2M devices. |
| `mmap` | 2 | Use kernel memory-mapped buffers (`V4L2_MEMORY_MMAP`). |
| `userptr` | 3 | Use user-space pointer buffers (`V4L2_MEMORY_USERPTR`). |
| `dmabuf` | 4 | Export buffers as DMABuf file descriptors (`V4L2_MEMORY_DMABUF`). Enables zero-copy sharing with downstream GPU/display elements. |
| `dmabuf-import` | 5 | Import DMABuf file descriptors from an external allocator into the V4L2 queue. |

# Internal Architecture

`v4l2h264dec` operates using two V4L2 queue objects internally:

* **Output queue** (`V4L2_BUF_TYPE_VIDEO_OUTPUT`) — receives compressed bitstream buffers from upstream
* **Capture queue** (`V4L2_BUF_TYPE_VIDEO_CAPTURE`) — produces decoded raw video frames for downstream

<img alt="" />

# State Transitions

| Transition | Decoder Behavior | Notes |
| - | - | - |
| `NULL → READY` | Element becomes ready. Static configuration available. | Configure `capture-io-mode` and `output-io-mode` before active decoding. |
| `READY → PAUSED` | Decoder activates. V4L2 session setup begins. | Caps negotiation: sink is `video/x-h264`, src is `video/x-raw` (NV12/NV21). |
| `PAUSED → PLAYING` | Continuous decode starts. | With `capture-io-mode=4` and `output-io-mode=4`, pipeline is configured for DMABuf zero-copy. |
| `PLAYING → PAUSED` | Decoding paused at GStreamer scheduling level. | No full decoder teardown. Resume continues from current position. |
| `PLAYING/PAUSED → READY` | Decoder stops. V4L2 streaming stopped, buffers released. | Key cleanup transition. |
| `READY → NULL` | Element fully shut down. All resources released. | |
| EOS while PLAYING | Decoder drains pending input, outputs remaining frames, then forwards EOS. | |
| FLUSH | Current decode interrupted. Queued buffers discarded. | Common during seek or pipeline reset. Decoder waits for new keyframe after flush. |

# Dynamic Resolution Change

`v4l2h264dec` supports mid-stream dynamic resolution changes without requiring a pipeline restart, handled through the V4L2 source change event mechanism:

1. At initialization, the decoder subscribes to `V4L2_EVENT_SOURCE_CHANGE` events on the capture queue.
2. When the driver detects a resolution change in the bitstream, it signals the event.
3. The decoder stops the capture queue, discards the existing buffer pool, re-negotiates format via `VIDIOC_G_FMT`, sets a new output state with updated dimensions, and reallocates the capture buffer pool.

This allows seamless decoding of adaptive bitrate streams and content with embedded resolution changes.

# Memory and Buffer Management

## DMABuf Usage

Setting `capture-io-mode=4` exports decoded frame buffers as DMABuf file descriptors. These can be imported directly by `waylandsink` or `qtivcomposer` without CPU memory copies, enabling zero-copy pipelines.

## Alignment Requirements

Decoded buffers follow Qualcomm hardware alignment requirements (e.g., 128-byte stride alignment).

## Format Support

Standard NV12 is common. UBWC (Universal Bandwidth Compression) is supported through formats like `NV12_Q08C` to reduce memory bandwidth.

## Latency

The decoder computes and reports pipeline latency based on the minimum number of capture buffers required by the driver and the frame duration:

```
latency = min_buffers × frame_duration
```

## Drain and Flush

* **Drain** — sends `V4L2_DEC_CMD_STOP` to signal end-of-stream; waits for all remaining frames to be produced
* **Flush** — stops both queues, resets buffer pools, and restarts streaming; used during seek operations

# Usage

<Note>
  Ensure you have followed the [prerequisites](/plugin-reference/v4l2h264dec#example-pipeline) before continuing
</Note>

### Single RTSP Stream — Decode and Display

Demonstrates real-time RTSP stream reception, H.264 network depayloading, and hardware-accelerated decoding with DMABuf zero-copy for Wayland preview.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e \
  rtspsrc location="rtsp://<user>:<password>@<ip>:<port>/<channel>" ! \
  queue ! rtpptdemux ! rtph264depay ! h264parse ! \
  v4l2h264dec output-io-mode=dmabuf capture-io-mode=dmabuf ! \
  video/x-raw,format=NV12 ! queue ! \
  waylandsink fullscreen=true
```

### Four-Stream Side-by-Side Composition

Demonstrates decoding four input streams using four hardware decoder instances, arranging them in a 2×2 grid using `qtivcomposer`, and displaying the composed frame.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e --gst-debug=2 \
  qtivcomposer name=mixer \
    sink_0::position="<0, 0>"     sink_0::dimensions="<960, 540>" \
    sink_1::position="<960, 0>"   sink_1::dimensions="<960, 540>" \
    sink_2::position="<0, 540>"   sink_2::dimensions="<960, 540>" \
    sink_3::position="<960, 540>" sink_3::dimensions="<960, 540>" \
  ! queue ! waylandsink fullscreen=true sync=true enable-last-sample=false \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    qtdemux ! queue ! h264parse ! v4l2h264dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    qtdemux ! queue ! h264parse ! v4l2h264dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    qtdemux ! queue ! h264parse ! v4l2h264dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    qtdemux ! queue ! h264parse ! v4l2h264dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer.
```


# v4l2h264enc
Source: https://imsdkdocs.qualcomm.com/plugin-reference/v4l2h264enc

Hardware-accelerated H.264/AVC video encoder using the V4L2 stateful encoder API on Qualcomm hardware.

# Overview

`v4l2h264enc` is a hardware-accelerated video encoder that uses the Video4Linux2 (V4L2) stateful encoder API to offload H.264 (AVC) video encoding to the Qualcomm Video Processing Unit (VPU).

This plugin is provided and maintained by the GStreamer community. This document focuses on its usage in conjunction with Qualcomm-specific QIM SDK GStreamer plugins, along with relevant use cases and internal architectural considerations.

`v4l2h264enc` is typically used in encoding pipelines for:

* Camera capture and local recording
* Video transcoding (e.g., H.265 → H.264)
* Live streaming (RTSP, HLS, UDP)
* Multi-stream simultaneous encoding

The element is responsible **only for encoding** raw video frames. It does not mux or parse streams on its own — those operations must be handled by peer GStreamer elements downstream (e.g., `h264parse`, `mp4mux`).

## Example Pipeline

<img alt="" />

<Steps>
  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/media/output
    ```
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e \
      qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
      queue ! v4l2h264enc output-io-mode=dmabuf-import capture-io-mode=dmabuf ! \
      queue ! h264parse ! mp4mux ! \
      filesink location=$HOME/Downloads/qimsdk_samples/media/output/output.mp4
    ```
  </Step>
</Steps>

```bash theme={null}
```

# Key Responsibilities

`v4l2h264enc` is responsible for:

* **Hardware acceleration** — offloads H.264/AVC encoding to the dedicated VPU
* **V4L2 state management** — manages the V4L2 stateful encoder state machine (device open, buffer allocation, stream on/off)
* **Buffer I/O management** — handles buffer exchange between GStreamer and the V4L2 driver using DMABuf, MMAP, or UserPtr modes
* **Format negotiation** — negotiates raw input formats supported by the hardware, including UBWC-compressed formats for reduced memory bandwidth
* **Profile and level selection** — negotiates H.264 profile and level with downstream elements based on hardware capabilities probed at registration time
* **Encoder parameter control** — supports runtime configuration of encoding parameters (bitrate, GOP size, QP values, entropy mode, etc.) via the `extra-controls` property
* **Multi-stream support** — supports multiple concurrent encoder instances (subject to hardware resource limits)

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html)<br />
         <Icon icon="arrow-turn-down-right" />GstVideoEncoder<br />
            <Icon icon="arrow-turn-down-right" />GstV4l2VideoEnc<br />
               <Icon icon="arrow-turn-down-right" />v4l2h264enc

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { NV12_Q08C, NV12_Q10LE32C, NV12, NV21 }` <br /> `width: [1, 32768]` <br /> `height: [1, 32768]` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `video/x-h264` | `stream-format: byte-stream` <br /> `alignment: au` <br /> `profiles: baseline, main, high, high-10, high-4:2:2, and more` <br /> `levels: 1–6.2` |
| Availability: *Always* | |
| Direction: *source* | |

<Note>
  The exact set of profiles and levels reported depends on what the underlying V4L2 driver enumerates via `V4L2_CID_MPEG_VIDEO_H264_PROFILE` and `V4L2_CID_MPEG_VIDEO_H264_LEVEL`.
</Note>

# Element Properties

| Property | Description |
| - | - |
| `capture-io-mode` | I/O mode for the capture queue (src pad). Controls how compressed output buffers are received from the driver.<br /><br />`Type: Enum `<br />`Default: 0, "auto"`<br />`Flags: readable/writable` |
| `device` | The V4L2 device node path. Set automatically at registration time and read-only after element creation.<br /><br />`Type: String`<br />`Default: "/dev/video32"`<br />`Flags: readable` |
| `device-fd` | File descriptor of the opened V4L2 device. A value of `-1` indicates that the device is not open.<br /><br />`Type: Integer`<br />`Default: -1`<br />`Flags: readable` |
| `device-name` | Human-readable name of the V4L2 device as reported by the driver via VIDIOC\_QUERYCAP.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable` |
| `extra-controls` | Extra V4L2 controls (CIDs) to set on the device, specified as a GstStructure. Applied via VIDIOC\_S\_EXT\_CTRLS and used to configure encoding parameters at runtime.<br /><br />`Type: GstStructure`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `output-io-mode` | I/O mode for the output queue (sink pad). Controls how raw input buffers are submitted to the driver.<br /><br />`Type: Enum`<br />`Default: 0, "auto"`<br />`Flags: readable/writable` |

## I/O Mode Values

Both `capture-io-mode` and `output-io-mode` accept the same `GstV4l2IOMode` enumeration:

| Value | Integer | Description |
| - | - | - |
| `auto` | 0 | Automatically select the best I/O mode. |
| `rw` | 1 | Use read/write system calls. Rarely used for M2M devices. |
| `mmap` | 2 | Use kernel memory-mapped buffers (`V4L2_MEMORY_MMAP`). |
| `userptr` | 3 | Use user-space pointer buffers (`V4L2_MEMORY_USERPTR`). |
| `dmabuf` | 4 | Export buffers as DMABuf file descriptors (`V4L2_MEMORY_DMABUF`). Use for `capture-io-mode` to export encoded output. |
| `dmabuf-import` | 5 | Import DMABuf file descriptors from an external allocator. Use for `output-io-mode` when upstream (camera, decoder) provides DMABuf handles for zero-copy input. |

# Internal Architecture

`v4l2h264enc` operates using two V4L2 queue objects internally:

* **Output queue** (`V4L2_BUF_TYPE_VIDEO_OUTPUT`) — receives raw video frame buffers from upstream
* **Capture queue** (`V4L2_BUF_TYPE_VIDEO_CAPTURE`) — produces encoded H.264 bitstream buffers for downstream

<img alt="" />

# State Transitions

| Transition | Encoder Behavior | Notes |
| - | - | - |
| `NULL → READY` | Element becomes ready. Static configuration available. | Configure `capture-io-mode`, `output-io-mode`, and `extra-controls` before active encoding. |
| `READY → PAUSED` | Encoder activates. V4L2 session setup begins. | Caps negotiation: sink is `video/x-raw`, src is `video/x-h264`. Profile and level are negotiated with downstream. |
| `PAUSED → PLAYING` | Continuous encoding starts. Raw frames consumed, H.264 AUs pushed downstream. | Normal running state. |
| `PLAYING → PAUSED` | Encoding paused at GStreamer scheduling level. | No full encoder teardown. Resume continues from current position. |
| `PLAYING/PAUSED → READY` | Encoder stops. V4L2 streaming stopped, buffers released. | Key cleanup transition for stopping recording or preparing for reuse. |
| `READY → NULL` | Element fully shut down. All resources released. | |
| EOS while PLAYING | Encoder drains pending raw input, outputs remaining encoded frames, then forwards EOS. | Sends `V4L2_ENC_CMD_STOP` via `VIDIOC_ENCODER_CMD`. |
| FLUSH | Current encode interrupted. Queued buffers discarded. | Common during seek or pipeline reset. Encoder waits for new valid input after flush. |

# Memory and Buffer Management

## DMABuf Usage

For zero-copy pipelines where raw frames originate from a hardware source (camera, GPU, or hardware decoder):

* Set `output-io-mode=dmabuf-import` (value `5`) to import DMABuf handles from upstream directly into the V4L2 output queue without CPU copies
* Set `capture-io-mode=dmabuf` (value `4`) to export encoded output as DMABuf file descriptors for downstream elements

## Alignment Requirements

Input buffers follow Qualcomm hardware alignment requirements (e.g., 128-byte stride alignment). UBWC-compressed input formats (`NV12_Q08C`) are supported for reduced memory bandwidth when the upstream source produces UBWC output.

## Encoder Parameter Control

Encoding parameters are configured via the `extra-controls` property using V4L2 control IDs. Common parameters include:

| Control | Description |
| - | - |
| `video_bitrate` | Target bitrate in bits per second |
| `video_gop_size` | GOP (Group of Pictures) size — interval between keyframes |
| `video_h264_i_frame_qp` | QP value for I-frames |
| `video_h264_p_frame_qp` | QP value for P-frames |
| `video_h264_b_frame_qp` | QP value for B-frames |
| `video_h264_entropy_mode` | Entropy coding mode (CAVLC=0, CABAC=1) |

Example:

```bash theme={null}
extra-controls="controls,video_bitrate=4000000,video_gop_size=30"
```

## Drain and Flush

* **Drain** — sends `V4L2_ENC_CMD_STOP` via `VIDIOC_ENCODER_CMD`; waits for all remaining encoded frames to be produced
* **Flush** — stops both queues, resets buffer pools, and restarts streaming; used during seek operations

# Usage

<Note>
  Ensure you have followed the [prerequisites](/plugin-reference/v4l2h264enc#example-pipeline) before continuing
</Note>

### Record Camera to H.264 MP4 File

Captures raw video from a camera source, encodes to H.264 with DMABuf zero-copy, and muxes into an MP4 container.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e \
  qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
  queue ! v4l2h264enc output-io-mode=dmabuf-import capture-io-mode=dmabuf ! \
  h264parse ! mp4mux ! \
  filesink location=$HOME/Downloads/qimsdk_samples/media/output/output.mp4
```

### Transcode H.265 to H.264

Decodes an H.265 stream from an MP4 file using the hardware decoder and re-encodes to H.264 using the hardware encoder. DMABuf is used for zero-copy transfer between decoder and encoder.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e \
  filesrc location=<h265_enc_mp4_file_path> ! qtdemux ! h265parse ! \
  v4l2h265dec output-io-mode=4 capture-io-mode=4 ! \
  queue ! v4l2h264enc output-io-mode=5 capture-io-mode=4 ! \
  h264parse ! mp4mux ! \
  filesink location=$HOME/Downloads/qimsdk_samples/media/output/output_h264.mp4
```

### Four-Stream Simultaneous Encoding

Encodes four concurrent H.264 streams from four sources using four independent hardware encoder instances.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e \
  videotestsrc num-buffers=200 ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
    queue ! v4l2h264enc output-io-mode=5 capture-io-mode=4 ! \
    h264parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/output0.mp4 \
  videotestsrc num-buffers=200 ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
    queue ! v4l2h264enc output-io-mode=5 capture-io-mode=4 ! \
    h264parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/output1.mp4 \
  videotestsrc num-buffers=200 ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
    queue ! v4l2h264enc output-io-mode=5 capture-io-mode=4 ! \
    h264parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/output2.mp4 \
  videotestsrc num-buffers=200 ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
    queue ! v4l2h264enc output-io-mode=5 capture-io-mode=4 ! \
    h264parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/output3.mp4
```


# v4l2h265dec
Source: https://imsdkdocs.qualcomm.com/plugin-reference/v4l2h265dec

Hardware-accelerated H.265/HEVC video decoder using the V4L2 stateful decoder API on Qualcomm hardware.

# Overview

`v4l2h265dec` is a hardware-accelerated video decoder that uses the Video4Linux2 (V4L2) stateful decoder API to offload H.265 (HEVC) video decoding to the Qualcomm Video Processing Unit (VPU).

This plugin is provided and maintained by the GStreamer community. This document focuses on its usage in conjunction with Qualcomm-specific QIM SDK GStreamer plugins, along with relevant use cases and internal architectural considerations.

`v4l2h265dec` is typically used to decode H.265 streams provided by:

* File sources
* RTSP sources
* HLS/HTTP streaming sources

The element is responsible **only for decoding** video frames. It does not demux or parse streams on its own — those operations must be handled by peer GStreamer elements upstream (e.g., `qtdemux` and `h265parse`).

## Example Pipeline

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    h265\_1080p.mp4 file is used here as an example.
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media"
      scp h265_1080p.mp4 <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export SRC_VIDEO_NAME=h265_1080p.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e \
      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
      qtdemux ! queue ! h265parse ! \
      v4l2h265dec capture-io-mode=4 output-io-mode=4 ! \
      video/x-raw,format=NV12 ! queue ! \
      waylandsink fullscreen=true sync=true
    ```
  </Step>
</Steps>

# Key Responsibilities

`v4l2h265dec` is responsible for:

* **Hardware acceleration** — offloads H.265/HEVC decoding to the dedicated VPU
* **V4L2 state management** — manages the V4L2 stateful decoder state machine (device open, buffer allocation, stream on/off)
* **Buffer I/O management** — handles buffer exchange between GStreamer and the V4L2 driver using DMABuf, MMAP, or UserPtr modes
* **Format negotiation** — negotiates raw output formats supported by the hardware, including UBWC-compressed formats (`NV12_Q08C`, `NV12_Q10LE32C`) for reduced memory bandwidth
* **Multi-stream support** — supports multiple concurrent decoder instances (subject to hardware resource limits)
* **Error handling** — supports decoder error controls such as `max-errors` and corrupted-frame discard behavior

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html)<br />
         <Icon icon="arrow-turn-down-right" />GstVideoDecoder<br />
            <Icon icon="arrow-turn-down-right" />GstV4l2VideoDec<br />
               <Icon icon="arrow-turn-down-right" />v4l2h265dec

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-h265` | `stream-format: byte-stream` <br /> `alignment: au` <br /> `profiles: { main, main-still-picture, main-10 }` <br /> `levels: 1–6.2` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { NV12_Q08C, NV12_Q10LE32C, NV12, NV21 }` <br /> `width: [1, 32768]` <br /> `height: [1, 32768]` |
| Availability: *Always* | |
| Direction: *source* | |

<Note>
  The exact set of profiles and levels reported depends on what the underlying V4L2 driver enumerates via `V4L2_CID_MPEG_VIDEO_HEVC_PROFILE` and `V4L2_CID_MPEG_VIDEO_HEVC_LEVEL`.
</Note>

# Element Properties

| Property | Description |
| - | - |
| `automatic-request-sync-point-flags` | Flags used when automatically requesting sync points.<br /><br />`Type: Flags`<br />`Default: 0x3 (corrupt-output+discard-input)`<br />`Flags: readable/writable` |
| `automatic-request-sync-points` | Automatically requests sync points such as keyframes or IDR frames when useful, for example after decoding errors.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `capture-io-mode` | I/O mode for the capture queue (src pad). Controls how decoded frame buffers are allocated and transferred.<br /><br />`Type: Enum `<br />`Default: 0, "auto"`<br />`Flags: readable/writable` |
| `device` | The V4L2 device node path. Set automatically at registration time and read-only after element creation.<br /><br />`Type: String`<br />`Default: "/dev/video32"`<br />`Flags: readable` |
| `device-fd` | File descriptor of the opened V4L2 device. A value of `-1` indicates that the device is not open.<br /><br />`Type: Integer`<br />`Default: -1`<br />`Flags: readable` |
| `device-name` | Human-readable name of the V4L2 device as reported by the driver via VIDIOC\_QUERYCAP.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable` |
| `discard-corrupted-frames` | When enabled, frames marked as corrupted by the driver are dropped instead of passed downstream.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `extra-controls` | Extra V4L2 controls (CIDs) to set on the device, specified as a GstStructure. Applied via VIDIOC\_S\_EXT\_CTRLS.<br /><br />`Type: GstStructure`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `max-errors` | Maximum number of consecutive decoder errors before the element returns a flow error. A value of `-1` means unlimited.<br /><br />`Type: Integer`<br />`Default: -1`<br />`Flags: readable/writable` |
| `min-force-key-unit-interval` | Minimum interval in nanoseconds between force-keyunit requests sent upstream. A value of `0` means no minimum interval.<br /><br />`Type: Unsigned Integer64`<br />`Default: 0`<br />`Flags: readable/writable` |
| `output-io-mode` | I/O mode for the output queue (sink pad). Controls how compressed input buffers are submitted to the driver.<br /><br />`Type: Enum `<br />`Default: 0, "auto"`<br />`Flags: readable/writable` |
| `qos` | Handles Quality-of-Service events from downstream. When enabled, the decoder may drop frames to maintain real-time playback.<br /><br />`Type: Boolean`<br />`Default: true`<br />`Flags: readable/writable` |

## I/O Mode Values

Both `capture-io-mode` and `output-io-mode` accept the same `GstV4l2IOMode` enumeration:

| Value | Integer | Description |
| - | - | - |
| `auto` | 0 | Automatically select the best I/O mode. Prefers DMABuf export for capture, MMAP for output. |
| `rw` | 1 | Use read/write system calls. Rarely used for M2M devices. |
| `mmap` | 2 | Use kernel memory-mapped buffers (`V4L2_MEMORY_MMAP`). |
| `userptr` | 3 | Use user-space pointer buffers (`V4L2_MEMORY_USERPTR`). |
| `dmabuf` | 4 | Export buffers as DMABuf file descriptors (`V4L2_MEMORY_DMABUF`). Enables zero-copy sharing with downstream GPU/display elements. |
| `dmabuf-import` | 5 | Import DMABuf file descriptors from an external allocator into the V4L2 queue. |

# Internal Architecture

`v4l2h265dec` operates using two V4L2 queue objects internally:

* **Output queue** (`V4L2_BUF_TYPE_VIDEO_OUTPUT`) — receives compressed bitstream buffers from upstream
* **Capture queue** (`V4L2_BUF_TYPE_VIDEO_CAPTURE`) — produces decoded raw video frames for downstream

<img alt="" />

# State Transitions

| Transition | Decoder Behavior | Notes |
| - | - | - |
| `NULL → READY` | Element becomes ready. Static configuration available. | Configure `capture-io-mode` and `output-io-mode` before active decoding. |
| `READY → PAUSED` | Decoder activates. V4L2 session setup begins. | Caps negotiation: sink is `video/x-h265`, src is `video/x-raw` (NV12/NV12\_Q08C). |
| `PAUSED → PLAYING` | Continuous decode starts. | With `capture-io-mode=4` and `output-io-mode=4`, pipeline is configured for DMABuf zero-copy. |
| `PLAYING → PAUSED` | Decoding paused at GStreamer scheduling level. | No full decoder teardown. Resume continues from current position. |
| `PLAYING/PAUSED → READY` | Decoder stops. V4L2 streaming stopped, buffers released. | Key cleanup transition. |
| `READY → NULL` | Element fully shut down. All resources released. | |
| EOS while PLAYING | Decoder drains pending input, outputs remaining frames, then forwards EOS. | |
| FLUSH | Current decode interrupted. Queued buffers discarded. | Common during seek or pipeline reset. Decoder waits for new IDR frame after flush. |

# Dynamic Resolution Change

`v4l2h265dec` supports mid-stream dynamic resolution changes without requiring a pipeline restart, handled through the V4L2 source change event mechanism:

1. At initialization, the decoder subscribes to `V4L2_EVENT_SOURCE_CHANGE` events on the capture queue.
2. When the driver detects a resolution change in the bitstream, it signals the event.
3. The decoder stops the capture queue, discards the existing buffer pool, re-negotiates format via `VIDIOC_G_FMT`, sets a new output state with updated dimensions, and reallocates the capture buffer pool.

This allows seamless decoding of adaptive bitrate streams and content with embedded resolution changes.

# Memory and Buffer Management

## DMABuf Usage

Setting `capture-io-mode=4` exports decoded frame buffers as DMABuf file descriptors. These can be imported directly by `waylandsink` or `qtivcomposer` without CPU memory copies, enabling zero-copy pipelines.

## Alignment Requirements

Decoded buffers follow Qualcomm hardware alignment requirements (e.g., 128-byte stride alignment).

## Format Support

Standard NV12 is common. UBWC (Universal Bandwidth Compression) is supported through formats like `NV12_Q08C` to reduce memory bandwidth.

## Codec Header Initialization

Before the first frame is decoded, the decoder sends the codec-specific header data (VPS/SPS/PPS) to the output queue. If the input caps contain `codec_data` (e.g., from an `hvcC` box in MP4), that data is sent first. Otherwise, the first input buffer itself is used as the initialization data.

## Latency

The decoder computes and reports pipeline latency based on the minimum number of capture buffers required by the driver and the frame duration:

```
latency = min_buffers × frame_duration
```

## Drain and Flush

* **Drain** — sends `V4L2_DEC_CMD_STOP` to signal end-of-stream; waits for all remaining frames to be produced
* **Flush** — stops both queues, resets buffer pools, and restarts streaming; used during seek operations

# Usage

<Note>
  Ensure you have followed the [prerequisites](/plugin-reference/v4l2h265dec#example-pipeline) before continuing
</Note>

### Single RTSP Stream — Decode and Display

Demonstrates real-time RTSP stream reception, H.265 network depayloading, and hardware-accelerated decoding with DMABuf zero-copy for Wayland preview.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e \
  rtspsrc location="rtsp://<user>:<password>@<ip>:<port>/<channel>" ! \
  queue ! rtpptdemux ! rtph265depay ! h265parse ! \
  v4l2h265dec output-io-mode=dmabuf capture-io-mode=dmabuf ! \
  video/x-raw,format=NV12 ! queue ! \
  waylandsink fullscreen=true
```

### Four-Stream Side-by-Side Composition

Demonstrates decoding four input streams using four hardware decoder instances, arranging them in a 2×2 grid using `qtivcomposer`, and displaying the composed frame.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e --gst-debug=2 \
  qtivcomposer name=mixer \
    sink_0::position="<0, 0>"     sink_0::dimensions="<960, 540>" \
    sink_1::position="<960, 0>"   sink_1::dimensions="<960, 540>" \
    sink_2::position="<0, 540>"   sink_2::dimensions="<960, 540>" \
    sink_3::position="<960, 540>" sink_3::dimensions="<960, 540>" \
  ! queue ! waylandsink fullscreen=true sync=true enable-last-sample=false \
  filesrc location=/data/input0_1080p_h265.mp4 ! \
    qtdemux ! queue ! h265parse ! v4l2h265dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=/data/input1_1080p_h265.mp4 ! \
    qtdemux ! queue ! h265parse ! v4l2h265dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=/data/input2_1080p_h265.mp4 ! \
    qtdemux ! queue ! h265parse ! v4l2h265dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=/data/input3_1080p_h265.mp4 ! \
    qtdemux ! queue ! h265parse ! v4l2h265dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer.
```


# v4l2h265enc
Source: https://imsdkdocs.qualcomm.com/plugin-reference/v4l2h265enc

Hardware-accelerated H.265/HEVC video encoder using the V4L2 stateful encoder API on Qualcomm hardware.

# Overview

`v4l2h265enc` is a hardware-accelerated video encoder that uses the Video4Linux2 (V4L2) stateful encoder API to offload H.265 (HEVC) video encoding to the Qualcomm Video Processing Unit (VPU).

This plugin is provided and maintained by the GStreamer community. This document focuses on its usage in conjunction with Qualcomm-specific QIM SDK GStreamer plugins, along with relevant use cases and internal architectural considerations.

`v4l2h265enc` is typically used in encoding pipelines for:

* Camera capture and local recording with higher compression efficiency than H.264
* Video transcoding (e.g., H.264 → H.265)
* Live streaming (RTSP, HLS, UDP)
* 4K and HDR10 content encoding
* Multi-stream simultaneous encoding

The element is responsible **only for encoding** raw video frames. It does not mux or parse streams on its own — those operations must be handled by peer GStreamer elements downstream (e.g., `h265parse`, `mp4mux`).

## Example Pipeline

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `Draw_1080p_180s_30FPS.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media ~/Downloads/qimsdk_samples/media/output"
      scp Draw_1080p_180s_30FPS.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export SRC_VIDEO_NAME=Draw_1080p_180s_30FPS.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e \
      qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
      queue ! v4l2h265enc output-io-mode=dmabuf-import capture-io-mode=dmabuf ! \
      queue ! h265parse ! mp4mux ! \
      filesink location=$HOME/Downloads/qimsdk_samples/media/output/output.mp4
    ```
  </Step>
</Steps>

# Key Responsibilities

`v4l2h265enc` is responsible for:

* **Hardware acceleration** — offloads H.265/HEVC encoding to the dedicated VPU
* **V4L2 state management** — manages the V4L2 stateful encoder state machine (device open, buffer allocation, stream on/off)
* **Buffer I/O management** — handles buffer exchange between GStreamer and the V4L2 driver using DMABuf, MMAP, or UserPtr modes
* **Format negotiation** — negotiates raw input formats supported by the hardware, including UBWC-compressed formats for reduced memory bandwidth
* **Profile and level selection** — negotiates H.265 profile and level with downstream elements based on hardware capabilities probed at registration time
* **Encoder parameter control** — supports runtime configuration of encoding parameters (bitrate, GOP size, QP values, etc.) via the `extra-controls` property
* **Multi-stream support** — supports multiple concurrent encoder instances (subject to hardware resource limits)

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html)<br />
         <Icon icon="arrow-turn-down-right" />GstVideoEncoder<br />
            <Icon icon="arrow-turn-down-right" />GstV4l2VideoEnc<br />
               <Icon icon="arrow-turn-down-right" />v4l2h265enc

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { NV12_Q08C, NV12_Q10LE32C, NV12, NV21 }` <br /> `width: [1, 32768]` <br /> `height: [1, 32768]` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `video/x-h265` | `stream-format: byte-stream` <br /> `alignment: au` <br /> `profiles: { main, main-still-picture, main-10 }` <br /> `levels: 1–6.2` |
| Availability: *Always* | |
| Direction: *source* | |

<Note>
  The exact set of profiles and levels reported depends on what the underlying V4L2 driver enumerates via `V4L2_CID_MPEG_VIDEO_HEVC_PROFILE` and `V4L2_CID_MPEG_VIDEO_HEVC_LEVEL`.
</Note>

# Element Properties

| Property | Description |
| - | - |
| `capture-io-mode` | I/O mode for the capture queue (src pad). Controls how compressed output buffers are received from the driver.<br /><br />`Type: Enum `<br />`Default: 0, "auto"`<br />`Flags: readable/writable` |
| `device` | The V4L2 device node path. Set automatically at registration time and read-only after element creation.<br /><br />`Type: String`<br />`Default: "/dev/video32"`<br />`Flags: readable` |
| `device-fd` | File descriptor of the opened V4L2 device. A value of `-1` indicates that the device is not open.<br /><br />`Type: Integer`<br />`Default: -1`<br />`Flags: readable` |
| `device-name` | Human-readable name of the V4L2 device as reported by the driver via VIDIOC\_QUERYCAP.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable` |
| `extra-controls` | Extra V4L2 controls (CIDs) to set on the device, specified as a GstStructure. Applied via VIDIOC\_S\_EXT\_CTRLS. Used to configure encoding parameters at runtime.<br /><br />`Type: GstStructure`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `output-io-mode` | I/O mode for the output queue (sink pad). Controls how raw input buffers are submitted to the driver.<br /><br />`Type: Enum `<br />`Default: 0, "auto"`<br />`Flags: readable/writable` |

## I/O Mode Values

Both `capture-io-mode` and `output-io-mode` accept the same `GstV4l2IOMode` enumeration:

| Value | Integer | Description |
| - | - | - |
| `auto` | 0 | Automatically select the best I/O mode. |
| `rw` | 1 | Use read/write system calls. Rarely used for M2M devices. |
| `mmap` | 2 | Use kernel memory-mapped buffers (`V4L2_MEMORY_MMAP`). |
| `userptr` | 3 | Use user-space pointer buffers (`V4L2_MEMORY_USERPTR`). |
| `dmabuf` | 4 | Export buffers as DMABuf file descriptors (`V4L2_MEMORY_DMABUF`). Use for `capture-io-mode` to export encoded output. |
| `dmabuf-import` | 5 | Import DMABuf file descriptors from an external allocator. Use for `output-io-mode` when upstream (camera, decoder) provides DMABuf handles for zero-copy input. |

# Internal Architecture

`v4l2h265enc` operates using two V4L2 queue objects internally:

* **Output queue** (`V4L2_BUF_TYPE_VIDEO_OUTPUT`) — receives raw video frame buffers from upstream
* **Capture queue** (`V4L2_BUF_TYPE_VIDEO_CAPTURE`) — produces encoded H.265 bitstream buffers for downstream

<img alt="" />

# State Transitions

| Transition | Encoder Behavior | Notes |
| - | - | - |
| `NULL → READY` | Element becomes ready. Static configuration available. | Configure `capture-io-mode`, `output-io-mode`, and `extra-controls` before active encoding. |
| `READY → PAUSED` | Encoder activates. V4L2 session setup begins. | Caps negotiation: sink is `video/x-raw`, src is `video/x-h265`. Profile and level are negotiated with downstream. |
| `PAUSED → PLAYING` | Continuous encoding starts. Raw frames consumed, H.265 AUs pushed downstream. | Normal running state. |
| `PLAYING → PAUSED` | Encoding paused at GStreamer scheduling level. | No full encoder teardown. Resume continues from current position. |
| `PLAYING/PAUSED → READY` | Encoder stops. V4L2 streaming stopped, buffers released. | Key cleanup transition for stopping recording or preparing for reuse. |
| `READY → NULL` | Element fully shut down. All resources released. | |
| EOS while PLAYING | Encoder drains pending raw input, outputs remaining encoded frames, then forwards EOS. | Sends `V4L2_ENC_CMD_STOP` via `VIDIOC_ENCODER_CMD`. |
| FLUSH | Current encode interrupted. Queued buffers discarded. | Common during seek or pipeline reset. Encoder waits for new valid input after flush. |

# Memory and Buffer Management

## DMABuf Usage

For zero-copy pipelines where raw frames originate from a hardware source (camera, GPU, or hardware decoder):

* Set `output-io-mode=dmabuf-import` (value `5`) to import DMABuf handles from upstream directly into the V4L2 output queue without CPU copies
* Set `capture-io-mode=dmabuf` (value `4`) to export encoded output as DMABuf file descriptors for downstream elements

## Alignment Requirements

Input buffers follow Qualcomm hardware alignment requirements (e.g., 128-byte stride alignment). UBWC-compressed input formats (`NV12_Q08C`, `NV12_Q10LE32C`) are supported for reduced memory bandwidth when the upstream source produces UBWC output.

## Encoder Parameter Control

Encoding parameters are configured via the `extra-controls` property using V4L2 control IDs. Common parameters include:

| Control | Description |
| - | - |
| `video_bitrate` | Target bitrate in bits per second |
| `video_gop_size` | GOP (Group of Pictures) size — interval between keyframes |
| `video_hevc_i_frame_qp` | QP value for I-frames |
| `video_hevc_p_frame_qp` | QP value for P-frames |
| `video_hevc_b_frame_qp` | QP value for B-frames |

Example:

```bash theme={null}
extra-controls="controls,video_bitrate=4000000,video_gop_size=30"
```

## Drain and Flush

* **Drain** — sends `V4L2_ENC_CMD_STOP` via `VIDIOC_ENCODER_CMD`; waits for all remaining encoded frames to be produced
* **Flush** — stops both queues, resets buffer pools, and restarts streaming; used during seek operations

# Usage

<Note>
  Ensure you have followed the [prerequisites](/plugin-reference/v4l2h265enc#example-pipeline) before continuing
</Note>

### Record Camera to H.265 MP4 File

Captures raw video from a camera source, encodes to H.265 with DMABuf zero-copy, and muxes into an MP4 container.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e \
  qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
  queue ! v4l2h265enc output-io-mode=dmabuf-import capture-io-mode=dmabuf ! \
  queue ! h265parse ! mp4mux ! \
  filesink location=$HOME/Downloads/qimsdk_samples/media/output/output.mp4
```

### Transcode H.264 to H.265

Decodes an H.264 stream from an MP4 file using the hardware decoder and re-encodes to H.265 using the hardware encoder. DMABuf is used for zero-copy transfer between decoder and encoder.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e \
  filesrc location=$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
  v4l2h264dec output-io-mode=4 capture-io-mode=4 ! \
  queue ! v4l2h265enc output-io-mode=5 capture-io-mode=4 ! \
  h265parse ! mp4mux ! \
  filesink location=$HOME/Downloads/qimsdk_samples/media/output/output_h265.mp4
```

### Four-Stream Simultaneous Encoding

Encodes four concurrent H.265 streams from four sources using four independent hardware encoder instances.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e \
  videotestsrc num-buffers=200 ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
    queue ! v4l2h265enc output-io-mode=5 capture-io-mode=4 ! \
    h265parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/output0.mp4 \
  videotestsrc num-buffers=200 ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
    queue ! v4l2h265enc output-io-mode=5 capture-io-mode=4 ! \
    h265parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/output1.mp4 \
  videotestsrc num-buffers=200 ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
    queue ! v4l2h265enc output-io-mode=5 capture-io-mode=4 ! \
    h265parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/output2.mp4 \
  videotestsrc num-buffers=200 ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
    queue ! v4l2h265enc output-io-mode=5 capture-io-mode=4 ! \
    h265parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/output3.mp4
```


# v4l2vp9dec
Source: https://imsdkdocs.qualcomm.com/plugin-reference/v4l2vp9dec

Hardware-accelerated VP9 video decoder using the V4L2 stateful decoder API on Qualcomm hardware.

# Overview

`v4l2vp9dec` is a hardware-accelerated video decoder that uses the Video4Linux2 (V4L2) stateful decoder API to offload VP9 video decoding to the Qualcomm Video Processing Unit (VPU).

This plugin is provided and maintained by the GStreamer community. This document focuses on its usage in conjunction with Qualcomm-specific QIM SDK GStreamer plugins, along with relevant use cases and internal architectural considerations.

`v4l2vp9dec` is typically used to decode VP9 streams provided by:

* File sources (WebM, Matroska/MKV containers)
* RTSP sources
* HLS/HTTP streaming sources (e.g., YouTube, web video)

The element is responsible **only for decoding** video frames. It does not demux or parse streams on its own — those operations must be handled by peer GStreamer elements upstream (e.g., `matroskademux`).

<Note>
  VP9 frames are self-contained and self-framing. Unlike H.264/H.265, no separate parse element (e.g., `vp9parse`) is required between the demuxer and `v4l2vp9dec`.
</Note>

## Example Pipeline

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    vp9\_1080p.webm file is used here as an example.
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media"
      scp vp9_1080p.webm <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export SRC_VIDEO_NAME=vp9_1080p.webm
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e \
      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
      matroskademux ! queue ! \
      v4l2vp9dec capture-io-mode=4 output-io-mode=4 ! \
      video/x-raw,format=NV12 ! queue ! \
      waylandsink fullscreen=true sync=true
    ```
  </Step>
</Steps>

# Key Responsibilities

`v4l2vp9dec` is responsible for:

* **Hardware acceleration** — offloads VP9 decoding to the dedicated VPU
* **V4L2 state management** — manages the V4L2 stateful decoder state machine (device open, buffer allocation, stream on/off)
* **Buffer I/O management** — handles buffer exchange between GStreamer and the V4L2 driver using DMABuf, MMAP, or UserPtr modes
* **Format negotiation** — negotiates raw output formats supported by the hardware, including UBWC-compressed formats (`NV12_Q08C`, `NV12_Q10LE32C`) for reduced memory bandwidth
* **Multi-stream support** — supports multiple concurrent decoder instances (subject to hardware resource limits)
* **Error handling** — supports decoder error controls such as `max-errors` and corrupted-frame discard behavior

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html)<br />
         <Icon icon="arrow-turn-down-right" />GstVideoDecoder<br />
            <Icon icon="arrow-turn-down-right" />GstV4l2VideoDec<br />
               <Icon icon="arrow-turn-down-right" />v4l2vp9dec

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-vp9` | `format: NA` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { NV12_Q08C, NV12_Q10LE32C, NV12, NV21 }` <br /> `width: [1, 32768]` <br /> `height: [1, 32768]` |
| Availability: *Always* | |
| Direction: *source* | |

<Note>
  The exact set of profiles reported depends on what the underlying V4L2 driver enumerates via `V4L2_CID_MPEG_VIDEO_VP9_PROFILE` (or `V4L2_CID_MPEG_VIDEO_VPX_PROFILE`). VP9 does not define decoder levels in the V4L2 API; therefore, no level probing is performed.
</Note>

# Element Properties

| Property | Description |
| - | - |
| `automatic-request-sync-point-flags` | Flags used when automatically requesting sync points.<br /><br />`Type: Flags`<br />`Default: 0x3 (corrupt-output+discard-input)`<br />`Flags: readable/writable` |
| `automatic-request-sync-points` | Automatically requests sync points (keyframes) when useful, such as after decoding errors.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `capture-io-mode` | I/O mode for the capture queue (src pad). Controls how decoded frame buffers are allocated and transferred.<br /><br />`Type: Enum `<br />`Default: 0, "auto"`<br />`Flags: readable/writable` |
| `device` | The V4L2 device node path. Set automatically at registration time. Read-only after element creation.<br /><br />`Type: String`<br />`Default: "/dev/video32"`<br />`Flags: readable` |
| `device-fd` | File descriptor of the opened V4L2 device. A value of `-1` indicates that the device is not open.<br /><br />`Type: Integer`<br />`Default: -1`<br />`Flags: readable` |
| `device-name` | Human-readable name of the V4L2 device as reported by the driver via VIDIOC\_QUERYCAP.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable` |
| `discard-corrupted-frames` | When enabled, frames marked as corrupted by the driver are dropped instead of being passed downstream.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `extra-controls` | Extra V4L2 controls (CIDs) to set on the device, specified as a GstStructure. Applied via VIDIOC\_S\_EXT\_CTRLS.<br /><br />`Type: GstStructure`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `max-errors` | Maximum number of consecutive decoder errors before the element returns a flow error. A value of `-1` means unlimited.<br /><br />`Type: Integer`<br />`Default: -1`<br />`Flags: readable/writable` |
| `min-force-key-unit-interval` | Minimum interval in nanoseconds between force-keyunit requests sent upstream. A value of `0` means no minimum interval.<br /><br />`Type: Unsigned Integer64`<br />`Default: 0`<br />`Flags: readable/writable` |
| `output-io-mode` | I/O mode for the output queue (sink pad). Controls how compressed input buffers are submitted to the driver.<br /><br />`Type: Enum `<br />`Default: 0, "auto"`<br />`Flags: readable/writable` |
| `qos` | Handles Quality-of-Service events from downstream. When enabled, the decoder may drop frames to maintain real-time playback.<br /><br />`Type: Boolean`<br />`Default: true`<br />`Flags: readable/writable` |

## I/O Mode Values

Both `capture-io-mode` and `output-io-mode` accept the same `GstV4l2IOMode` enumeration:

| Value | Integer | Description |
| - | - | - |
| `auto` | 0 | Automatically select the best I/O mode. Prefers DMABuf export for capture, MMAP for output. |
| `rw` | 1 | Use read/write system calls. Rarely used for M2M devices. |
| `mmap` | 2 | Use kernel memory-mapped buffers (`V4L2_MEMORY_MMAP`). |
| `userptr` | 3 | Use user-space pointer buffers (`V4L2_MEMORY_USERPTR`). |
| `dmabuf` | 4 | Export buffers as DMABuf file descriptors (`V4L2_MEMORY_DMABUF`). Enables zero-copy sharing with downstream GPU/display elements. |
| `dmabuf-import` | 5 | Import DMABuf file descriptors from an external allocator into the V4L2 queue. |

# Internal Architecture

`v4l2vp9dec` operates using two V4L2 queue objects internally:

* **Output queue** (`V4L2_BUF_TYPE_VIDEO_OUTPUT`) — receives compressed VP9 frame buffers from upstream
* **Capture queue** (`V4L2_BUF_TYPE_VIDEO_CAPTURE`) — produces decoded raw video frames for downstream

<img alt="" />

# State Transitions

| Transition | Decoder Behavior | Notes |
| - | - | - |
| `NULL → READY` | Element becomes ready. Static configuration available. | Configure `capture-io-mode` and `output-io-mode` before active decoding. |
| `READY → PAUSED` | Decoder activates. V4L2 session setup begins. | Caps negotiation: sink is `video/x-vp9`, src is `video/x-raw` (NV12/NV12\_Q08C). VP9 frames from WebM/MKV are self-framing; no separate parse element is required. |
| `PAUSED → PLAYING` | Continuous decode starts. | With `capture-io-mode=4` and `output-io-mode=4`, pipeline is configured for DMABuf zero-copy. |
| `PLAYING → PAUSED` | Decoding paused at GStreamer scheduling level. | No full decoder teardown. Resume continues from current position. |
| `PLAYING/PAUSED → READY` | Decoder stops. V4L2 streaming stopped, buffers released. | Key cleanup transition. |
| `READY → NULL` | Element fully shut down. All resources released. | |
| EOS while PLAYING | Decoder drains pending input, outputs remaining frames, then forwards EOS. | |
| FLUSH | Current decode interrupted. Queued buffers discarded. | Common during seek or pipeline reset. Decoder waits for new VP9 keyframe after flush. |

# Dynamic Resolution Change

`v4l2vp9dec` supports mid-stream dynamic resolution changes without requiring a pipeline restart, handled through the V4L2 source change event mechanism:

1. At initialization, the decoder subscribes to `V4L2_EVENT_SOURCE_CHANGE` events on the capture queue.
2. When the driver detects a resolution change in the bitstream, it signals the event.
3. The decoder stops the capture queue, discards the existing buffer pool, re-negotiates format via `VIDIOC_G_FMT`, sets a new output state with updated dimensions, and reallocates the capture buffer pool.

This allows seamless decoding of adaptive bitrate VP9 streams and content with embedded resolution changes.

# Memory and Buffer Management

## DMABuf Usage

Setting `capture-io-mode=4` exports decoded frame buffers as DMABuf file descriptors. These can be imported directly by `waylandsink` or `qtivcomposer` without CPU memory copies, enabling zero-copy pipelines.

## Alignment Requirements

Decoded buffers follow Qualcomm hardware alignment requirements (e.g., 128-byte stride alignment).

## Format Support

Standard NV12 is common. UBWC (Universal Bandwidth Compression) is supported through formats like `NV12_Q08C` to reduce memory bandwidth.

## Codec Header Initialization

VP9 is a frame-based codec — each VP9 frame is self-contained and does not require a separate out-of-band codec header. The first input buffer is used directly as the initialization data. If the input caps contain `codec_data` (e.g., VP9 codec private data from a WebM container), that data is sent first.

## Latency

The decoder computes and reports pipeline latency based on the minimum number of capture buffers required by the driver and the frame duration:

```
latency = min_buffers × frame_duration
```

## Drain and Flush

* **Drain** — sends `V4L2_DEC_CMD_STOP` to signal end-of-stream; waits for all remaining frames to be produced
* **Flush** — stops both queues, resets buffer pools, and restarts streaming; used during seek operations

# Usage

<Note>
  Ensure you have followed the [prerequisites](/plugin-reference/v4l2vp9dec#example-pipeline) before continuing
</Note>

### Decode VP9 from a WebM File and Display

Demonstrates hardware-accelerated VP9 decoding from a WebM file with DMABuf zero-copy for Wayland preview.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
  matroskademux ! queue ! \
  v4l2vp9dec output-io-mode=dmabuf capture-io-mode=dmabuf ! \
  video/x-raw,format=NV12 ! queue ! \
  waylandsink fullscreen=true
```

### Four-Stream Side-by-Side Composition

Demonstrates decoding four VP9 input streams using four hardware decoder instances, arranging them in a 2×2 grid using `qtivcomposer`, and displaying the composed frame.

<img alt="" />

```bash theme={null}
gst-launch-1.0 -e --gst-debug=2 \
  qtivcomposer name=mixer \
    sink_0::position="<0, 0>"     sink_0::dimensions="<960, 540>" \
    sink_1::position="<960, 0>"   sink_1::dimensions="<960, 540>" \
    sink_2::position="<0, 540>"   sink_2::dimensions="<960, 540>" \
    sink_3::position="<960, 540>" sink_3::dimensions="<960, 540>" \
  ! queue ! waylandsink fullscreen=true sync=true enable-last-sample=false \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    matroskademux ! queue ! v4l2vp9dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    matroskademux ! queue ! v4l2vp9dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    matroskademux ! queue ! v4l2vp9dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    matroskademux ! queue ! v4l2vp9dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer.
```


# waylandsink
Source: https://imsdkdocs.qualcomm.com/plugin-reference/waylandsink

Display plugin for weston compositor

# Overview

`waylandsink` is a GStreamer sink element used to render raw video frames to a display through a **Wayland-based compositor**. It supports rendering to the device’s built-in display as well as external displays connected via DisplayPort (DP) or HDMI. The element is based on the Wayland protocol and operates as a client of a Wayland-based compositor.

This plugin is provided and maintained by the GStreamer community([waylandsink](https://gstreamer.freedesktop.org/documentation/waylandsink/index.html?gi-language=c))

This document focuses on its usage in conjunction with Qualcomm-specific QIM SDK GStreamer plugins, along with relevant use cases and internal architectural considerations.

waylandsink is typically used at the end of pipelines such as:

* video playback
* camera preview
* composed multi-stream display
* AI visualization pipelines

The element is responsible only for **displaying video frames**. It does not perform decoding, composition, or other video transformations on its own. Those operations must be handled by peer GStreamer elements before the video frames reaches `waylandsink`.

When required, `waylandsink` can create its own top-level Wayland window. It can also render into an application-provided Wayland surface through the `GstVideoOverlay` interface, allowing it to be embedded into a larger application UI.

<img alt="" />

## Example Pipeline

<img alt="" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/media"
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    Run below command on your device

    ```bash theme={null}
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    ```
  </Step>

  <Step title="Run the pipeline">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    waylandsink fullscreen=true sync=true enable-last-sample=false
    ```
  </Step>
</Steps>

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)<br />
   <Icon icon="arrow-turn-down-right" />[GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html)<br />
      <Icon icon="arrow-turn-down-right" />[GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html)<br />
         <Icon icon="arrow-turn-down-right" />[GstBaseSink](https://gstreamer.freedesktop.org/documentation/base/gstbasesink.html)<br />
            <Icon icon="arrow-turn-down-right" />[GstVideoSink](https://gstreamer.freedesktop.org/documentation/video/gstvideosink.html)<br />
               <Icon icon="arrow-turn-down-right" />[waylandsink](https://gstreamer.freedesktop.org/documentation/waylandsink/index.html)

# Pad templates

### sink

| Capabilities | |
| - | - |
| `video/x-raw` | `format: { BGR10A2_LE, RGB10A2_LE, AYUV, RGBA, ARGB, BGRA, ABGR, BGR10x2_LE, RGB10x2_LE, P010_10LE, NV12_10LE40, Y444, v308, RGBx, xRGB, BGRx, xBGR, RGB, BGR, Y42B, NV16, NV61, YUY2, YVYU, UYVY, I420, YV12, NV12, NV21, Y41B, YUV9, YVU9, BGR16, RGB16, NV12_Q08C }` <br /> `width: [1, 2147483647]` <br /> `height: [1, 2147483647]` <br /> `framerate: [0/1, 2147483647/1]` |
| `video/x-raw(memory:DMABuf)` | `format: { DMA_DRM }` <br /> `width: [1, 2147483647]` <br /> `height: [1, 2147483647]` <br /> `framerate: [0/1, 2147483647/1]` |
| Availability: *Always* | |
| Direction: *sink* | |

# Element Properties

| Property | Description |
| - | - |
| `display` | Wayland display name to connect to, if a display is not provided through `GstContext`.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `drm-device` | DRM device selection property available in newer upstream versions. Marked as construct-only in upstream documentation.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `force-aspect-ratio` | Preserves the video aspect ratio when rendering, where supported by the plugin version.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `fullscreen` | Requests that the Wayland surface be made fullscreen.<br /><br />`Type: Boolean`<br />`Default: false`<br />`Flags: readable/writable` |
| `fullscreen-output` | Selects the fullscreen output where supported by the plugin version.<br /><br />`Type: String`<br />`Default: NULL`<br />`Flags: readable/writable` |
| `rotate-method` | Selects the video orientation or rotation method where supported by the plugin version.<br /><br />`Type: Enum `<br />`Default: identity`<br /> `Range:`<br />     `(0): identity - GST_VIDEO_ORIENTATION_IDENTITY` <br />     `(1): 90r - GST_VIDEO_ORIENTATION_90R`<br />     `(2): 180 - GST_VIDEO_ORIENTATION_180`<br />     `(3): 90l - GST_VIDEO_ORIENTATION_90L`<br />     `(4): horiz - GST_VIDEO_ORIENTATION_HORIZ`<br />     `(5): vert - GST_VIDEO_ORIENTATION_VERT`<br />     `(6): ul-lr - GST_VIDEO_ORIENTATION_UL_LR`<br />     `(7): ur-ll - GST_VIDEO_ORIENTATION_UR_LL`<br />     `(8): auto - GST_VIDEO_ORIENTATION_AUTO`<br />     `(9): custom - GST_VIDEO_ORIENTATION_CUSTOM` <br /> `Flags: readable/writable` <br /> `Example: rotate-method="identity" (or) rotate-method=0` |

# Internal Architecture

At a high level, the element receives raw video buffers from the GStreamer pipeline, converts or imports them into Wayland-compatible buffers when needed, attaches them to a Wayland surface, and commits them for display through the compositor.

The main internal components are:

| Component | Role |
| - | - |
| `GstWaylandSink` | Core sink element implementation. Manages state transitions, caps negotiation, properties, buffer handling, and frame rendering. |
| `GstWlDisplay` | Manages the Wayland display connection and tracks compositor-supported buffer formats, including SHM and DMABuf formats. |
| `GstWlWindow` | Represents the Wayland rendering surface or window. It can be created internally by the sink or bound to an application-provided surface. |
| `GstWlBuffer` | Wraps a Wayland `wl_buffer` associated with a `GstBuffer`. |
| Wayland SHM allocator | Creates shared-memory Wayland buffers when rendering uses the SHM path. |
| Linux DMABuf import path | Imports DMABuf-backed video buffers into Wayland buffers when supported by the compositor. |

# State Transition behaviour

`waylandsink` behaviour is closely tied to normal GStreamer state transitions.

| State Transition | Behavior |
| - | - |
| `NULL → READY` | Creates or reuses a Wayland display connection. If no usable display is available, the transition fails. |
| `READY / PAUSED / PLAYING` | Negotiates caps, prepares buffer handling, and renders incoming frames. |
| `PAUSED → READY` | Clears the last rendered buffer and releases or resets the window state as required. |
| `READY → NULL` | Releases internal resources, including buffer pools and, when applicable, the Wayland display connection. |

# Caps Negotiation

During caps negotiation, `waylandsink`:

* Starts from its sink pad template caps.
* Queries the connected Wayland compositor for supported formats.
* Filters the advertised caps to formats that are both pipeline-compatible and compositor-supported.
* Stores the negotiated `GstVideoInfo`.
* Determines whether the rendering path will use DMA-Buf or SHM (Shared memory backend).
* Prepares an internal buffer pool If required.

# Caps Negotiation

The negotiated output path depends not only on the pipeline caps, but also on compositor capabilities and the memory type of the incoming buffers.

When a frame arrives at `waylandsink`, the element first checks whether a valid Wayland window or surface is available. If no surface is currently associated with the sink, it prepares to receive one through the `GstVideoOverlay` interface. If the application does not provide a window handle, `waylandsink` creates its own Wayland toplevel window.

Next, the element checks whether the incoming `GstBuffer` already has an associated Wayland buffer for the current display. If it does, the buffer can be rendered directly. If not, `waylandsink` attempts to create a Wayland-compatible buffer from the incoming memory.

The preferred path is to import the buffer through **DMABuf** when the input memory and compositor both support it. If DMABuf import is not possible, the element falls back to creating a **shared-memory (SHM)** Wayland buffer. If the incoming memory cannot be used directly through either path, the video frame is copied into an internal SHM-backed buffer pool.

Once a Wayland-compatible buffer has been prepared, it is attached to the Wayland surface and committed to the compositor for display.

# Buffer and Memory Management

`waylandsink` supports multiple rendering paths depending on negotiated caps, compositor support, and the memory backing the incoming buffers.

## Direct Wayland Buffer Reuse

If the incoming `GstBuffer` already carries a `GstWlBuffer` associated with the current Wayland display, the sink can render it directly. This is the preferred path because it avoids creating a new Wayland buffer for each frame.

## DMA Buffer Import

When peer upstream plugin provides `video/x-raw(memory:DMABuf)` buffers and the compositor supports the negotiated format, `waylandsink` imports the DMA Buffer into a Wayland buffer. This path reduces copies and is generally the most efficient option for display pipelines built around DMA-backed buffers.

## SHM Rendering

If the input memory is suitable for Wayland shared-memory rendering, the sink creates a Wayland SHM buffer and uses that for presentation.

## Internal Copy Fallback

If the incoming buffer cannot be reused or imported directly, `waylandsink` falls back to an internal buffer pool and copies the frame into SHM-compatible memory before rendering. This path ensures compatibility, but increases CPU usage and memory bandwidth compared to direct Wayland buffer reuse or DMA Buffer import.

## Buffer Lifetime

Once a buffer is attached to the Wayland surface, `waylandsink` keeps the corresponding GstBuffer alive until the compositor signals that it has finished using the underlying wl\_buffer. Only after that release event can the buffer be returned to the pool or freed.

## Frame Synchronization

`waylandsink` uses Wayland frame-callback mechanisms to synchronize presentation with the compositor. This prevents unbounded queuing of frames when the display path is slower than the incoming stream. If a previous frame is still pending presentation, newer frames may be dropped rather than queued indefinitely.

This behavior helps maintain responsive real-time display characteristics, especially in preview and live-rendering pipelines.

## GAP Buffer Handling

`waylandsink` is GAP-aware and correctly handles input buffers marked with `GST_BUFFER_FLAG_GAP`.

When a GAP buffer is received, the sink skips rendering for that input while preserving normal timing and synchronization behavior. This allows the pipeline to represent the absence of valid frame data for a given timestamp without treating it as an error or stall.

**Call Flow**
Below is the call flow depicting how waylandsink is setup and a buffer flows through it:

<img alt="" />

# Usage

<Note>
  Ensure you have followed the [prerequisites](/plugin-reference/waylandsink#example-pipeline) before continuing
</Note>

Before running display pipelines on target, ensure the Wayland environment is configured for the target image.

Use the path appropriate for the platform

### Two-stream Picture-in-Picture display

This example demonstrates picture-in-picture composition of two decoded input streams using [`qtivcomposer`](/plugin-reference/qtivcomposer). The two streams are combined into a single composed output, which is then rendered for display using `waylandsink`.

<img alt="" />

```
gst-launch-1.0 -e --gst-debug=2 \
  qtivcomposer name=mixer \
    sink_0::position="<0, 0>" \
    sink_0::dimensions="<1920, 1080>" \
    sink_1::position="<1440, 810>" \
    sink_1::dimensions="<480, 270>" \
  ! queue ! \
  waylandsink fullscreen=true sync=true enable-last-sample=false \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    qtdemux ! queue ! h264parse ! v4l2h264dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    qtdemux ! queue ! h264parse ! v4l2h264dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer.
```

[`qtivcomposer`](/plugin-reference/qtivcomposer) performs the Picture-in-Picture layout. `waylandsink` displays the final composed frame.

### Four-stream side-by-side video composition

This pipeline decodes four input streams, arranges them in a 2x2 grid using [`qtivcomposer`](/plugin-reference/qtivcomposer), and displays the composed frame using `waylandsink`.

<img alt="" />

```
gst-launch-1.0 -e --gst-debug=2 \
  qtivcomposer name=mixer \
    sink_0::position="<0, 0>" \
    sink_0::dimensions="<960, 540>" \
    sink_1::position="<960, 0>" \
    sink_1::dimensions="<960, 540>" \
    sink_2::position="<0, 540>" \
    sink_2::dimensions="<960, 540>" \
    sink_3::position="<960, 540>" \
    sink_3::dimensions="<960, 540>" \
  ! queue ! \
  waylandsink fullscreen=true sync=true enable-last-sample=false \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    qtdemux ! queue ! h264parse ! v4l2h264dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    qtdemux ! queue ! h264parse ! v4l2h264dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    qtdemux ! queue ! h264parse ! v4l2h264dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer. \
  filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! \
    qtdemux ! queue ! h264parse ! v4l2h264dec output-io-mode=4 capture-io-mode=4 ! queue ! mixer.
```

The 2x2 layout is performed by [`qtivcomposer`](/plugin-reference/qtivcomposer). `waylandsink` is only the display endpoint.


# Concepts of Building Native GStreamer Application
Source: https://imsdkdocs.qualcomm.com/qimsdk-overview/building-native-app



This guide walks through the essential steps and concepts required to build a native GStreamer application.

## Overview

A GStreamer application is built around a **pipeline** — a runtime container that organizes elements into a coherent data path and manages their execution as a single unit. Elements are linked in data-flow order to form a complete processing chain, and execution is governed by a state model that propagates lifecycle transitions across all elements simultaneously.

<img alt="Pipeline Diagram" />

A **main loop** keeps the application alive during execution, dispatching bus messages, signals, and callbacks while keeping control logic cleanly decoupled from data processing. Shutdown is handled gracefully by sending an **End-of-Stream (EOS)** event, ensuring all in-flight data is fully processed before the pipeline transitions to a stopped state and resources are released.

### Application Lifecycle

A minimal GStreamer application follows a well-defined sequence of stages:

| Stage | Description |
| - | - |
| **Initialization** | Initialize the GStreamer library before any other operations. |
| **Pipeline Construction** | Create elements, add them to the pipeline, and link them in data-flow order. |
| **Configuration** | Set element properties and apply caps filters to define runtime behavior and format constraints. |
| **Execution** | Transition the pipeline to `PLAYING` to start data flow. |
| **Runtime Operation** | Run the main loop to keep the application alive and process events. |
| **Shutdown** | Stop the pipeline gracefully, release resources, and clean up. |

***

## Concepts

The following steps and concepts are required to build a complete GStreamer application:

* **GStreamer Initialization** — Preparing the library for use before any pipeline operations are performed.
* **Pipeline and Element Instantiation** — Creating the pipeline container, instantiating individual elements, adding them to the pipeline, and linking them in the intended data-flow order.
* **Pipeline Configuration** — Setting element properties programmatically or through pipeline strings to control runtime behavior.
* **Pipeline Execution States** — Understanding the GStreamer state machine (`NULL`, `READY`, `PAUSED`, `PLAYING`), how state transitions are applied, and how they propagate to all contained elements.
* **Caps Negotiation** — How GStreamer automatically negotiates compatible data formats between linked elements, and how caps filters are used to constrain or enforce specific formats.
* **Main Loop** — Keeping the application alive during pipeline execution and serving as the event dispatcher for signals, callbacks, and bus messages.
* **Interaction with Pipeline during Runtime via Bus** — The asynchronous communication channel through which the pipeline delivers runtime notifications — including errors, state changes, and end-of-stream conditions — to the application.
* **Dynamic Pads** — Handling elements such as demuxers that expose output pads only at runtime, requiring deferred linking via the `pad-added` signal.
* **Tee and Queue** — Splitting a stream into multiple parallel branches using `tee`, and decoupling branch execution using `queue` elements to prevent blocking and enable independent threading.
* **Error Handling** — Responding to pipeline errors delivered through the bus and performing a controlled shutdown.
* **Application Shutdown and Teardown** — Handling external interrupt signals cleanly using `g_unix_signal_add()`, with support for both immediate and graceful EOS-based shutdown strategies, followed by safely transitioning the pipeline to `NULL` and releasing all allocated resources.
* **Application Interaction via appsink** — Accessing pipeline data directly from application code using the `appsink` element for custom processing, integration with external systems, or frame export.

***

## 1. GStreamer Initialization

### Simple Initialization

Include `gst/gst.h` to access GStreamer library functions, and call `gst_init(&argc, &argv)` at the start of the application to initialize the library and parse GStreamer-specific command-line options.

```c theme={null}
#include <gst/gst.h>

int
main (int argc, char *argv[])
{
  gst_init (&argc, &argv);

  // ...

  return 0;
}
```

### The GOption Interface

GOption-based initialization can be used if the application needs to parse custom-defined input parameters.

```c theme={null}
#include <gst/gst.h>

int
main (int argc, char *argv[])
{
  gboolean silent = FALSE;
  gchar *savefile = NULL;
  GOptionContext *ctx;
  GError *err = NULL;

  GOptionEntry entries[] = {
    { "silent", 's', 0, G_OPTION_ARG_NONE, &silent,
      "do not output status information", NULL },
    { "output", 'o', 0, G_OPTION_ARG_STRING, &savefile,
      "save xml representation of pipeline to FILE and exit", "FILE" },
    { NULL }
  };

  ctx = g_option_context_new ("- Your application");
  g_option_context_add_main_entries (ctx, entries, NULL);
  g_option_context_add_group (ctx, gst_init_get_option_group ());
  if (!g_option_context_parse (ctx, &argc, &argv, &err)) {
    g_print ("Failed to initialize: %s\n", err->message);
    g_clear_error (&err);
    g_option_context_free (ctx);
    return 1;
  }
  g_option_context_free (ctx);

  printf ("Run me with --help to see the Application options appended.\n");

  return 0;
}
```

<Info>
  A `GOptionTable` can be used to define application-specific command-line options and passed to the GLib initialization function along with the option group returned from `gst_init_get_option_group()`. This ensures that application options are parsed alongside the standard GStreamer options.
</Info>

***

## 2. Pipeline and Element Instantiation

Constructing a pipeline follows three sequential steps:

1. **2.1** Creating elements as independent `GstElement` objects
2. **2.2** Adding them to the pipeline container
3. **2.3** Linking them in the intended data-flow order

Since a pipeline is itself a `GstBin`, elements are added using `gst_bin_add()` or `gst_bin_add_many()`, which transfers ownership to the pipeline and enables lifecycle and state management across all contained elements.

### 2.1 Creating Elements

Elements are `GstElement` objects that form the functional building blocks of a GStreamer pipeline, each responsible for a specific role in the data flow:

* **Source** — producing data
* **Transform** — processing data
* **Sink** — consuming data

An element is created using `gst_element_factory_make("factory-name", "instance-name")`.

```c theme={null}
// Create elements
source    = gst_element_factory_make ("v4l2src",        "usb_camera_src");
transform = gst_element_factory_make ("qtivtransform",  "video_transform");
sink      = gst_element_factory_make ("filesink",       "sink");

if (!source || !transform || !sink) {
  g_printerr ("ERROR: Failed to create elements.\n");
}
```

### 2.2 Adding Elements to the Pipeline

```c theme={null}
gst_bin_add_many (GST_BIN (appctx->pipeline), source, transform, sink, NULL);
```

* `appctx->pipeline` is a `GstPipeline` (a subclass of `GstBin`)
* `gst_bin_add_many` transfers **ownership** of the elements to the pipeline
* After this call, the pipeline manages their lifecycle and propagates state changes

### 2.3 Linking Elements

```c theme={null}
ret = gst_element_link_many (source, transform, sink, NULL);
if (!ret) {
  g_printerr ("ERROR: Failed to link the pipeline.\n");
}
```

The data path for this example is:

```
v4l2src → qtivtransform → filesink
```

* `v4l2src` — produces video data (from a camera)
* `qtivtransform` — processes or converts the data
* `filesink` — consumes and writes data to a file

### Constructing the Pipeline from a String

The entire process of creating, adding, and linking elements can also be done implicitly using a pipeline string:

```c theme={null}
gchar *pipeline_str =
  "v4l2src name=usb_camera_src ! "
  "qtivtransform name=video_transform ! "
  "filesink name=sink";

appctx->pipeline = gst_parse_launch (pipeline_str, &error);

if (!appctx->pipeline) {
  g_printerr ("ERROR: Failed to create pipeline: %s\n",
              error ? error->message : "unknown error");
  if (error)
    g_error_free (error);
}
```

***

## 3. Pipeline and Element Configuration

Once elements are created and placed in the pipeline, they must be configured to define their runtime behavior.

### Programmatic Configuration

#### Configuring the Source

```c theme={null}
// Configure source element (camera device)
g_object_set (source, "device", "/dev/video0", NULL);
```

#### Configuring the Sink

```c theme={null}
// Configure sink element (output file)
g_object_set (sink, "location", "/tmp/output.raw", NULL);
```

### Configuration via Pipeline String

When using `gst_parse_launch()`, element configuration is embedded directly in the pipeline string using `key=value` syntax:

```c theme={null}
gchar *pipeline_str =
  "v4l2src name=usb_camera_src device=/dev/video0 ! "
  "qtivtransform name=video_transform ! "
  "filesink name=sink location=/tmp/output.raw";

appctx->pipeline = gst_parse_launch (pipeline_str, &error);
```

### Comparison: Programmatic vs. String Configuration

| Approach | Best For | Flexibility |
| - | - | - |
| **Pipeline String** | Prototyping, static pipelines | Compact, readable, single expression |
| **Programmatic** | Production applications | Dynamic, runtime-configurable, supports post-creation property changes |

<Info>
  In practice, many applications **combine both approaches** — using a pipeline string to define the overall structure while retrieving specific elements by name for further programmatic configuration.
</Info>

***

## 4. Pipeline Execution States

Once the pipeline is constructed and configured, its execution is governed by a well-defined state model.

### 4.1 The State Model

GStreamer defines a small set of states that represent different stages in the lifecycle of a pipeline:

<img alt="Pipeline Diagram" />

| State | Description |
| - | - |
| `NULL` | Initial and final state. No resources are allocated; pipeline is completely inactive. |
| `READY` | Elements have prepared necessary resources (e.g., opened devices/files), but no data is flowing. |
| `PAUSED` | Pipeline is prepared for processing and may pre-roll initial buffers, but is not running in real time. |
| `PLAYING` | Fully active state — data flows continuously and all elements perform their intended processing. |

### 4.2 State Transitions

Pipeline execution is controlled through state transitions:

```c theme={null}
gst_element_set_state (pipeline, GST_STATE_PLAYING);
```

<Info>
  Transitioning from `NULL` to `PLAYING` implicitly passes through `READY` and `PAUSED`. The pipeline manages state propagation to all child elements automatically.
</Info>

***

## 5. Caps Negotiation

Beyond configuring element properties, building a functional pipeline requires that linked elements agree on the format of the data they exchange. GStreamer handles this automatically through **caps negotiation**.

### Role of Caps

Caps (short for **capabilities**) describe the structure of the data flowing between elements. They typically include:

* Media type (e.g., `video`, `audio`)
* Encoding format
* Resolution or sample rate
* Pixel format or layout

### Negotiation Process

When the pipeline transitions to `PAUSED` or `PLAYING`, GStreamer initiates the negotiation process:

1. Upstream elements propose their supported output formats.
2. Downstream elements select formats they can accept.
3. A final format agreement is established for each pad link.

### Influencing Negotiation with Caps Filters

Caps filters are used in the following scenarios:

| Scenario | Description |
| - | - |
| **Format selection** | Ensures a specific format is selected when multiple compatible formats exist. |
| **Pipeline consistency** | Enforces a uniform format across all processing stages. |
| **Format validation / debugging** | Explicitly defines expected upstream output for analysis. |

#### Setting Caps Programmatically

```c theme={null}
capsfilter = gst_element_factory_make ("capsfilter", "input_capsfilter");

caps = gst_caps_new_simple ("video/x-raw",
    "format",    G_TYPE_STRING,       "NV12",
    "width",     G_TYPE_INT,          1920,
    "height",    G_TYPE_INT,          1080,
    "framerate", GST_TYPE_FRACTION,   30, 1,
    NULL);

g_object_set (G_OBJECT (capsfilter), "caps", caps, NULL);
gst_caps_unref (caps);
```

#### Setting Caps via Pipeline String

```c theme={null}
gchar *pipeline_str =
  "v4l2src name=usb_camera_src device=/dev/video0 ! "
  "video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! "
  "qtivtransform name=video_transform ! "
  "filesink name=sink location=/tmp/output.raw";

appctx->pipeline = gst_parse_launch (pipeline_str, &error);
```

***

## 6. Main Loop

Once the pipeline is transitioned to `PLAYING`, the application must remain active for the duration of execution using `g_main_loop_run()`:

```c theme={null}
g_main_loop_run (appctx.mloop);
```

The main loop serves two purposes:

* **Thread synchronization** — blocks the main thread, keeping the application alive.
* **Event dispatching** — processes and routes events including signals, callbacks, and pipeline notifications.

### Exiting the Main Loop

```c theme={null}
g_main_loop_quit (appctx.mloop);
```

The loop is typically stopped under one of the following conditions:

| Condition | Description |
| - | - |
| **End-of-stream** | Triggered when a source reaches its natural end or after an explicit EOS event. |
| **External signal** | Such as `Ctrl+C`, used to initiate teardown in console-based applications. |
| **Application-specific condition** | Any internal logic that determines the pipeline has completed its intended operation. |

***

## 7. Interaction During Runtime

While the pipeline is running, the application interacts through:

1. State change requests
2. Bus messages
3. Callbacks

### Bus

The **GStreamer bus** provides an asynchronous communication channel from the pipeline to the application.

#### Accessing the Bus

```c theme={null}
GstBus *bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));

gst_bus_add_signal_watch (bus);
g_signal_connect (bus, "message::eos", G_CALLBACK (eos_cb), user_data);
```

#### Types of Messages

| Message Type | Description |
| - | - |
| **End-Of-Stream (EOS)** | All data has been fully processed; pipeline has reached the natural end of the stream. |
| **Error** | A failure has occurred inside the pipeline, requiring stop and cleanup. |
| **State Changes** | Informs the application of transitions between pipeline states; useful for debugging. |

#### Integration with the Main Loop

When a signal watch is registered, incoming bus messages are dispatched as callbacks through the main loop, allowing the application to react to pipeline events as part of the same event-driven model.

***

## 8. Dynamic Pads

Some elements do not expose all their pads at creation time — their pads appear **dynamically during runtime**. The most common example is a **demuxer** (e.g., `qtdemux`).

### Static vs. Dynamic Pads

| Type | Description |
| - | - |
| **Static Pads** | Available immediately after creation; can be linked directly via `gst_element_link()`. |
| **Dynamic Pads** | Created at runtime based on the actual data being processed; signaled via `pad-added`. |

### Linking Strategy

1. Create and add all relevant elements to the pipeline.
2. Link all statically available elements.
3. Register a callback for the `pad-added` signal on the dynamic element.
4. Perform the remaining links inside the callback when the pad becomes available.

```c theme={null}
g_signal_connect (demux, "pad-added", G_CALLBACK (on_pad_added), downstream_element);
```

```c theme={null}
static void
on_pad_added (GstElement *element, GstPad *pad, gpointer user_data)
{
  GstElement *next = (GstElement *) user_data;
  GstPad *sinkpad = gst_element_get_static_pad (next, "sink");

  if (!gst_pad_is_linked (sinkpad))
    gst_pad_link (pad, sinkpad);

  gst_object_unref (sinkpad);
}
```

### Behavior in Pipeline Strings

Higher-level elements like `decodebin` handle dynamic pads internally:

```
filesrc ! decodebin ! autovideosink
```

<Info>
  When working with lower-level or specialized elements such as explicit demuxers, manual pad handling is required.
</Info>

<Callout type="warning">
  Care must be taken to avoid linking the same pad more than once, particularly in callbacks that may be invoked multiple times.
</Callout>

***

## 9. Tee and Queue — Branching the Pipeline

Many real-world applications require the same data to be processed in multiple ways simultaneously. GStreamer supports this through **`tee`** and **`queue`**.

### Tee — Splitting the Stream

The `tee` element duplicates an incoming stream and forwards it to multiple output branches:

```
tee name=t
t. ! branch1
t. ! branch2
```

### Queue — Decoupling Branch Execution

A `queue` element must be inserted at the start of each branch:

```
tee name=t
t. ! queue ! branch1
t. ! queue ! branch2
```

A `queue` element introduces two key behaviors:

* **Buffering** — holds incoming buffers in internal storage, decoupling production and consumption rates.
* **Thread boundary** — data is pushed and consumed on separate threads, allowing each branch to execute independently.

### Queue Usage Beyond Branching

The `queue` element also benefits linear pipelines:

```
element1 ! queue ! element2
```

This pattern is particularly beneficial for:

| Scenario | Benefit |
| - | - |
| **Mixed HW/SW processing** | Prevents faster hardware components from being stalled by slower software ones. |
| **Variable processing latency** | Absorbs transient slowdowns and smooths out data flow. |
| **Backpressure isolation** | Prevents backpressure from propagating across pipeline boundaries. |

***

## 10. Error Handling

Errors are delivered via the **GStreamer bus** as `GST_MESSAGE_ERROR` messages. Typical causes include:

| Cause | Description |
| - | - |
| **Input/Source failures** | Cannot open or access a file, device, or network stream. |
| **Data integrity issues** | Decoders/parsers encounter malformed or unsupported input. |
| **Resource limitations** | Required hardware or software dependencies are unavailable. |
| **Caps negotiation failures** | Elements cannot agree on a compatible data format. |
| **Internal element failures** | An element encounters an unexpected condition. |

### Handling Error Messages

```c theme={null}
g_signal_connect (bus, "message::error", G_CALLBACK (error_cb), user_data);
```

```c theme={null}
static void
error_cb (GstBus *bus, GstMessage *msg, gpointer userdata)
{
  GError *err = NULL;
  gchar *debug_info = NULL;

  gst_message_parse_error (msg, &err, &debug_info);

  g_printerr ("ERROR: %s\n", err->message);

  if (debug_info)
    g_printerr ("Debug details: %s\n", debug_info);

  g_clear_error (&err);
  g_free (debug_info);

  g_main_loop_quit ((GMainLoop *) userdata);
}
```

### Recommended Error Response Sequence

<Steps>
  <Step title="Log the Error">
    Record both the error message and debug information for troubleshooting.
  </Step>

  <Step title="Stop the Main Loop">
    Terminate the event loop to halt further message processing.
  </Step>

  <Step title="Set Pipeline to GST_STATE_NULL">
    Stops all elements, terminates internal threads, and releases all allocated resources.
  </Step>

  <Step title="Perform Application Cleanup">
    Free application-level resources, destroy pipeline objects, and exit gracefully.
  </Step>
</Steps>

<Callout type="warning">
  Errors are **asynchronous** and can arrive at any time. They are typically **fatal** — in most cases, a current pipeline instance cannot continue execution. Recovery requires explicit design at the application level.
</Callout>

***

## 11. Application Shutdown and Teardown

Properly shutting down a GStreamer application involves two coordinated steps: handling the signal that initiates shutdown and stopping the pipeline in a controlled manner.

### Shutdown Strategies

<Tabs>
  <Tab title="Immediate Shutdown">
    Quit the main loop directly — fast but does not guarantee all in-flight buffers are processed.

    ```c theme={null}
    g_main_loop_quit (appctx.mloop);
    ```

    **Use case:** Live previews or non-persistent data streams.
  </Tab>

  <Tab title="Graceful Shutdown (Recommended)">
    Send an EOS event to the pipeline, allowing all in-flight data to be fully processed before stopping.

    ```c theme={null}
    gst_element_send_event (appctx.pipeline, gst_event_new_eos ());
    ```

    **Use case:** File-based outputs such as video recording.
  </Tab>
</Tabs>

### Comparison

| | Immediate Stop | Graceful Shutdown (EOS) |
| - | - | - |
| **Speed** | Fast | Slower |
| **In-flight data** | May be discarded | Fully processed |
| **Use case** | Live previews, non-critical streams | File recording, persistent output |

### Typical Signal Handler

```c theme={null}
gboolean
handle_interrupt_signal (gpointer userdata)
{
  GstAppContext *appctx = (GstAppContext *) userdata;

  g_print ("Received Ctrl+C, sending EOS...\n");

  if (appctx && appctx->pipeline)
    gst_element_send_event (appctx->pipeline, gst_event_new_eos ());
  else if (appctx && appctx->mloop)
    g_main_loop_quit (appctx->mloop);

  return G_SOURCE_CONTINUE;
}
```

The main loop is then exited from the EOS bus message callback:

```c theme={null}
g_signal_connect (bus, "message::eos", G_CALLBACK (eos_cb), appctx->mloop);
```

### Teardown

Once the main loop exits, transition the pipeline to `NULL` and release all resources:

```c theme={null}
gst_element_set_state (pipeline, GST_STATE_NULL);
```

### Handling Ctrl+C

```c theme={null}
g_unix_signal_add (SIGINT, handle_interrupt_signal, &appctx);
```

### Complete Graceful Shutdown Sequence

<Steps>
  <Step title="User presses Ctrl+C">
    SIGINT is triggered.
  </Step>

  <Step title="Signal routed to handler">
    `g_unix_signal_add()` routes SIGINT to the signal handler via the main loop.
  </Step>

  <Step title="Application sends EOS">
    `gst_event_new_eos()` is sent to the pipeline.
  </Step>

  <Step title="Pipeline finishes processing">
    All remaining in-flight data is processed.
  </Step>

  <Step title="Pipeline posts EOS message on bus">
    The pipeline notifies the application via the bus.
  </Step>

  <Step title="EOS callback quits the main loop">
    `g_main_loop_quit()` is called.
  </Step>

  <Step title="Application transitions pipeline to NULL">
    All resources are released and the application exits gracefully.
  </Step>
</Steps>

***

## 12. Application Interaction via appsink

The `appsink` element acts as a bridge between the pipeline's internal data flow and application-level code, enabling direct access to individual buffers as they are produced.

### How appsink Works

Register a callback for incoming samples:

```c theme={null}
g_signal_connect (appsink, "new-sample", G_CALLBACK (on_new_sample), user_data);
```

Inside the callback, retrieve and process the sample:

```c theme={null}
g_signal_emit_by_name (appsink, "pull-sample", &sample, &ret);

buffer = gst_sample_get_buffer (sample);
gst_buffer_map (buffer, &mapinfo, GST_MAP_READ);

// Process raw bytes from mapinfo.data ...

gst_buffer_unmap (buffer, &mapinfo);
gst_sample_unref (sample);
```

| API | Description |
| - | - |
| `pull-sample` | Retrieves the next available `GstSample` from the appsink. |
| `gst_sample_get_buffer()` | Extracts the underlying `GstBuffer` from the sample. |
| `gst_buffer_map()` | Maps the buffer memory into the application's address space with read access. |

### Use Cases

| Use Case | Description |
| - | - |
| **Frame export** | Capturing individual video frames for encoding, storage, or display outside GStreamer. |
| **External system integration** | Forwarding processed data to third-party libraries, inference engines, or custom rendering systems. |
| **Custom post-processing** | Applying application-specific logic not expressible as a standard GStreamer element. |

<Callout type="warning">
  **Resource management:** Always unmap buffers with `gst_buffer_unmap()` and unreference samples with `gst_sample_unref()` after processing. Failing to do so causes memory leaks or buffer pool exhaustion.

  **Callback efficiency:** The `new-sample` callback is invoked from within the pipeline's streaming thread. Avoid heavy or blocking operations inside the callback — offload long-running tasks to a separate thread.

  **Pipeline independence:** If the application cannot consume samples fast enough, the `appsink` queue may fill up. Control this behavior via `appsink` properties such as `max-buffers` and `drop`.
</Callout>


# Building Production Quality Pipelines
Source: https://imsdkdocs.qualcomm.com/qimsdk-overview/building-native-object-detection-application



## Overview

This guide walks through the complete implementation of a GStreamer-based object detection application using Qualcomm hardware-accelerated plugins. The application:

* Reads an **H.264-encoded MP4 file** from disk
* Performs **YOLOv8 object detection** using a TFLite inference pipeline
* **Overlays** detection bounding boxes and labels onto video frames
* **Renders** the annotated output to a Wayland display

<Info>
  The complete application source code is available in the [repository](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-detection-display-example). For foundational concepts — including pipeline construction, state management, caps negotiation, bus handling, and dynamic pads — refer to [Building a Native GStreamer Application](../qimsdk-overview/building-native-app).
</Info>

***

## Pipeline Diagram

<img alt="Pipeline Diagram" />

***

## Implementation Steps

<Steps>
  <Step title="Step 1 — Define the Application Context">
    The application consolidates all runtime state into a single context structure:

    ```c theme={null}
    struct _GstAppContext
    {
      GstElement *pipeline;
      GMainLoop  *mloop;
      gchar      *file;
      gchar      *model;
      gchar      *labels;
      GstElement *parse;
    };
    ```

    | Field | Description |
    | - | - |
    | `pipeline` | Top-level GStreamer pipeline object for all pipeline operations |
    | `mloop` | GLib main loop — keeps the application alive and dispatches events |
    | `file` | Path to the input media file |
    | `model` | Path to the TFLite inference model |
    | `labels` | Path to the class label file |
    | `parse` | Reference to `h264parse`, stored explicitly for deferred dynamic pad linking |

    <Info>
      This pattern avoids global variables, keeps application state self-contained, and simplifies callback implementations that require access to pipeline objects.
    </Info>
  </Step>

  <Step title="Step 2 — Create the Pipeline Object">
    ```c theme={null}
    pipeline = gst_pipeline_new ("gst-detection-display-example");
    appctx.pipeline = pipeline;
    ```

    A `GstPipeline` object is created using `gst_pipeline_new()`, establishing the top-level container that owns and manages all elements added to it.

    As a `GstBin` subclass, it is responsible for:

    * **Element lifecycle management**
    * **State propagation** across all contained elements
    * **Synchronization** of clocks and timing

    All subsequent pipeline operations — element management, state transitions, and bus access — are performed through this object.
  </Step>

  <Step title="Step 3 — Instantiate Pipeline Elements">
    ```c theme={null}
    source  = gst_element_factory_make ("filesrc",               "file_src");
    demux   = gst_element_factory_make ("qtdemux",               "file_qtdemux");
    parse   = gst_element_factory_make ("h264parse",             "file_h264_parse");
    decoder = gst_element_factory_make ("v4l2h264dec",           "v4l2h264dec");
    mlbin   = gst_element_factory_make ("qtimlvideotflitebin",   "qtimlvideotflitebin");
    overlay = gst_element_factory_make ("qtivoverlay",           "overlay");
    sink    = gst_element_factory_make ("waylandsink",           "waylandsink");
    ```

    Each element is instantiated using `gst_element_factory_make()`. The first argument identifies the element by its factory name; the second assigns a unique instance name (retrievable later via `gst_bin_get_by_name()`).

    | Element | Role |
    | - | - |
    | `filesrc` | Reads the input MP4 file from disk |
    | `qtdemux` | Demuxes the MP4 container into elementary streams |
    | `h264parse` | Parses the H.264 bitstream for downstream decoding |
    | `v4l2h264dec` | Hardware-accelerated H.264 video decoder |
    | `qtimlvideotflitebin` | Runs TFLite-based YOLOv8 object detection inference |
    | `qtivoverlay` | Renders detection bounding boxes and labels onto video frames |
    | `waylandsink` | Displays annotated video output on a Wayland surface |
  </Step>

  <Step title="Step 4 — Configure Elements">
    Before the pipeline starts, element properties are set to define runtime behavior.

    **Input source** — configure `filesrc` with the path to the input file:

    ```c theme={null}
    g_object_set (source, "location", appctx->file, NULL);
    ```

    **AI inference** — configure `qtimlvideotflitebin` with delegate, post-processing module, model, and labels:

    ```c theme={null}
    gst_element_set_enum_property (mlbin, "inference-delegate",  "external");
    gst_element_set_enum_property (mlbin, "postprocess-module",  "yolov8");

    g_object_set (mlbin,
        "inference-model",    appctx->model,
        "postprocess-labels", appctx->labels,
        NULL);
    ```

    | Property | Description |
    | - | - |
    | `inference-delegate` | Selects the hardware backend for model execution |
    | `postprocess-module` | Specifies the post-processing algorithm (`yolov8` decoding) |
    | `inference-model` | Path to the compiled TFLite model |
    | `postprocess-labels` | Path to the class label file |

    **Display sink** — configure `waylandsink` for synchronized, full-screen rendering:

    ```c theme={null}
    g_object_set (sink, "sync",       TRUE, NULL);
    g_object_set (sink, "fullscreen", TRUE, NULL);
    ```

    | Property | Description |
    | - | - |
    | `sync` | Enables clock-synchronized rendering for smooth playback |
    | `fullscreen` | Instructs the sink to occupy the full display surface |
  </Step>

  <Step title="Step 5 — Add Elements to the Pipeline">
    ```c theme={null}
    gst_bin_add_many (GST_BIN (appctx->pipeline),
        source, demux, parse, decoder, mlbin, overlay, sink, NULL);
    ```

    `gst_bin_add_many()` transfers ownership of all elements to the pipeline, making it responsible for their lifecycle — including state transitions and resource cleanup. At this point, elements share the same execution context but are **not yet connected**.
  </Step>

  <Step title="Step 6 — Link Static Pipeline Elements">
    ```c theme={null}
    gst_element_link (source, demux);
    gst_element_link_many (parse, decoder, mlbin, overlay, sink, NULL);
    ```

    This establishes the following static connections:

    ```
    filesrc → qtdemux
    h264parse → v4l2h264dec → qtimlvideotflitebin → qtivoverlay → waylandsink
    ```

    <Callout type="warning">
      The link between `qtdemux` and `h264parse` is intentionally omitted here. `qtdemux` exposes its output pads **dynamically at runtime** — this connection is established in Step 7 via dynamic pad handling.
    </Callout>
  </Step>

  <Step title="Step 7 — Handle Dynamic Pads (Demuxer)">
    `qtdemux` determines the number and type of elementary streams only after parsing input data, dynamically creating output pads for each discovered stream. Since these pads do not exist at construction time, they cannot be linked statically.

    A callback is registered for the `pad-added` signal on the demuxer:

    ```c theme={null}
    g_signal_connect (demux, "pad-added",
        G_CALLBACK (gst_app_qtdemux_pad_added_cb), appctx);
    ```

    When a new pad is exposed at runtime, the callback inspects its type and links it to `h264parse` if it carries an H.264 video stream — all other stream types (e.g., audio) are ignored:

    ```c theme={null}
    static void
    gst_app_qtdemux_pad_added_cb (GstElement *element, GstPad *srcpad, gpointer userdata)
    {
      gchar *name = gst_pad_get_name (srcpad);

      if (!g_str_has_prefix (name, "video/x-h264")) {
        g_free (name);
        return;
      }

      gst_app_link_dynamic_src_pad (srcpad, appctx->parse);
      g_free (name);
    }
    ```

    This completes the missing link:

    ```
    qtdemux → h264parse
    ```

    The **full data path is now established end-to-end**:

    ```
    filesrc → qtdemux → h264parse → v4l2h264dec → qtimlvideotflitebin → qtivoverlay → waylandsink
    ```
  </Step>

  <Step title="Step 8 — Set Up Bus Message Handling">
    The application retrieves the pipeline bus and registers callbacks for error, EOS, and warning messages:

    ```c theme={null}
    bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
    gst_bus_add_signal_watch (bus);

    g_signal_connect (bus, "message::error",   G_CALLBACK (gst_app_error_cb),   mloop);
    g_signal_connect (bus, "message::eos",     G_CALLBACK (gst_app_eos_cb),     mloop);
    g_signal_connect (bus, "message::warning", G_CALLBACK (gst_app_warning_cb), NULL);
    ```

    | Signal | Trigger | Action |
    | - | - | - |
    | `message::error` | Fatal error in a pipeline element | Log error and initiate shutdown |
    | `message::eos` | All data processed; stream end reached | Exit main loop and proceed to teardown |
    | `message::warning` | Non-fatal condition | Log warning; pipeline continues |
  </Step>

  <Step title="Step 9 — Handle Ctrl+C (Interrupt Signal)">
    A `SIGINT` handler is registered using `g_unix_signal_add()` to route the OS signal safely into the GLib main loop:

    ```c theme={null}
    g_unix_signal_add (SIGINT, gst_app_handle_interrupt_signal, &appctx);
    ```

    The signal handler checks the current pipeline state and responds accordingly:

    ```c theme={null}
    if (state == GST_STATE_PLAYING) {
      gst_element_send_event (appctx->pipeline, gst_event_new_eos ());
    } else {
      g_main_loop_quit (appctx->mloop);
    }
    ```

    | Pipeline State | Action |
    | - | - |
    | `GST_STATE_PLAYING` | Send EOS event — allows in-flight buffers to drain before stopping |
    | Any other state | Quit main loop immediately |

    <Info>
      Sending EOS before stopping is particularly important for file-based outputs where finalization steps (e.g., writing file headers/footers) must complete.
    </Info>
  </Step>

  <Step title="Step 10 — Start the Pipeline">
    ```c theme={null}
    gst_element_set_state (pipeline, GST_STATE_PLAYING);
    ```

    This single call propagates the state transition to all contained elements. Each element progresses through intermediate `READY` and `PAUSED` states before reaching `PLAYING`, at which point data begins to flow from `filesrc` through to `waylandsink`.
  </Step>

  <Step title="Step 11 — Run the Main Loop">
    ```c theme={null}
    g_main_loop_run (mloop);
    ```

    The application enters the GLib main loop, which:

    * **Blocks** the main thread for the duration of pipeline execution
    * **Dispatches** bus messages and invokes registered callbacks as events occur
    * **Exits** only when explicitly quit by the EOS callback, error handler, or interrupt signal handler
  </Step>

  <Step title="Step 12 — Handle End-of-Stream">
    When the pipeline processes the last buffer and EOS propagates to the sink, an EOS bus message is delivered to the registered callback:

    ```c theme={null}
    static void
    gst_app_eos_cb (GstBus *bus, GstMessage *msg, gpointer userdata)
    {
      g_main_loop_quit ((GMainLoop *) userdata);
    }
    ```

    The callback quits the main loop, allowing the main thread to resume and proceed with pipeline teardown and resource cleanup.
  </Step>

  <Step title="Step 13 — Handle Errors">
    If a fatal error occurs at any stage of pipeline execution, the error callback is invoked:

    ```c theme={null}
    static void
    gst_app_error_cb (GstBus *bus, GstMessage *msg, gpointer userdata)
    {
      GError *err       = NULL;
      gchar  *debug_info = NULL;

      gst_message_parse_error (msg, &err, &debug_info);
      g_printerr ("ERROR: %s\n", err->message);

      if (debug_info)
        g_printerr ("Debug details: %s\n", debug_info);

      g_clear_error (&err);
      g_free (debug_info);

      g_main_loop_quit ((GMainLoop *) userdata);
    }
    ```

    Errors are treated as **fatal**. The callback:

    1. Logs the error message and debug information for diagnostics
    2. Quits the main loop to initiate controlled shutdown
    3. The application subsequently transitions the pipeline to `NULL` and releases all resources
  </Step>

  <Step title="Step 14 — Teardown and Resource Cleanup">
    Once the main loop exits — whether due to EOS, error, or interrupt — the application performs an orderly teardown:

    ```c theme={null}
    gst_element_set_state (pipeline, GST_STATE_NULL);
    gst_object_unref (pipeline);
    g_main_loop_unref (mloop);
    ```

    | Call | Effect |
    | - | - |
    | `gst_element_set_state(pipeline, GST_STATE_NULL)` | Transitions pipeline and all elements to `NULL` — stops processing, releases device handles, closes files, frees internal memory |
    | `gst_object_unref(pipeline)` | Releases the application's reference to the pipeline; destroys it and all owned elements if no other references are held |
    | `g_main_loop_unref(mloop)` | Releases the main loop object and frees associated resources |

    <Info>
      This sequence ensures all resources are released in the correct order and that the application exits in a **well-defined, clean state**.
    </Info>
  </Step>
</Steps>

***

## Summary

This guide covered the full implementation of a GStreamer YOLOv8 object detection application across 14 steps:

| Phase | Steps | Key Operations |
| - | - | - |
| **Initialization** | 1—2 | Define app context, create pipeline |
| **Element Setup** | 3—5 | Instantiate, configure, and add elements |
| **Linking** | 6—7 | Static links + dynamic pad handling for `qtdemux` |
| **Event Handling** | 8—9 | Bus message callbacks, SIGINT handler |
| **Execution** | 10—11 | Set `PLAYING` state, run main loop |
| **Shutdown** | 12—14 | EOS/error handling, teardown and resource cleanup |


# Image Classification
Source: https://imsdkdocs.qualcomm.com/qimsdk-overview/classification

Building image classification pipelines with QIM SDK

In an image classification system, the pipeline analyzes each frame of a video stream and assigns labels that reflect the scene's content such as identified objects or scene categories. Let's walk through an example of building an image classification pipeline with the QIM SDK using the ResNeXt101 image classification model which can be [downloaded from Qualcomm's AI Hub](https://aihub.qualcomm.com/iot/models/resnext101).

Here is what our pipeline in this example will look like:

<img alt="Diagram of image classification pipeline" />

<Note>You can refer to the [Building AI Pipelines](../qimsdk-overview/sdkoverview) for more general information about each element of an AI pipeline</Note>

## Run example on device

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | ResNeXt101 W8A8 model | [Qualcomm AI Hub — ResNeXt101](https://aihub.qualcomm.com/iot/models/resnext101) | `resnet101-w8a8.tflite` |
    | Classification labels | <a href="../labels/imagenet.txt">imagenet.txt</a> | `imagenet.txt` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    Create the required directories and transfer the downloaded files to your device.

    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
      scp resnet101-w8a8.tflite  <user>@<device-ip>:~/models/
      scp imagenet.txt           <user>@<device-ip>:~/labels/
      scp ai_demo_sample.mp4                <user>@<device-ip>:~/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    ```bash theme={null}
    export MODEL_NAME=resnet101-w8a8.tflite
    export LABELS_NAME=imagenet.txt
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    export VIDEO_SOURCE="filesrc location=$HOME/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12"
    ```
  </Step>

  <Step title="Run example on device">
    <Tabs>
      <Tab title="GStreamer Command line">
        ```bash theme={null}
        gst-launch-1.0 $VIDEO_SOURCE ! \
          tee name=t \
          t. ! qtimlvconverter name=preprocess ! queue ! \
               qtimltflite name=inference delegate=external \
                 external-delegate-path=libQnnTFLiteDelegate.so \
                 external-delegate-options="QNNExternalDelegate,backend_type=htp;" \
                 model=$HOME/models/$MODEL_NAME ! queue ! \
               qtimlpostprocess name=postprocess results=1 module=mobilenet-softmax \
                 labels=$HOME/labels/$LABELS_NAME settings='{"confidence": 51.0}' ! \
               text/x-raw ! metamux. \
          t. ! qtimetamux name=metamux ! qtivoverlay ! waylandsink sync=true fullscreen=true
        ```
      </Tab>

      <Tab title="GStreamer native application">
        * **Application source code:** [gst-ai-video-classification](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-video-classification)

        * **Build your application:**

          <Tabs>
            <Tab title="Yocto">
              <a href="../advanced/yocto-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>

            <Tab title="Ubuntu">
              <a href="../advanced/ubuntu-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>
          </Tabs>

        * **Run:**

          ```bash theme={null}
          gst-ai-video-classification -s "$VIDEO_SOURCE" -o display
          ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

## Expected output

The result of the video classification is visually overlaid in the top-left corner of the frame.

<img alt="Image of a camel classification" />

## Stream Splitting via tee

One of the powerful features of **GStreamer** is the ability to split a video or audio stream into multiple branches, allowing the same stream to be processed or consumed in different ways simultaneously. In this example, we are using the `tee` element to split the original video stream. One branch runs through the AI processing pipeline to generate classifications, and at the end it is recombined with the original video stream so that the detected label can be displayed on top of the original image. Note that each branch connected to a tee runs on its own thread, so you often need `queue` elements after each branch to avoid blocking.

## Combining inference results with the original image

The [qtimetamux](../plugin-reference/qtimetamux) attaches the AI inference results to the original NV12 video frame as custom GStreamer metadata. This ensures synchronization between the video frame and its associated AI metadata, allowing downstream elements to seamlessly access both for visualization, network streaming, or automated decision-making.

## Adding a text overlay on top of the image

The [qtivoverlay](../plugin-reference/qtivoverlay) element reads the AI metadata and renders visual overlays, such as bounding boxes and labels, directly onto the video frame without requiring buffer duplication.


# Daisy Chaining
Source: https://imsdkdocs.qualcomm.com/qimsdk-overview/daisychaining

Daisy Chaining AI Models

Daisy chaining refers to the sequential execution of multiple machine learning models, where each model specializes in a particular task. This example chains a [YoloX](https://aihub.qualcomm.com/iot/models/yolox) object detection model with an [HRNet Pose](https://aihub.qualcomm.com/iot/models/hrnet_pose) estimation model — YoloX first detects people in the frame, then HRNet estimates the pose of each detected person.

<img alt="gst-ai-video-daisychain-new" />

**Stage 1 — Person Detection:** [`qtimlvconverter`](../plugin-reference/qtimlvconverter) converts the NV12 frame to a tensor. [`qtimltflite`](../plugin-reference/qtimltflite) runs FootTrackNet inference. [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) parses the output into bounding boxes for detected persons.

**Stage 2 — Pose Estimation:** [`qtimlvconverter`](../plugin-reference/qtimlvconverter) in `roi-batch-cumulative` mode crops and centers each detected person's bounding box into individual tensors. [`qtimltflite`](../plugin-reference/qtimltflite) runs HRNet inference per person. [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) produces keypoints and skeleton connections.

All results are mapped back to the original frame and rendered by [`qtivoverlay`](../plugin-reference/qtivoverlay).

## Run example on device

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_quantized.tflite` |
    | HRNet Pose W8A8 model | [Qualcomm AI Hub — HRNet Pose](https://aihub.qualcomm.com/iot/models/hrnet_pose) | `hrnet_pose_quantized.tflite` |
    | Detection labels | <a href="../labels/coco.txt">coco.txt</a> | `coco.txt` |
    | Pose labels | <a href="../labels/coco_pose.txt">coco\_pose.txt</a> | `coco_pose.txt` |
    | Pose settings | <a href="../labels/hrnet_settings.json">hrnet\_settings.json</a> | `hrnet_pose_settings.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    Create the required directories and transfer the downloaded files to your device.

    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
      scp yolox_quantized.tflite  <user>@<device-ip>:~/models/
      scp hrnet_pose_quantized.tflite        <user>@<device-ip>:~/models/
      scp coco.txt                     <user>@<device-ip>:~/labels/
      scp coco_pose.txt                <user>@<device-ip>:~/labels/
      scp hrnet_pose_settings.json     <user>@<device-ip>:~/labels/
      scp ai_demo_sample.mp4                    <user>@<device-ip>:~/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    ```bash theme={null}
    export MODEL_NAME_1=yolox_quantized.tflite
    export MODEL_NAME_2=hrnet_pose_quantized.tflite
    export LABELS_NAME_1=coco.txt
    export LABELS_NAME_2=coco_pose.txt
    export HRNET_SETTINGS=hrnet_pose_settings.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    export VIDEO_SOURCE="filesrc location=$HOME/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12"
    ```
  </Step>

  <Step title="Run example on device">
    <Tabs>
      <Tab title="GStreamer Command line">
        ```bash theme={null}
        gst-launch-1.0 $VIDEO_SOURCE ! \
          tee name=t1 \
          t1. ! qtimlvconverter name=detection-preprocess ! queue ! \
               qtimltflite name=detection-inference delegate=external \
                 external-delegate-path=libQnnTFLiteDelegate.so \
                 external-delegate-options="QNNExternalDelegate,backend_type=htp;" \
                 model=$HOME/models/$MODEL_NAME_1 ! queue ! \
               qtimlpostprocess name=detection-postprocess results=8 module=yolov8 \
                 labels=$HOME/labels/$LABELS_NAME_1 settings='{"confidence": 51.0}' ! \
               text/x-raw ! metamux_1. \
          t1. ! qtimetamux name=metamux_1 ! queue ! \
          tee name=t2 \
          t2. ! qtimlvconverter name=pose-preprocess mode=roi-batch-cumulative \
                 image-disposition=centre ! queue ! \
               qtimltflite name=pose-inference delegate=external \
                 external-delegate-path=libQnnTFLiteDelegate.so \
                 external-delegate-options="QNNExternalDelegate,backend_type=htp,htp_performance_mode=(string)2;" \
                 model=$HOME/models/$MODEL_NAME_2 ! queue ! \
               qtimlpostprocess name=pose-postprocess results=1 module=hrnet \
                 labels=$HOME/labels/$LABELS_NAME_2 \
                 settings=$HOME/labels/$HRNET_SETTINGS ! \
               text/x-raw ! metamux_2. \
          t2. ! qtimetamux name=metamux_2 ! qtivoverlay ! waylandsink fullscreen=true sync=true
        ```
      </Tab>

      <Tab title="GStreamer native application">
        * **Application source code:** [gst-ai-video-daisychain-pose-estimation](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-video-daisychain-pose-estimation)

        * **Build your application:**

          <Tabs>
            <Tab title="Yocto">
              <a href="../advanced/yocto-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>

            <Tab title="Ubuntu">
              <a href="../advanced/ubuntu-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>
          </Tabs>

        * **Run:**

          ```bash theme={null}
          gst-ai-video-daisychain-pose-estimation -s "$VIDEO_SOURCE" -o display
          ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

## Expected output

Bounding boxes (from YOLOX) and skeleton keypoints (from HRNet) are overlaid on each video frame in real time.

## Exploring output options

The [`waylandsink`](../plugin-reference/waylandsink) in the pipeline above can be replaced with other output elements:

**Encode to file:**

```bash theme={null}
v4l2h264enc output-io-mode=5 capture-io-mode=4 ! queue ! h264parse ! mp4mux ! filesink location=$HOME/media/output.mp4
```

**Save raw frames:**

```bash theme={null}
filesink location=$HOME/media/frame.bin
```

**Stream over RTSP:**

```bash theme={null}
v4l2h264enc output-io-mode=5 capture-io-mode=4 ! queue ! h264parse config-interval=1 ! queue ! qtirtspbin address=0.0.0.0 port=8900
```

Access the stream at `rtsp://<device-ip>:8900/live`.


# Data Sources
Source: https://imsdkdocs.qualcomm.com/qimsdk-overview/datasources



Every AI pipeline starts with a data source — the raw video or audio that the model will analyze. QIM SDK uses GStreamer plugins to connect your pipeline to whatever source you're working with: a video file for testing, a USB webcam on your desk, a built-in ISP camera on the device, or a live RTSP stream from a network camera.

Each source type produces frames in a slightly different format, but they all feed into the same pipeline structure. The sections below show the GStreamer snippet for each source type — copy the one that matches your setup and use it as the first stage of your pipeline.

<note>
  The examples on this page run commands on your device. You'll need SSH access with the device's IP address before proceeding. If you haven't done this yet, see the [Installation Guide](../installation).
</note>

<Steps>
  <Step title="Choose your source">
    Paste one of the following `export VIDEO_SOURCE` commands into your device shell:

    <AccordionGroup>
      <Accordion title="File input">
        Use a pre-recorded video file to test your pipeline. This is the easiest way to get started — you can replay the same clip repeatedly to validate your model.

        **1. Download sample video on your host machine**

        | File | Download | Save as |
        | - | - | - |
        | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

        Then copy it to your device:

        ```bash SCP (SSH) theme={null}
        ssh <user>@<device-ip> "mkdir -p ~/media"
        scp ai_demo_sample.mp4  <user>@<device-ip>:~/media/
        ```

        **2. Set the source**

        <CodeGroup>
          ```bash H.264 theme={null}
          export VIDEO_SOURCE="filesrc location=$HOME/media/ai_demo_sample.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12"
          ```

          ```bash H.265 theme={null}
          export VIDEO_SOURCE="filesrc location=$HOME/media/<H.265 encoded mp4 file> ! qtdemux ! h265parse ! v4l2h265dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12"
          ```
        </CodeGroup>
      </Accordion>

      <Accordion title="USB Camera">
        USB cameras show up as V4L2 devices (e.g. `/dev/video0`). Run this on the device to find your camera's node:

        ```bash theme={null}
        for dev in /dev/video*; do
          echo "$dev: $(cat /sys/class/video4linux/$(basename $dev)/name)"
        done
        ```

        Then set the source using the device node you identified:

        ```bash theme={null}
        export VIDEO_SOURCE="v4l2src device=/dev/video<#> ! video/x-raw,width=1920,height=1080"
        ```

        **Note**: Adjust the resolution if your camera does not support 1080p.
      </Accordion>

      <Accordion title="Built-in Camera">
        Built-in cameras are accessed using the [`qticamsrc`](../plugin-reference/qticamsrc) plugin, which handles ISP hardware initialization on Qualcomm SoCs.

        ```bash theme={null}
        export VIDEO_SOURCE="qticamsrc camera=0 ! video/x-raw,width=1920,height=1080"
        ```

        **Note**: If multiple cameras are connected, set the `camera` property to select the one you want.
      </Accordion>

      <Accordion title="RTSP (Network) Camera">
        RTSP cameras stream encoded video (H.264 or H.265) over the network. The `rtspsrc` plugin receives the stream and the hardware decoder converts it to NV12 for the pipeline.

        ```bash theme={null}
        export VIDEO_SOURCE="rtspsrc location=<RTSP-URL> ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12"
        ```

        The resolution is determined by the camera and cannot be changed in GStreamer.

        The RTSP URL format varies by manufacturer. Common formats:

        ```
        rtsp://username:password@ip_address:port/stream_path
        rtsp://ip_address:port/h264/ch1/main/av_stream
        rtsp://username:password@ip_address:port/Streaming/Channels/101
        ```

        If you're unsure, search for *"\[your camera model] RTSP URL"* or check the manufacturer's documentation.
      </Accordion>
    </AccordionGroup>
  </Step>

  <Step title="Run example on device">
    <img alt="data source" />

    This pipeline captures frames from the video source in NV12 (YUV) format. The command-line example writes the raw frames directly to a file with `filesink`, because `appsink` only delivers buffers to an application that pulls them. The native application uses `appsink` instead, which notifies the application when a new frame is available, with a `queue` letting the source and consumer run in parallel.

    <Tabs>
      <Tab title="GStreamer Command line">
        ```bash theme={null}
        gst-launch-1.0 -e $VIDEO_SOURCE ! \
          filesink location=$HOME/media/frame.bin
        ```
      </Tab>

      <Tab title="GStreamer native application">
        * **Application source code:** [gst-video-source-example](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-video-source-example)

        * **Build your application:**

          <Tabs>
            <Tab title="Yocto">
              <a href="../advanced/yocto-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>

            <Tab title="Ubuntu">
              <a href="../advanced/ubuntu-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>
          </Tabs>

        * **Run:**

          ```bash theme={null}
          gst-video-source-example -s "$VIDEO_SOURCE"
          ```
      </Tab>
    </Tabs>
  </Step>
</Steps>


# Image Segmentation
Source: https://imsdkdocs.qualcomm.com/qimsdk-overview/imgsegmentation



Segmentation tasks differ fundamentally from classification and detection. While classification and detection models output discrete results — class labels, bounding boxes, confidence scores — segmentation models generate pixel-wise masks that delineate object boundaries within each frame. This example uses the [DeepLabV3+ MobileNet](https://aihub.qualcomm.com/iot/models/deeplabv3_plus_mobilenet) model from Qualcomm AI Hub.

<img alt="gst-ai-video-segmentation-new" />

For segmentation, the [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) plugin outputs an RGBA image mask rather than structured metadata. This mask is blended with the original video frame using [`qtivcomposer`](../plugin-reference/qtivcomposer) with `sink_1::alpha=0.5`. The [`qtivoverlay`](../plugin-reference/qtivoverlay) plugin is not needed for segmentation.

The order of inputs to [`qtivcomposer`](../plugin-reference/qtivcomposer) matters — the video frame must be connected first, and the segmentation mask second, so the mask is correctly composited on top.

## Run example on device

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | DeepLabV3+ MobileNet W8A8 model | [Qualcomm AI Hub — DeepLabV3+](https://aihub.qualcomm.com/iot/models/deeplabv3_plus_mobilenet) | `deeplabv3_plus_mobilenet.tflite` |
    | Segmentation labels | <a href="../labels/dv3-argmax.json">dv3-argmax.json</a> | `dv3-argmax.json` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    Create the required directories and transfer the downloaded files to your device.

    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
      scp deeplabv3_plus_mobilenet.tflite  <user>@<device-ip>:~/models/
      scp dv3-argmax.json                   <user>@<device-ip>:~/labels/
      scp ai_demo_sample.mp4                         <user>@<device-ip>:~/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    ```bash theme={null}
    export MODEL_NAME=deeplabv3_plus_mobilenet.tflite
    export LABELS_NAME=dv3-argmax.json
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    export VIDEO_SOURCE="filesrc location=$HOME/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12"
    ```
  </Step>

  <Step title="Run example on device">
    <Tabs>
      <Tab title="GStreamer Command line">
        ```bash theme={null}
        gst-launch-1.0 $VIDEO_SOURCE ! \
          tee name=t \
          t. ! queue ! mixer. \
          t. ! qtimlvconverter name=preprocess ! queue ! \
               qtimltflite name=inference delegate=external \
                 external-delegate-path=libQnnTFLiteDelegate.so \
                 external-delegate-options="QNNExternalDelegate,backend_type=htp;" \
                 model=$HOME/models/$MODEL_NAME ! queue ! \
               qtimlpostprocess name=postprocess module=deeplab-argmax \
                 labels=$HOME/labels/$LABELS_NAME ! mixer. \
          qtivcomposer name=mixer sink_1::alpha=0.5 ! video/x-raw,format=NV12 ! \
          waylandsink sync=true fullscreen=true
        ```
      </Tab>

      <Tab title="GStreamer native application">
        * **Application source code:** [gst-ai-video-segmentation](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-video-segmentation)

        * **Build your application:**

          <Tabs>
            <Tab title="Yocto">
              <a href="../advanced/yocto-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>

            <Tab title="Ubuntu">
              <a href="../advanced/ubuntu-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>
          </Tabs>

        * **Run:**

          ```bash theme={null}
          gst-ai-video-segmentation -s "$VIDEO_SOURCE" -o display
          ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

## Expected output

The segmentation mask is blended on top of the original video frame in real time.

<img alt="image-segmentation-background" />


# Inferencing
Source: https://imsdkdocs.qualcomm.com/qimsdk-overview/inference

Pipeline construction for running AI inference

The inference plugin is responsible for executing the AI model on the prepared input tensor. The QIM SDK supports multiple inference runtimes, each encapsulated within a dedicated GStreamer plugin. This architecture allows for straightforward replacement and integration of inference engines, depending on the target platform or model format.

Supported runtimes include:

* [**qtimlsnpe**](../plugin-reference/qtimltflite) — SNPE (Qualcomm Neural Processing): Executes models in DLC format on Qualcomm Snapdragon platforms.
* [**qtimlqnn**](../plugin-reference/qtimlqnn) — QNN (Qualcomm AI Engine Direct): Supports models optimized for QNN.
* [**qtimltflite**](../plugin-reference/qtimltflite) — TFLite / Lite-RT: Enables execution of TensorFlow Lite models.
* [**qtimlonnx**](../plugin-reference/qtimlonnx) — Enables execution of ONNX models.

All plugins leverage hardware acceleration provided by Qualcomm NPUs and GPUs for optimal performance.

### Run example on device

The example below uses the [ResNeXt101](https://aihub.qualcomm.com/iot/models/resnext101) model with the [`qtimltflite`](../plugin-reference/qtimltflite) plugin to classify objects in a video stream.

<img alt="Inference pipeline diagram" />

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | ResNeXt101 W8A8 model | [Qualcomm AI Hub — ResNeXt101](https://aihub.qualcomm.com/iot/models/resnext101) | `resnext101-w8a8.tflite` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    Create the required directories and transfer the downloaded files to your device.

    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
      scp resnext101-w8a8.tflite  <user>@<device-ip>:~/models/
      scp ai_demo_sample.mp4                <user>@<device-ip>:~/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    ```bash theme={null}
    export MODEL_NAME=resnext101-w8a8.tflite
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    export VIDEO_SOURCE="filesrc location=$HOME/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12"
    ```
  </Step>

  <Step title="Run example on device">
    <Tabs>
      <Tab title="GStreamer Command line">
        ```bash theme={null}
        gst-launch-1.0 -e $VIDEO_SOURCE ! queue ! \
          qtimlvconverter name=preprocess ! queue ! \
          qtimltflite name=inference \
            delegate=external \
            external-delegate-path=libQnnTFLiteDelegate.so \
            external-delegate-options="QNNExternalDelegate,backend_type=htp;" \
            model=$HOME/models/$MODEL_NAME ! queue ! \
          multifilesink location=$HOME/media/tensor.bin sync=false
        ```
      </Tab>

      <Tab title="GStreamer native application">
        * **Application source code:** [gst-ai-video-inference](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-video-inference)

        * **Build your application:**

          <Tabs>
            <Tab title="Yocto">
              <a href="../advanced/yocto-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>

            <Tab title="Ubuntu">
              <a href="../advanced/ubuntu-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>
          </Tabs>

        * **Run:**

          ```bash theme={null}
          gst-ai-video-inference -s "$VIDEO_SOURCE"
          ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

### Expected output

The pipeline classifies objects in the video stream in real time. The [`qtimltflite`](../plugin-reference/qtimltflite) plugin automatically reads tensor specifications from the model and propagates them to adjacent plugins — no manual tensor configuration is required.

By default, all QIM SDK inference plugins perform dequantization on output tensors automatically.

Now that we are able to take video input from a data source and do hardware accelerated preprocessing and inferencing on each frame, let's turn our attention to the post processing of the results and generating meaningful data.


# Object Detection
Source: https://imsdkdocs.qualcomm.com/qimsdk-overview/objectdetection

Building object detection pipelines with QIM SDK

In this example, the pipeline systematically analyzes each frame of a video stream to identify and localize multiple objects — such as people, vehicles, or other entities — within each frame. For each detected object, the pipeline provides bounding boxes and confidence scores. This example uses the [YOLOX](https://aihub.qualcomm.com/iot/models/yolox) model from Qualcomm AI Hub.

<img alt="gst-ai-video-detection" />

The detection pipeline is structurally identical to the classification pipeline, with two key differences:

* The inference plugin is configured with a detection model (YOLOX) instead of a classification model
* The [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) plugin uses the `yolov8` module with a higher result count to capture multiple detections per frame

## Run example on device

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_quantized.tflite` |
    | Detection labels | <a href="../labels/coco.txt">coco.txt</a> | `coco.txt` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    Create the required directories and transfer the downloaded files to your device.

    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
      scp yolox_quantized.tflite  <user>@<device-ip>:~/models/
      scp coco.txt                 <user>@<device-ip>:~/labels/
      scp ai_demo_sample.mp4                <user>@<device-ip>:~/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    ```bash theme={null}
    export MODEL_NAME=yolox_quantized.tflite
    export LABELS_NAME=coco.txt
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    export VIDEO_SOURCE="filesrc location=$HOME/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12"
    ```
  </Step>

  <Step title="Run example on device">
    <Tabs>
      <Tab title="GStreamer Command line">
        ```bash theme={null}
        gst-launch-1.0 $VIDEO_SOURCE ! \
          tee name=t \
          t. ! qtimlvconverter name=preprocess ! queue ! \
               qtimltflite name=inference delegate=external \
                 external-delegate-path=libQnnTFLiteDelegate.so \
                 external-delegate-options="QNNExternalDelegate,backend_type=htp;" \
                 model=$HOME/models/$MODEL_NAME ! queue ! \
               qtimlpostprocess name=postprocess results=8 module=yolov8 \
                 labels=$HOME/labels/$LABELS_NAME settings='{"confidence": 51.0}' bbox-stabilization=true ! \
               text/x-raw ! metamux. \
          t. ! qtimetamux name=metamux ! qtivoverlay ! waylandsink sync=true fullscreen=true
        ```
      </Tab>

      <Tab title="GStreamer native application">
        * **Application source code:** [gst-ai-video-detection](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-video-detection)

        * **Build your application:**

          <Tabs>
            <Tab title="Yocto">
              <a href="../advanced/yocto-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>

            <Tab title="Ubuntu">
              <a href="../advanced/ubuntu-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>
          </Tabs>

        * **Run:**

          ```bash theme={null}
          gst-ai-video-detection -s "$VIDEO_SOURCE" -o display
          ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

## Expected output

Detection results are visually overlaid on each video frame — bounding boxes and class labels are rendered on top of the original image in real time.

<img alt="gst-ai-video-detection" />


# Postprocessing
Source: https://imsdkdocs.qualcomm.com/qimsdk-overview/postprocessing

Postprocessing of AI inference results

Output tensors produced by inference models typically require post-processing to render results usable for downstream components or interpretable by applications. For instance:

* **Classification outputs** are arrays of confidence scores that must be interpreted, such as by selecting the top classes exceeding a specified threshold.
* **Raw tensor data** may need conversion into formats expected by subsequent plugins or processing stages.

Post-processing ensures that output data is appropriately structured, filtered, and formatted for purposes such as visualization, logging, or further inference.

Within the QIM SDK, the [qtimlpostprocess](../plugin-reference/qtimlpostprocess) plugin manages post-processing tasks. This plugin converts raw model outputs into **GStreamer ML metadata**, which may include:

* Label strings (e.g., "cat", "car", "person")
* Confidence scores
* Color information (for visualization overlays)
* Bounding boxes
* Key points and their connections
* Segmentation masks
* Tensors (for cases where subsequent AI stages require modified outputs from previous models)

This metadata facilitates display, logging, and decision-making in downstream components.

Given the diversity of model architectures and output tensor types, each requires specific logic to interpret and convert raw data into meaningful results. To accommodate this, parsing logic is not embedded directly within the plugin; instead, it is provided as loadable modules tailored to individual models.

<img alt="Postprocess sample diagram with mobile-softmax module" />

Each post-processing module implements parsing logic tailored to a specific class of models. For example, dedicated modules are available for MobileNet, YOLOv5, YOLOv8, and others. The QIM SDK provides a comprehensive set of modules supporting object detection, image classification, semantic segmentation, super resolution, audio classification and pose estimation models.

The list of supported post-processing modules:

<AccordionGroup>
  <Accordion title="Image Classification">
    * `mobilenet-softmax`
    * `mobilenet`
    * `ocr-recognizer`
    * `ocr`
    * `qfr-softmax`
    * `qfr`
  </Accordion>

  <Accordion title="Object Detection">
    * `easy-textdt`
    * `easy-ocr-detector`
    * `mediapipe-pose`
    * `qfd`
    * `qpd`
    * `ssd-mobilenet`
    * `yolo-nas`
    * `yolov5`
    * `yolov8`
    * `palmd`
  </Accordion>

  <Accordion title="Semantic Segmentation">
    * `deeplab-argmax`
    * `yolov8-seg`
  </Accordion>

  <Accordion title="Depth Estimation">
    * `midas-v2`
  </Accordion>

  <Accordion title="Pose Estimation">
    * `hrnet`
    * `lite-3dmm`
    * `posenet`
    * `hlandmark`
    * `mediapipe-pose-landmark`
  </Accordion>

  <Accordion title="Super Resolution">
    * `srnet`
  </Accordion>

  <Accordion title="Audio Classification">
    * `wave2vec`
    * `yamnet`
  </Accordion>

  <Accordion title="Tensor Generation">
    * `tensor`
  </Accordion>
</AccordionGroup>

The supported models for all the above categories can be found in the [Supported Models](../supportedmodels/) section.

You can also list all available modules on device by running:

```bash theme={null}
gst-inspect-1.0 qtimlpostprocess | grep -E "^\s+\([0-9]+\)"
```

If a suitable pre-integrated module is not available for a particular model, developers can [create custom modules and load them into the pipeline](../plugin-reference/qtimlpostprocess#post-processing-module). These custom modules encapsulate model-specific parsing logic and can be adapted to handle unique tensor formats or specialized post-processing requirements.

The module interface is implemented in pure C++ without dependencies on GStreamer or GLib, resulting in a lightweight and easily integrable solution. To develop a custom module, only the interface header file supplied by the QIM SDK and an ARM64-compatible compiler are required. Please refer to the [custom postprocessing plugin page for more details.](../plugin-reference/qtimlpostprocess#guidelines-for-writing-a-custom-post-processing-module)

### Plugin Configuration Options

The [qtimlpostprocess](../plugin-reference/qtimlpostprocess) plugin offers several configuration options to control how model outputs are interpreted and prepared for downstream use:

* **Module:** Specifies the post-processing module to load. Each module contains parsing logic for a particular model class (e.g., classification, object detection, segmentation). The actual interpretation of the output tensor occurs within the selected module.
* **Labels File:** Path to a file containing class labels. Supported formats include:

  * Newline-separated labels (commonly used in the ML community)
  * JSON format (supports additional metadata such as display color and class filtering)

  The QIM SDK includes a built-in parser capable of auto-detecting these formats.
* **Settings:** A JSON object containing module-specific configuration parameters. These settings vary by model type. An example could be `confidence_threshold` which is applicable to classification and detection models.
* **Results:** Specifies the maximum number of results to return. This option is useful for limiting the number of top predictions in classification scenarios and ensuring compatibility with downstream plugins that may only support a fixed number of results.

The diagram below shows the AI pipeline used in this example — face detection with bounding box overlay:

<img alt="Image of AI pipeline using face detection" />

The decoded video frames pass through [`qtimlvconverter`](../plugin-reference/qtimlvconverter) (preprocessing) and [`qtimltflite`](../plugin-reference/qtimltflite) (inference), then into [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess), which parses the output tensors and produces GStreamer ML metadata containing bounding boxes and confidence scores. The [`qtivoverlay`](../plugin-reference/qtivoverlay) plugin then renders those results directly onto the video frames for display.

## Run example on device

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | ResNeXt101 W8A8 model | [Qualcomm AI Hub — ResNeXt101](https://aihub.qualcomm.com/iot/models/resnext101) | `resnext101-w8a8.tflite` |
    | Classification labels | <a href="../labels/imagenet.txt">imagenet.txt</a> | `imagenet.txt` |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    Create the required directories and transfer the downloaded files to your device.

    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>
      ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
      scp resnext101-w8a8.tflite  <user>@<device-ip>:~/models/
      scp imagenet.txt            <user>@<device-ip>:~/labels/
      scp ai_demo_sample.mp4                <user>@<device-ip>:~/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    ```bash theme={null}
    export MODEL_NAME=resnext101-w8a8.tflite
    export LABELS_NAME=imagenet.txt
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    export VIDEO_SOURCE="filesrc location=$HOME/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12"
    ```
  </Step>

  <Step title="Run example on device">
    <Tabs>
      <Tab title="GStreamer Command line">
        ```bash theme={null}
        gst-launch-1.0 -e $VIDEO_SOURCE ! queue ! \
          qtimlvconverter name=preprocess ! queue ! \
          qtimltflite name=inference \
            delegate=external \
            external-delegate-path=libQnnTFLiteDelegate.so \
            external-delegate-options="QNNExternalDelegate,backend_type=htp;" \
            model=$HOME/models/$MODEL_NAME ! queue ! \
          qtimlpostprocess name=postprocess \
            results=1 \
            module=mobilenet-softmax \
            labels=$HOME/labels/$LABELS_NAME \
            settings="{\"confidence\": 51.0}" ! \
          text/x-raw ! queue ! \
          qtimlmetaparser module=json ! queue ! \
          filesink location=$HOME/media/result.json sync=false
        ```
      </Tab>

      <Tab title="GStreamer native application">
        * **Application source code:** [gst-ai-video-postprocess](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-video-postprocess)

        * **Build your application:**

          <Tabs>
            <Tab title="Yocto">
              <a href="../advanced/yocto-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>

            <Tab title="Ubuntu">
              <a href="../advanced/ubuntu-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>
          </Tabs>

        * **Run:**

          ```bash theme={null}
          gst-ai-video-postprocess -s "$VIDEO_SOURCE"
          ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

## External postprocessing

The pre-integrated modules cover the most common model families. However, when you have a model whose output tensors are not handled by any existing module — a custom architecture, an experimental head, or a proprietary decoding scheme — you can parse the raw tensors directly in your application instead of writing a loadable C++ module. [qtimlpostprocess](../plugin-reference/qtimlpostprocess) exposes a set of **`process-*` signals** that hand the inference output tensors to a callback in your application, which then produces the ML results (detections, classifications, poses, segmentations, depth maps) or writes new output tensors.

Unlike a loadable module, this path keeps the parsing logic inside your own application process — convenient for prototyping, for logic that depends on application state, or when you prefer Python over a compiled module. It is engaged by leaving the `module` property **unset** and connecting a handler to the `process-*` signal that matches your model.

<Info>The full list of signals, the result-structure field reference, and complete C/C++ and Python examples are documented in [Guidelines for Writing an External Post-Processing Callback](../plugin-reference/qtimlpostprocess#guidelines-for-writing-an-external-post-processing-callback).</Info>


# Preprocessing
Source: https://imsdkdocs.qualcomm.com/qimsdk-overview/preprocessing

Converting captured frames into tensors for inferencing

# Converting video frames to tensors

Video sources frequently deliver frames in diverse formats, resolutions, and color spaces that may not align with the requirements of machine learning models. Since models are typically trained on data with fixed resolution, format, and normalization parameters, it is essential to preprocess incoming frames to ensure compatibility with the model's expectations.

Common preprocessing steps include:

* **Cropping**: Select a specific region of the input frame to focus on. This uses provided ROI metadata to determine the crop region for each frame.
* **Rescaling**: Adjusts the spatial dimensions of input frames to match the expected tensor size for the target model to ensure compatibility and consistent performance.
* **Format Conversion**: Translates pixel data between supported formats (e.g., YUV to RGB) to meet the input requirements of the target model.
* **Batching**: Aggregates multiple frames or images into batches to optimize inference throughput and leverage parallel processing capabilities.
* **Normalization**: Applies pixel value scaling and normalization techniques (such as mean subtraction and standard deviation division) to standardize input data for improved model accuracy.

To perform efficient transformation the [qtimlvconverter](../plugin-reference/qtimlvconverter) runs on GPU and executes these steps in a chain. It resizes the entire input frame or the cropped region to fit the target tensor dimensions while preserving the full field of view. In this step, only the resolution changes, while the format and data range remain the same. By default it maintains the original aspect ratio, ensuring that geometric shapes of the objects remain undistorted in the output. If the input frame's aspect ratio differs from that of the target tensor, the resized frame is positioned in the top-left corner, and any remaining area is filled with a black background. This method ensures that the entire tensor is populated, preserves the integrity of the original image content, and avoids cropping or distortion.  This makes it well-suited for models that require consistent spatial representation.

<img alt="Preprocessing example 1" />

### Displacement Modes

[qtimlvconverter](../plugin-reference/qtimlvconverter) support three displacment modes:

* **top-left** (default) - Keeps the original aspect ratio of the image and places it in the **top-left corner** of the output tensor.
* **centre** - Also preserves the aspect ratio, but **centers the image within the tensor**. This is a common choice for models that expect the main object to be in the middle.
* **stretch** - Ignores the original aspect ratio and **stretches** the image to completely fill the tensor. This can introduce distortion but ensures full coverage, which some models require.
* **centre-crop** - Ignore the source image AR (Aspect Ratio) and if required crop the source around its center to fit completely inside the output tensor.
  <img alt="Preprocessing example 2" />

### Supported Tensor Layouts

At the final stage of preprocessing, the output tensor must be arranged in a specific layout format. The [qtimlvconverter](../plugin-reference/qtimlvconverter) element supports the following tensor dimension layouts:

* **NHWC** — most commonly used by the **QIM SDK** for video AI models
* **NCHW** - contains the same data as **NHWC** but organized differently
* **NDHWC** - represent sequences of video frames, where each tensor includes multiple consecutive frames processed together for temporal analysis
* **NDCHW** - like **NDHWC**, but the channels within each frame are stored in planar order
* **HWC** / **CHW** - the single-image (no batch dimension) equivalents of **NHWC** / **NCHW**

The layout is detected automatically from the number of dimensions and their sizes, so no manual configuration is required.

Each letter in these layouts represents a specific dimension:

* **N** — Batch size (number of frames or images on which inference is performed)
* **D** — Depth (e.g., number of consecutive frames in a video stream)
* **H** — Height
* **W** — Width
* **C** — Channels (e.g., 3 for RGB/BGR, 1 for grayscale)

These layouts define how data is organized in memory and are crucial for ensuring compatibility with machine learning models.

**NCHW** and **NHWC** formats contain the same data but organize it differently. Let's take the **RGB color format** as an example:

* In **NHWC** layout, the **Red, Green, and Blue** values for each pixel are stored **sequentially** in the buffer — one pixel at a time.
* In **NCHW** layout, all **Red** pixel values are stored first, followed by all **Green** values, and then all **Blue** values.

This difference in memory arrangement can affect performance and compatibility depending on the model architecture and hardware.

<img alt="Tensor layout example 1" />

**Supported Tensors by** [qtimlvconverter](../plugin-reference/qtimlvconverter):

| | |
| - | - |
| Tensor Shape | HWC, CHW, NHWC, NCHW, NDHWC, NDCHW |
| Data Format | uint8, int8, uint16, int16, uint32, int32, float32, float16 |
| Data Range | any |
| Color Format | RGB, BGR, Grayscale |

<Note>If the output tensor shape does not match any supported layout, the layout is treated as **unknown**. Such tensors can still be produced, but only through [external preprocessing](#external-preprocessing) — the application connects the `process` signal and fills the tensor itself.</Note>

<Info>For more technical details and advanced features, please refer to the [qtimlvconverter documentation](../plugin-reference/qtimlvconverter).</Info>

# Preprocessing Sample

This example demonstrates how [qtimlvconverter](../plugin-reference/qtimlvconverter) can be used in Python to convert video frames into tensors.

<img alt="Preprocessing Pipelines" />

Here's how each component works:

* The video source supplies frames in NV12 (YUV) format at a resolution of 1920×1080 (1080p).
* The [qtimlvconverter](../plugin-reference/qtimlvconverter) element is configured to generate a quantized tensor of type uint8 with shape \[1, 224, 224, 3], corresponding to an RGB888 image. By default, [qtimlvconverter](../plugin-reference/qtimlvconverter) preserves the aspect ratio of the input frame. In this scenario, the original 16:9 frame is downscaled to 224×126 pixels to fit within the target tensor dimensions, ensuring that the field of view remains intact and free from cropping or distortion. The frame is then converted from NV12 to RGB888 format. As the model in this example will be quantized to accept uint8 data, no further normalization is necessary; the RGB888 format directly matches the model's input requirements. The resulting RGB888 frame, with a resolution of 224×126, is positioned in the top-left corner of the output tensor. The remaining area—unused rows beneath the image—is filled with black pixels, ensuring the tensor is fully populated while maintaining the original aspect ratio.
* To access the tensor within a python application, the **appsink** element is utilized.
* In this example, the Python script receives tensors for further processing. However, for demonstration purposes, we simply store the tensor as a file in the file system. The script does not save each tensor to a separate file—instead, it overwrites the same file repeatedly to avoid memory flooding and excessive disk usage.

### Run example on device

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/media"
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    ```bash theme={null}
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    export VIDEO_SOURCE="filesrc location=$HOME/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12"
    ```
  </Step>

  <Step title="Build and run application">
    <Tabs>
      <Tab title="GStreamer Command line">
        ```bash theme={null}
        gst-launch-1.0 -e $VIDEO_SOURCE ! queue ! \
          qtimlvconverter name=preprocess ! \
          "neural-network/tensors,type=INT8,dimensions=<<1,224,224,3>>" ! queue ! \
          filesink location=$HOME/media/tensor.bin sync=false
        ```
      </Tab>

      <Tab title="GStreamer native application">
        * **Application source code:** [gst-ai-video-preprocess](https://github.com/qualcomm/gst-plugins-imsdk/tree/main/gst-sample-apps/gst-ai-video-preprocess)

        * **Build your application:**

          <Tabs>
            <Tab title="Yocto">
              <a href="../advanced/yocto-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>

            <Tab title="Ubuntu">
              <a href="../advanced/ubuntu-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>
          </Tabs>

        * **Run:**

          ```bash theme={null}
          gst-ai-video-preprocess -s "$VIDEO_SOURCE"
          ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

### Expected output

During execution, timing information is displayed in the console to provide insight into processing performance. The Python script saves the output tensor to the file `$HOME/media/tensor.bin`.

Given that the tensor has a shape of \[1, 224, 224, 3] and a data type of uint8, it effectively represents an RGB888 image. As a result, the tensor can be opened and viewed using any image viewer that supports the RGB888 format.

# AI preprocessing with custom crop

In some cases, instead of resizing or repositioning the entire image within the input tensor, you may need to crop a specific region and feed only that portion to the model. This approach is useful when we are interested in analyzing only a specific part of the field of view (FOV), rather than the entire image.

While the **image-disposition** property of [qtimlvconverter](../plugin-reference/qtimlvconverter) helps with placement and aspect ratio handling, cropping is a separate preprocessing step that gives you more control over which part of the image the model processes.

To perform cropping, you can insert a [qtivtransform](../plugin-reference/qtivtransform) step before [qtimlvconverter](../plugin-reference/qtimlvconverter). This element allows you to crop the input image before it reaches the converter, giving you **precise control** over the region of interest used for inference.

<img alt="Preprocessing Pipelines" />

Here's how each component works:

* **source** - this is your input stream.
* [qtivtransform](../plugin-reference/qtivtransform) - this stage allows you to apply transformations to the image, such as cropping, resizing, or rotating. In this context, you can define a crop region via properties passed directly to [qtivtransform](../plugin-reference/qtivtransform), enabling you to extract a specific part of the input image before it reaches the model.
* [qtimlvconverter](../plugin-reference/qtimlvconverter) - After cropping, this element prepares the image for inference. It handles scaling, color convert, normalization and positioning based on the image-disposition property (e.g., top-left, centre, stretch).
* **appsink** - This is the final stage where the tensor is passed to the application
  <Warning>The output resolution of [qtivtransform](../plugin-reference/qtivtransform) must be specified manually. To avoid stretching or distortion, **the aspect ratio of the output resolution must match the aspect ratio of the crop window**. If you're unsure what resolution to choose, you can simply set the **crop width and height as the output resolution** — this ensures a 1:1 mapping without scaling.</Warning>

By combining input cropping via [qtivtransform](../plugin-reference/qtivtransform) with placement control via image-disposition in [qtimlvconverter](../plugin-reference/qtimlvconverter), you can achieve any desired transformation—whether it's focusing on a specific region, preserving aspect ratio, or aligning the image within the tensor. This flexibility is especially valuable when adapting to different model requirements or optimizing inference performance.

The image below demonstrate how specific region in input image could be converted into a tensor:

<img alt="Preprocessing example with crop" />

The next example demonstrates how to create a tensor with a custom crop<br />

### Run example on device

<Steps>
  <Step title="Download Required Files">
    | File | Download | Save as |
    | - | - | - |
    | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

    <Note>
      If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
      `unzip filename.zip`
    </Note>
  </Step>

  <Step title="Copy files to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
      # Run from your host machine — replace <user> and <device-ip>

      ssh <user>@<device-ip> "mkdir -p ~/media"
      scp ai_demo_sample.mp4   <user>@<device-ip>:~/media/
      ```
    </CodeGroup>
  </Step>

  <Step title="Connect to device">
    <CodeGroup>
      ```bash SCP (SSH) theme={null}
      ssh <user>@<device-ip>
      ```
    </CodeGroup>
  </Step>

  <Step title="Set environment variables">
    ```bash theme={null}
    export SRC_VIDEO_NAME=ai_demo_sample.mp4
    export VIDEO_SOURCE="filesrc location=$HOME/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12"
    ```
  </Step>

  <Step title="Build and run application">
    <Tabs>
      <Tab title="GStreamer Command line">
        ```bash theme={null}
        gst-launch-1.0 -e $VIDEO_SOURCE ! queue ! \
          qtimlvconverter name=preprocess ! \
          "neural-network/tensors,type=INT8,dimensions=<<1,224,224,3>>" ! queue ! \
          filesink location=$HOME/media/tensor.bin
        ```
      </Tab>

      <Tab title="GStreamer C/C++ application">
        * **Application source code:** [gst-ai-video-preprocess-crop](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-video-preprocess-crop)

        * **Build your application:**

          <Tabs>
            <Tab title="Yocto">
              <a href="../advanced/yocto-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>

            <Tab title="Ubuntu">
              <a href="../advanced/ubuntu-build#steps-to-build-custom-application">
                Steps to build custom application
              </a>
            </Tab>
          </Tabs>

        * **Run:**
          ```bash theme={null}
          gst-ai-video-preprocess-crop -s "$VIDEO_SOURCE"
          ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

### Expected output

This example showcases how to convert a predefined region. During execution, timing information is displayed in the console to provide insight into processing performance. The Python script saves the output tensor to the file `$HOME/media/tensor.bin`. Given that the tensor has a shape of \[1, 224, 224, 3] and a data type of uint8, it effectively represents an RGB888 image. As a result, the tensor can be opened and viewed using any image viewer that supports the RGB888 format.

While [qtivtransform](../plugin-reference/qtivtransform) supports runtime crop window updates, this solution is not always scalable. In more advanced use cases, [qtimlvconverter](../plugin-reference/qtimlvconverter) can perform cropping on its own, without relying on [qtivtransform](../plugin-reference/qtivtransform). For instance, if you have two models working sequentially (ex. a detection model followed by a pose estimation model) [qtimlvconverter](../plugin-reference/qtimlvconverter) can automatically crop and generate a tensor for each bounding box detected by the first stage.

<Info>For more technical details and advanced features, please refer to the [qtimlvconverter documentation](../plugin-reference/qtimlvconverter).</Info>
Having established the process for converting video frames to tensors, the next step involves executing Inference on these tensors.

# External preprocessing

The built-in engines of [qtimlvconverter](../plugin-reference/qtimlvconverter) cover the most common preprocessing needs. However, some models require a transformation that the built-in engines do not implement — for example a custom color space conversion, a bespoke normalization curve, a non-typical arrangement of the pixels, or a proprietary algorithm. For these cases, [qtimlvconverter](../plugin-reference/qtimlvconverter) can hand the work over to your application through the **`process`** signal, letting you populate the output tensor yourself while still benefiting from the element's negotiation, buffer pooling, and batching. This works identically from both C/C++ and Python applications.

It is engaged by setting the `engine` property to `none` and connecting a handler to the `process` signal.

<Info>The signal layout, the input/output structure reference, and complete C/C++ and Python examples are documented in [Guidelines for Writing an External Preprocessing Callback](../plugin-reference/qtimlvconverter#guidelines-for-writing-an-external-preprocessing-callback).</Info>


# QIM SDK Overview
Source: https://imsdkdocs.qualcomm.com/qimsdk-overview/sdkoverview

Step by step guide for building AI pipelines using QIM SDK

<Note>
  **Before you begin:** QIM SDK must be installed — see the [Installation Guide](../installation).
</Note>

# The 5 basic stages of vision AI pipelines

The diagram below shows the five stages of a typical vision AI pipeline. <br />
⚡ Highlighted steps are hardware-accelerated by QIM SDK.

```mermaid theme={null}
flowchart LR
    classDef hw fill:#31017D,color:#ffffff,stroke:#1a3aad,font-weight:bold,min-width:160px,padding:12px
    A["Step 1\nData Source"] --> B["⚡ Step 2\nAI Preprocessing"] --> C["⚡ Step 3\nAI Inference"] --> D["⚡ Step 4\nPost-Processing"] --> E["Step 5\nUse AI Metadata"]
    class A,B,C,D,E hw
```

<Steps>
  <Step title="Data Source">
    Capture frames from the selected media source, ensuring compatibility and efficient data acquisition.
  </Step>

  <Step title="AI Preprocessing">
    Prepare raw frames for the AI model by resizing, reformatting, and normalizing pixel data into the tensor format the model expects. QIM SDK accelerates this step with the [qtimlvconverter](../plugin-reference/qtimlvconverter) plugin, which performs color space conversion and tensor layout transformations directly on the GPU — eliminating CPU bottlenecks before inference.
  </Step>

  <Step title="AI Inference">
    Run the prepared tensors through the AI model to generate predictions. QIM SDK accelerates inference with the [qtimltflite](../plugin-reference/qtimltflite) and [qtimlqnn](../plugin-reference/qtimlqnn) plugins, which delegate computation to the on-device NPU or HTP — delivering low-latency, high-throughput inference without taxing the CPU.
  </Step>

  <Step title="Post-Processing">
    Decode the model's output tensors into actionable results — such as bounding boxes, class labels, confidence scores, or segmentation masks. QIM SDK's [qtimlpostprocess](../plugin-reference/qtimlpostprocess) plugin interprets model outputs and formats them as metadata, which can then be rendered onto video frames using the GPU-accelerated [qtivoverlay](../plugin-reference/qtivoverlay) plugin.
  </Step>

  <Step title="Using the AI Metadata">
    Use the structured AI metadata output to drive downstream workflows. Common applications include:

    * **Overlay on video** — draw bounding boxes, labels, or masks directly on frames for real-time visualization
    * **Stream enrichment** — embed metadata into RTSP or WebRTC streams for remote monitoring
    * **Cloud/edge messaging** — publish to MQTT, Kafka, or other backends for storage, alerting, or further analysis
    * **Automated actions** — trigger events or control external systems based on detection results
  </Step>
</Steps>


# Quickstart Guide
Source: https://imsdkdocs.qualcomm.com/quickstart

Explore object detection pipelines with QIM SDK

## Object detection

<video />

**Given a video frame, identify objects, and draw bounding boxes around them**.

<Note>
  Ensure you have QIM SDK installed. [QIM SDK Installation Guide](/installation)
</Note>

## Prerequisites

Run these commands on your device first:

<Accordion title="Set environment variables">
  ```bash theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/models $HOME/Downloads/qimsdk_samples/labels $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/media/output
  export MODEL_NAME=yolox_w8a8.tflite
  export LABELS_NAME=yolov8.json
  export SRC_VIDEO_NAME=ai_demo_sample.mp4
  ```
</Accordion>

<Accordion title="Download the label, model, and video file">
  ```bash theme={null}
  # Download YOLO-X W8A8 model
  curl -L -o $HOME/models/$MODEL_NAME \
    https://huggingface.co/Qualcomm/Yolo-X/resolve/v0.30.5/Yolo-X_w8a8.tflite

  # Download detection labels
  curl -L -o $HOME/labels/$LABELS_NAME \
    https://imsdkdocs.qualcomm.com/labels/yolov8.json

  # Download sample video
  curl -L -o $HOME/media/$SRC_VIDEO_NAME \
    https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4
  ```
</Accordion>

***

## Object detection pipeline command

<Note>
  Ensure you have followed the [prerequisites](/quickstart#prerequisites) before continuing
</Note>

<Steps>
  <Step title="Run the pipeline command">
    ```bash theme={null}
    gst-launch-1.0 -e --gst-debug=2 \
    filesrc location=$HOME/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
    tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
    t. ! queue ! qtimlvconverter ! queue ! \
    qtimltflite model=$HOME/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
    qtimlpostprocess module=yolov8 labels=$HOME/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! obj_mux.
    ```
  </Step>

  <Step title="View results">
    Your display shows the video feed with bounding boxes and class labels rendered over each detected object. The pipeline processes frames in real time <br /> Press `Ctrl+C` to stop the pipeline gracefully.

    * In case you want to build around this demo, command line might not be the most robust solution
    * You could paste it into a shell file...
    * But if you want other code to interact with this, you would want a cpp or python file (pipeline application).
  </Step>
</Steps>

***

## How it works

The pipeline reads an H.264 video file, hardware-decodes it, branches the decoded stream, runs YOLO-X inference on the Qualcomm® AI Engine (HTP backend), post-processes the bounding-box results, blends the annotations back onto the original frame using a hardware compositor, and displays the output to a screen.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

***

## Next Steps

You've run an object detection pipeline on Qualcomm® hardware. Here's where to go next:

<CardGroup>
  <Card title="AI Sample Pipelines" icon="book-open" href="/sample-pipelines/aipipelines">
    Ready-to-run GStreamer pipelines for classification, segmentation, pose estimation, super resolution, and more.
  </Card>

  <Card title="Blogs" icon="message" href="/blogs/blog-index">
    Real-world examples built by the QIM SDK community — covering object detection, PPE compliance, security cameras, and more.
  </Card>

  <Card title="Supported Models" icon="people-carry-box" href="/supportedmodels">
    Full catalogue of quantized TFLite models tested on Qualcomm® hardware, with pipeline commands for each.
  </Card>

  <Card title="Plugin Reference" icon="book" href="/plugin-reference">
    API-level documentation for every QIM SDK GStreamer plugin — properties, caps, and usage examples.
  </Card>
</CardGroup>


# Release Notes
Source: https://imsdkdocs.qualcomm.com/relnotes

What's new for QIM SDK release

<ReleaseSelector>
  <div>
    ## QIM SDK Release Notes

    ### Build AI and Multimedia Applications Faster with QIM SDK

    <div>
      <p>QIM SDK simplifies the journey from <strong>idea to production</strong>—without requiring developers to be GStreamer experts.</p>

      <p>This release introduces a more consistent and accessible development experience across the application lifecycle. Whether you are building your first AI application, developing a high-performance C++ solution, or designing an advanced multi-stream AI pipeline, QIM SDK provides the tools and flexibility needed to move faster.</p>
    </div>

    ### Release Highlights

    <table>
      <thead>
        <tr>
          <th>Build</th>
          <th>Accelerate</th>
          <th>Customize</th>
          <th>Deploy</th>
          <th>Expand</th>
        </tr>
      </thead>

      <tbody>
        <tr>
          <td>Python and C++ App Builders</td>
          <td>AI Coding Skills</td>
          <td>AI Pre/Post-Processing Hooks</td>
          <td>Containerized Docker Delivery</td>
          <td>Expanded ML Pipeline Capabilities</td>
        </tr>
      </tbody>
    </table>

    ...and other improvements across ML Bin capabilities, video analytics, and core pipeline plugins, covered below. Together, these enhancements help developers focus on application logic rather than the complexities of pipeline construction and deployment—while retaining control over **pipeline behavior, performance, and deployment**.

    > **From Python prototypes to production-ready C++ applications, QIM SDK delivers a simpler, faster, and more consistent approach to AI and multimedia development.**

    ***

    ## New Features

    ### Python App Builder

    A brand-new Python SDK enables developers to build full-featured AI and multimedia applications using familiar Python workflows. It follows the same core concepts as the C++ App Builder, combining Python's productivity with native-class performance so developers can focus on business and AI logic while production-grade multimedia pipelines run underneath.

    #### Designed for Python Developers

    * YAML-driven pipeline creation.
    * Intuitive Python APIs.
    * Zero-copy tensor access to AI model inputs and outputs.
    * Built-in ML Bin wrappers.
    * Seamless integration with popular Python libraries.
    * Faster prototyping and iteration cycles.

    See the full [Python App Builder guide](app-builder/python-quickstart) for the complete API reference and walkthroughs.

    ***

    ### C++ App Builder

    A brand-new, object-oriented C++ SDK, alongside GStreamer, for building high-performance AI and multimedia applications. It gives C++ developers a clean, strongly-typed pipeline API with zero-copy tensor access to AI model inputs and outputs.

    #### Designed for C++ Developers

    * Built-in ML Bin wrappers.
    * Native APIs for custom AI pre-processing and post-processing.
    * YAML-driven pipeline creation, simplifying deployment, portability, and configuration management.

    > Whether you build in Python or C++, the QIM SDK abstracts away GStreamer's complexity — no hand-written caps required. Both App Builders offer consistent high-level abstractions for pipelines, elements, sources, and sinks, simplified lifecycle management, stream filtering and routing, direct access to AI and multimedia components, and structured logging and diagnostics.

    See the full [C++ App Builder guide](app-builder/cpp-quickstart) for the complete API reference and walkthroughs.

    ***

    ### Customize AI Pre/Post-Processing Without Custom Plugins

    Implement model-specific AI logic without creating custom plugins. Previously, customizing AI pre-processing or post-processing meant writing, building, and loading a custom GStreamer plugin. Now, that same logic can live directly in your application code — as callbacks in C/C++ or Python, whether you're using raw GStreamer or either App Builder — using signals exposed by the ML pre/post-processing plugins, with no plugin to build or load.

    #### Benefits

    * Callback access to tensors, metadata, frame geometry, and inference outputs — including detections, classifications, poses, segmentation maps, and depth maps.
    * Faster model integration.
    * Reduced development complexity.
    * Freedom to implement proprietary AI algorithms without exposing them in a shared plugin.
    * Full compatibility with QIM SDK's visualization, metadata, tracking, and analytics components — downstream plugins consume the results exactly as if a built-in module had produced them.
    * No modifications required to QIM SDK internals.
    * Backward compatibility with existing applications.

    ***

    ### QIM SDK Docker

    QIM SDK in a Docker container means the full Qualcomm multimedia and AI software stack is delivered as containerized build and deploy images instead of being manually installed directly on a host or target system — reproducible and secure by default. In practice, a build container compiles and assembles all required components, including cross-compiled artifacts for Qualcomm ARM64 targets. The build then produces a deploy container that contains only the runtime pieces needed on the device.

    #### Key Benefits

    * **Reproducible, versioned builds:** Consistent toolchains and package versions across developers and CI, with predictable, rollback-friendly deployments.
    * **Build and runtime separation:** A self-contained build image produces a lean deploy image with only runtime essentials — no eSDK or host platform content required.
    * **Secure-by-default runtime:** Runs as a non-root user, with explicit CDI-based hardware passthrough for GPU, camera, and media devices.
    * **Broad platform support:** Debian Trixie base with GStreamer 1.26, validated across QLI 2.x, Ubuntu Server/Desktop, and QCS615, QCS6490, IQ8, and IQ9 targets.

    See the full [QIM SDK Docker guide](advanced/docker) for build and deployment instructions.

    ***

    ### Build Applications Using AI Coding Agents

    QIM SDK now includes AI Coding Skills that give coding agents — such as Claude Code, Cursor, Codex, and other AI development assistants — the QIM SDK-specific knowledge to generate complete Python, C++, or GStreamer applications directly from natural-language pipeline descriptions.

    #### Benefits

    * Accelerate prototyping and experimentation.
    * Automatically generate applications across:
      * Python App Builder
      * C++ App Builder
      * GStreamer pipelines

    See the full [AI Coding Agent guide](app-builder-coding-agent) for setup and usage details.

    ***

    ## Expanded ML Bin Capabilities

    ML Bins continue to simplify AI deployment by combining pre-processing, inference, and post-processing into a single pipeline component.

    | ML Bin | Runtime |
    | - | - |
    | `qtimlvideotflitebin` | LiteRT / TensorFlow Lite |
    | `qtimlvideoqnnbin` | Qualcomm QNN |
    | `qtimlvideosnpebin` | SNPE |

    ### New Capabilities

    * Segmentation and depth estimation metadata support.
    * End-to-end metadata propagation from inference through post-processing and across the pipeline.
    * Metadata muxing support.
    * Visualization overlay integration.

    ### Enhanced AI Pipeline Support

    * Simplified deployment with consistent AI processing and metadata handling across all three ML Bin runtimes — TensorFlow Lite, QNN, and SNPE.
    * Works the same way whether you're building with the Python/C++ App Builders or customizing AI workflows directly in application code.

    ***

    ## Video Analytics Enhancements

    ### Auto-Framing Support

    New intelligent auto-framing capabilities automatically adjust camera views to keep subjects optimally framed.

    #### Features

    * Built-in stabilization.
    * Multiple movement strategies.
    * Pluggable algorithm framework.
    * Advanced filtering and smoothing controls.

    ***

    ### Object ROI Merge

    A new Object ROI Merge capability consolidates overlapping person detections into a unified region of interest.

    #### Benefits

    * Reduced duplicate detections.
    * Improved tracking consistency.
    * Cleaner metadata generation.
    * Better accuracy for multi-stage AI pipelines.

    ***

    ## Core Enhancements / Improvements

    This release also includes several targeted improvements to the ML pre/post-processing and compositing plugins:

    * **qtimlvconverter — additional tensor layouts:** Added support for the **HWC** and **CHW** (single-image, no batch dimension) and **NDCHW** (planar temporal) tensor layouts, alongside the existing NHWC, NCHW, and NDHWC. The layout is detected automatically from the tensor shape. See [Preprocessing](qimsdk-overview/preprocessing) and the [Image/Video Tensor Characteristics](plugin-reference/qtimlvconverter#image/video-tensor-characteristics).
    * **External pre/post-processing signals:** A new `process` signal on [qtimlvconverter](plugin-reference/qtimlvconverter) and `process-*` signals on [qtimlpostprocess](plugin-reference/qtimlpostprocess) let applications handle frame-to-tensor conversion and inference output parsing directly in C/C++ or Python, without writing a loadable module — and `process` is the only way to convert tensors whose shape matches none of the supported layouts. See [External preprocessing](qimsdk-overview/preprocessing#external-preprocessing) and [External postprocessing](qimsdk-overview/postprocessing#external-postprocessing).
    * **qtimlpostprocess — automatic image mask resolution negotiation:** [qtimlvconverter](plugin-reference/qtimlvconverter) now propagates the model input tensor resolution downstream via a custom event, which [qtimlpostprocess](plugin-reference/qtimlpostprocess) uses to automatically negotiate the image mask resolution. Hardcoded `format`/`width`/`height` caps filters after `qtimlpostprocess` are no longer required in most pipelines — only `RGBA`/`RGBx` need to be specified if a fixed format is desired.
    * **ROI-aware image mask output:** In image mask (video) output mode, `qtimlpostprocess` now attaches an `ImageRegion` ROI meta reflecting the valid region based on the input source aspect ratio (populating only the pixels that contain real data), and [qtivcomposer](plugin-reference/qtivcomposer) automatically uses that ROI as the blit source instead of the full frame.

    ***

    ## Comprehensive Developer Documentation

    QIM SDK's documentation experience has been completely redesigned to support developers at every stage of their journey — from **Getting Started guides** and **AI pipeline tutorials and solution blueprints** to **30+ reference applications**, with full GitHub integration from concepts through to source code.

    ### Documentation Highlights

    * Production and commercial deployment guidance.
    * Comprehensive API documentation for GStreamer, Python, and C++.
    * Detailed plugin reference guides.
    * Multi-stream design patterns.
    * Multi-stage and daisy-chain AI blueprints.
    * AI Coding Skills integration guides.
    * End-to-end blogs and tutorials.

    ### Fully Integrated with GitHub

    Each guide links directly into the QIM SDK source repositories, so developers can jump from a concept or API reference straight to the matching implementation example without leaving the docs — moving quickly from learning to prototyping to production deployment in one unified experience.

    ***

    ## Why This Release Matters

    This release transforms QIM SDK from a GStreamer-centric toolkit into a developer-first application platform for AI and multimedia workloads. Developers can now:

    * Build applications in Python, C++, or GStreamer.
    * Generate applications using AI Coding Agents.
    * Customize AI workflows through application code.
    * Deploy advanced AI pipelines with minimal boilerplate.
    * Ship AI and multimedia applications via reproducible, versioned Docker containers.
    * Access comprehensive documentation, blueprints, and reference applications to accelerate development.

    The end result: faster development, lower learning curve, and greater flexibility for building production-grade AI and multimedia solutions on Qualcomm platforms.
  </div>
</ReleaseSelector>


# AI
Source: https://imsdkdocs.qualcomm.com/sample-application/ai-sample-applications

Ready-to-run C++ applications demonstrating QIM SDK AI inference capabilities

This section covers QIM SDK AI sample applications that demonstrate vision, audio, and multi-model inference on Qualcomm platforms using LiteRT models accelerated on Qualcomm AI hardware.

The following tables list all available GStreamer C/C++ AI applications and their platform support. Select the appropriate configuration tab for your setup.

<Tabs>
  <Tab title="Config #1">
    | Application | Source code | Description | Input sources | Runtime | QCS6490 / IQ-8275 / IQ-9075 | IQ-X7181/IQ-X5121 | IQ-615 / IQ-2390 |
    | - | - | - | - | - | :-: | :-: | :-: |
    | Image classification | [`gst-ai-classification`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-classification) | Classification on streams from a file source or RTSP. | Camera, file, RTSP, USB | cpu | ✓ | ✓ | ✓ |
    | Object detection | [`gst-ai-object-detection`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-object-detection) | Object detection on streams from a file source or RTSP. | Camera, file, RTSP, USB | cpu | ✓ | ✓ | × |
    | Pose detection | [`gst-ai-pose-detection`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-pose-detection) | Pose detection on streams from a file source or RTSP. | File, RTSP, USB | cpu | ✓ | ✓ | ✓ |
    | Image segmentation | [`gst-ai-segmentation`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-segmentation) | Image segmentation on streams from a file source or RTSP. | File, RTSP | cpu | ✓ | ✓ | × |
    | Daisy chain detection + classification | [`gst-ai-daisychain-detection-classification`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-daisychain-detection-classification) | Cascaded object detection and classification. | File, RTSP, USB | cpu | ✓ | ✓ | × |
    | Daisy chain detection + pose | [`gst-ai-daisychain-detection-pose`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-daisychain-detection-pose) | Cascaded object detection and pose detection. | File, RTSP, USB | cpu | ✓ | ✓ | × |
    | Monodepth | [`gst-ai-monodepth`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-monodepth) | Monocular depth estimation from file or RTSP. | File, RTSP | cpu | ✓ | ✓ | × |
    | Face detection | [`gst-ai-face-detection`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-face-detection) | Face detection from file or RTSP. | File, RTSP | cpu | ✓ | ✓ | ✓ |
    | Audio classification | [`gst-ai-audio-classification`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-audio-classification) | Audio event classification from microphone or file. | Audio, file | cpu | ✓ | ✓ | ✓ |
    | Metadata parsing | [`gst-ai-metadata-parser-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-metadata-parser-example) | Parse ML metadata and count people from file or RTSP. | File, RTSP | cpu | ✓ | ✓ | × |
    | AI USB camera | [`gst-ai-usb-camera-app`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-usb-camera-app) | USB camera streaming with optional object detection. | USB | cpu | ✓ | ✓ | × |
    | AI event encoder | [`gst-ai-event-encoder`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-event-encoder) | Encode video only when a person is detected. | File, RTSP | cpu | ✓ | ✓ | × |
  </Tab>

  <Tab title="Config #2">
    | Application | Source code | Description | Input sources | Runtime | QCS6490 / IQ-8275 / IQ-9075 | IQ-X7181/IQ-X5121 | IQ-615 / IQ-2390 |
    | - | - | - | - | - | :-: | :-: | :-: |
    | Image classification | [`gst-ai-classification`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-classification) | Classification on streams from a camera, file, or RTSP. | Camera, file, RTSP, USB | dsp, gpu, cpu | ✓ | ✓ | ✓ |
    | Object detection | [`gst-ai-object-detection`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-object-detection) | Object detection on streams from a camera, file, or RTSP. | Camera, file, RTSP, USB | dsp, gpu, cpu | ✓ | ✓ | × |
    | Pose detection | [`gst-ai-pose-detection`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-pose-detection) | Pose detection on streams from camera, file, or RTSP. | Camera, file, RTSP, USB | dsp, gpu, cpu | ✓ | ✓ | ✓ |
    | Image segmentation | [`gst-ai-segmentation`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-segmentation) | Image segmentation on streams from camera, file, or RTSP. | Camera, file, RTSP | dsp, gpu, cpu | ✓ | ✓ | × |
    | Parallel inferencing | [`gst-ai-parallel-inference`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-parallel-inference) | Run multiple AI models simultaneously on one stream. | Camera, file, RTSP | dsp | ✓ | ✓ | × |
    | Multi input/output object detection | [`gst-ai-multi-input-output-object-detection`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-multi-input-output-object-detection) | Object detection across multiple I/O stream combinations. | Camera, file, RTSP | dsp | ✓ | ✓ | × |
    | Daisy chain detection + classification | [`gst-ai-daisychain-detection-classification`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-daisychain-detection-classification) | Cascaded object detection and classification. | Camera, file, RTSP, USB | dsp, gpu, cpu | ✓ | ✓ | × |
    | Daisy chain detection + pose | [`gst-ai-daisychain-detection-pose`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-daisychain-detection-pose) | Cascaded object detection and pose detection. | Camera, file, RTSP, USB | dsp, gpu, cpu | ✓ | ✓ | × |
    | Monodepth | [`gst-ai-monodepth`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-monodepth) | Monocular depth estimation from camera, file, or RTSP. | Camera, file, RTSP | dsp, gpu, cpu | ✓ | ✓ | × |
    | Video super-resolution | [`gst-ai-superresolution`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-superresolution) | Upscale low-resolution video to high resolution. | File | dsp | ✓ | ✓ | × |
    | Multistream inference | [`gst-ai-multistream-inference`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-multistream-inference) | AI inference across multiple simultaneous streams. | Camera, file, RTSP | dsp | ✓ | ✓ | × |
    | Multistream batch inference | [`gst-ai-multistream-batch-inference`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-multistream-batch-inference) | Batched AI inference on up to 24 file streams. | File | dsp | ✓ | ✓ | × |
    | Face detection | [`gst-ai-face-detection`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-face-detection) | Face detection from camera, file, or RTSP. | Camera, file, RTSP | dsp, gpu, cpu | ✓ | ✓ | ✓ |
    | Face recognition | [`gst-ai-face-recognition`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-face-recognition) | Face recognition from camera or RTSP. | Camera, RTSP | dsp | ✓ | ✓ | ✓ |
    | Audio classification | [`gst-ai-audio-classification`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-audio-classification) | Audio event classification from microphone or file. | Audio, file | gpu, cpu | ✓ | ✓ | ✓ |
    | Metadata parsing | [`gst-ai-metadata-parser-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-metadata-parser-example) | Parse ML metadata and count people. | Camera, file, RTSP | dsp, gpu, cpu | ✓ | ✓ | × |
    | AI USB camera | [`gst-ai-usb-camera-app`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-usb-camera-app) | USB camera streaming with optional object detection. | USB | dsp, gpu, cpu | ✓ | ✓ | × |
    | Gesture Recognition | [`gst-gesture-recognition`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-gesture-recognition) | Hand Gesture Recognition. | Camera, file, RTSP, USB | gpu | ✓ | ✓ | ✓ |
    | Hardware benchmarking Application | [`QCS6490-Vision-AI-Demo`](https://github.com/Avnet/QCS6490-Vision-AI-Demo/tree/QLI_2.0) | A GUI application to monitor hardware utilization with sample applications | Camera and USB | dsp | ✓ | ✓ | × |
    | AI event encoder | [`gst-ai-event-encoder`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-ai-event-encoder) | Encode video only when a person is detected. | Camera, file, RTSP | dsp, gpu, cpu | ✓ | ✓ | × |
  </Tab>
</Tabs>

***

## Prerequisites

<Accordion title="Try me">
  <Note>
    Some of the steps in the pre-requisties will be removed from future releases
    once the necessary fixes are mainlined.
  </Note>

  <Steps>
    <Step title="Set up Wi-Fi">
      Connect to the Wireless Access Point (Wi-Fi Router):

      ```bash theme={null}
      nmcli dev wifi connect <WiFi-SSID> password <WiFi-password>
      ```

      Check the connection and device status:

      ```bash theme={null}
      nmcli -p device
      ```

      **Login to the target device**

      Locate the IP address of the device according to the type of network connection,
      using the device shell on the Linux host:

      ```bash theme={null}
      ip address show
      ```

      Use the IP address from the Linux host to establish an SSH connection to the device:

      ```bash theme={null}
      ssh root@<ip-address>
      ```

      Example:

      ```bash theme={null}
      ssh root@192.168.0.222
      ```

      Connect to the SSH shell using the following password:

      ```bash theme={null}
      oelinux123
      ```
    </Step>

    <Step title="Download Models and Artifacts">
      On the target device, obtain the `download_artifacts_2.x.sh` script, set executable
      permissions, and run it to download the model, media, and label files:

      ```bash theme={null}
      cd /tmp/
      curl -L -O https://raw.githubusercontent.com/qualcomm/sample-apps-for-qualcomm-linux/refs/heads/main/qualcomm-linux/scripts/download_artifacts_2.x.sh
      chmod +x download_artifacts_2.x.sh
      ./download_artifacts_2.x.sh
      ```

      <Note>
        On `Qualcomm Ubuntu`, run the above command with `sudo`.
      </Note>
    </Step>

    <Step title="Enable qticamsrc">
      In the terminal of the target device, run the following command to enable the
      `qticamsrc` on Config #2:

      ```bash theme={null}
      echo -n "camx" > /var/data
      efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -w -f /var/data
      efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -p
      sync
      reboot
      ```
    </Step>

    <Step title="Enable Audio and GPU Delegate">
      In the terminal of the target device, run the following command to enable audio:

      ```bash theme={null}
      systemctl stop pipewire wireplumber pipewire.socket pipewire-manager.socket
      chmod 777 /dev/dma_heap/system
      adsprpcd audiopd &
      systemctl start pipewire wireplumber
      wpctl status
      ```

      To set the default devices for sink and source, get the device numbers from
      `wpctl status` and run the following command:

      ```bash theme={null}
      wpctl set-default <device ID>
      ```

      In the terminal of the target device, run the following command to enable the
      GPU delegate and backend:

      ```bash theme={null}
      mount -o rw,remount /usr
      ```

      ```bash theme={null}
      export OCL_ICD_FILENAMES=/usr/lib/libOpenCL_adreno.so.1
      ```
    </Step>

    <Step title="(Optional) Download the YOLOv8 model">
      Use one of the following methods:

      <Tabs>
        <Tab title="Download with a script">
          <Steps>
            <Step title="Create a Qualcomm AI Hub account">
              Create a [Qualcomm AI Hub account](https://app.aihub.qualcomm.com/account/).
            </Step>

            <Step title="Copy your API key">
              Go to **Settings** in the upper right corner and copy your API key.
            </Step>

            <Step title="Run the export script on the host computer">
              ```shell theme={null}
              curl -L -O https://raw.githubusercontent.com/qualcomm/sample-apps-for-qualcomm-linux/refs/heads/main/qualcomm-linux/scripts/export_model.sh
              chmod +x export_model.sh
              ```

              Replace `<API_KEY>` with your API key:

              ```shell theme={null}
              ./export_model.sh --api-key=<API_KEY>
              ```

              The models are downloaded to the `export_assets` directory.
            </Step>

            <Step title="Copy the model to the `${HOME}/Downloads/qimsdk_samples/models/` directory on the device">
              ```shell theme={null}
              scp <working-directory>/export_assets/yolov8_det-tflite-w8a8/yolov8_det.tflite $USER@<IP_ADDRESS_OF_TARGET_DEVICE>:~/Downloads/qimsdk_samples/models/
              ```
            </Step>
          </Steps>
        </Tab>

        <Tab title="Export with AI Hub APIs">
          Use the [YOLOv8-Detection-Quantized](https://github.com/qualcomm/ai-hub-models/tree/v0.52.0/src/qai_hub_models/models/yolov8_det) model from AI Hub.

          Current releases use the following SDK versions:

          * `Qualcomm Linux (GA 2.1)`: `Qualcomm AI Runtime SDK v2.50.0.260828`

          For example, to export the YOLOv8 QNN model, run:

          ```shell theme={null}
          qai-hub-models export yolov8_det --quantize w8a8 --target-runtime=qnn_context_binary --device="Dragonwing RB3 Gen 2 Vision Kit" --compile-options="--qairt_version 2.50" --profile-options "--qairt_version 2.50"
          ```

          Update `--qairt_version` to match the version installed on your **target device**. To check the installed version:

          ```shell theme={null}
          qairt-net-run --version
          ```

          To list the QAIRT SDK versions supported by AI Hub for the export command, run this command on the **host device**:

          ```shell theme={null}
          qai-hub list-frameworks
          ```

          To list supported devices, run this command on the **host device**:

          ```shell theme={null}
          qai-hub list-devices
          ```

          For example, to export the YOLOv8 LiteRT model, run:

          ```shell theme={null}
          qai-hub-models export yolov8_det --quantize w8a8 --target-runtime=tflite --device="Dragonwing RB3 Gen 2 Vision Kit"
          ```
        </Tab>

        <Tab title="Generate a batch model">
          To change the batch size, update `<N>` in the following export command:

          ```shell theme={null}
          qai-hub-models export <Model_Name> --batch-size <N> --device="Dragonwing RB3 Gen 2 Vision Kit"
          ```

          For example, to export the YOLOv8 LiteRT model with batch size 4:

          ```shell theme={null}
          qai-hub-models export yolov8_det --quantize w8a8 --target-runtime=tflite --device="Dragonwing RB3 Gen 2 Vision Kit" --batch-size 4
          ```
        </Tab>
      </Tabs>
    </Step>
  </Steps>
</Accordion>

## AI Vision Applications

### Object Detection

The [**gst-ai-object-detection**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-object-detection/main.c) application allows you to detect objects within images and videos. The use cases show the execution of [**YOLOv5**](https://github.com/ultralytics/yolov5), [**YOLOv8**](https://github.com/ultralytics/ultralytics) and [**YOLOX**](https://github.com/Megvii-BaseDetection/YOLOX/blob/main/README.md) on Qualcomm AI HW accelerator.

The following figure shows the pipeline, which receives the input from a live camera feed, file, USB source, or an RTSP stream, preprocesses it, runs inferences on AI hardware. The results are either displayed on the screen, saved as an encoded MP4 file, or streamed over the RTSP server. For information about the plugins used in the pipeline flow, see [**Pipeline flow**](#object-detection-pipeline-flow)

<Note>
  The `onnx` runtime is not supported for this sample application in the `QLI GA 2.1` release.
</Note>

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-object-detection`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-object-detection/main.c)

<Accordion title="Try me">
  When the software image includes the **qticamsrc** plugin, the camera framework uses it by default. If absent, the framework switches to **libcamera** instead. Since Config #1 lacks support for **qticamsrc**, the system defaults to **libcamera**.

  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | IMX577 camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - | - |
  | Config #1 | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes |
  | Config #2 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |

  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | Qualcomm Neural Processing SDK | `yolox.dlc` | `yolox.json` |
  | LiteRT | `yolov8_det_w8a8.tflite` / `yolox_quantized.tflite` | `yolox.json` |
  | Qualcomm AI Engine Direct | `yolov8_det_quantized.bin` | `yolov8.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-object-detection --config-file=/etc/configs/config_detection.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config_detection.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-object-detection -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-object-detection --config-file=/path/to/config_detection.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config_detection.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config_detection.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config_detection.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config_detection.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config_detection.json "$XDG_CONFIG_HOME/qimsdk_samples/config_detection.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#object-detection-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-object-detection
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-object-detection application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#object-detection-config-json-description) for all fields.

  ```json theme={null}
  {
    "file-path": "<path-to-input-video-file>",
    "ml-framework": "<snpe or tflite or qnn framework>",
    "yolo-model-type": "<yolov8 or yolonas or yolov5 or yolox>",
    "model": "<path-to-model-file>",
    "labels": "<path-to-label-file>",
    "threshold": "<post processing threshold, integer value from 1 to 100>",
    "runtime": "<dsp, gpu, cpu runtime>",
    "output-type": "waylandsink or filesink or rtspsink",
    "snpe-tensors": "<model output tensor name>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `file-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |
  | `output-file` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |

  <Note>
    For USB camera input, set the `video-format`, `resolution`, and `framerate` parameters in the config file
    to match the camera capabilities, see [Configure USB camera](https://dragonwingdocs.qualcomm.com/System/Interfaces/usb#configure-usb-camera).
  </Note>

  <Note>
    The `snpe-tensors` field applies only to the SNPE runtime. To retrieve the output tensor names for a DLC model, open the model in [**Netron**](https://netron.app/).
  </Note>

  <Note>
    When using DLC models from the AI Hub, the `snpe-tensors` field is optional.
  </Note>

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        Camera source, LiteRT model, DSP runtime

        ```json theme={null}
        {
          "camera": 0,
          "ml-framework": "tflite",
          "yolo-model-type": "yolox",
          "model": "yolox_quantized.tflite",
          "labels": "yolox.json",
          "threshold": 40,
          "runtime": "dsp",
          "output-type": "waylandsink"
        }

        ```
      </Tab>

      <Tab title="Config #1">
        Camera source, LiteRT model, CPU runtime

        ```json theme={null}
        {
          "ml-framework": "tflite",
          "yolo-model-type": "yolox",
          "model": "yolox_quantized.tflite",
          "labels": "yolox.json",
          "threshold": 40,
          "runtime": "cpu",
          "output-type": "waylandsink"
        }
        ```

        <Note>
          Config #1 supports only LiteRT models and the CPU runtime.
        </Note>
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  Detected objects with bounding boxes and labels are overlaid on the video and displayed on the local display.

  <img alt="Output Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the object detection pipeline:

  | Plugin | Description |
  | :- | :- |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | • Captures the live stream from camera.<br /> • Uses `tee` to split the stream for inferencing. |
  | `filesrc` | • Captures the video stream using `filesrc`, followed by `qtdemux`, which demultiplexes the stream.<br />• Uses `tee` to split the stream for inferencing. |
  | `rtspsrc` | • Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction.<br />• Uses `tee` to split the stream for inferencing. |
  | `v4l2src` | • Captures the live stream from USB camera.<br />• Uses `tee` to split the stream for inferencing. |
  | `h264parse` | Parses the H.264 video bitstream. |
  | [`v4l2h264dec`](/plugin-reference/v4l2h264dec) | Hardware-decodes H.264 video to raw frames. |
  | [`qtimlvconverter`](/plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data:<br />   • Color conversion<br />   • Scaling (up or down)<br />   • Normalization<br />3. Converts the preprocessed video stream to a tensor stream.<br />The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimlsnpe`](/plugin-reference/qtimlsnpe)<br />[`qtimltflite`](/plugin-reference/qtimltflite)<br />[`qtimlqnn`](/plugin-reference/qtimlqnn) | 1. After the inference runtime receives the tensor stream on its sink pad, it runs inference using the provided model.<br />2. Produces a tensor stream with the inference results on its source pad. |
  | [`qtimlpostprocess`](/plugin-reference/qtimlpostprocess) | Handles inference results from any object detection model.<br />1. Applies a threshold to the chosen number of results.<br />2. Loads the YOLO (YOLOv5, YOLOv8, or YOLO-NAS) module.<br />3. Produces video frames with only bounding boxes that can be overlaid on objects.<br />4. Sends these processed frames to the sink pad of `qtivcomposer`. |
  | [`qtivcomposer`](/plugin-reference/qtivcomposer) | 1. Composes frames with contents from its sink pads.<br />2. Pushes the GStreamer buffers containing these composed frames to its source pad. |
  | [`waylandsink`](/plugin-reference/waylandsink) | 1. Submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |
  | `filesink` | Receives the video stream on sink pad and saves it as an H.264-encoded MP4 file. |
  | [`qtirtspbin`](../plugin-reference/qtirtspbin) | 1. Serves as a network sink.<br />2. Transmits UDP packets to the network. |

  #### Config JSON Field Description

  | Field | Values / Description | |
  | - | - | - |
  | `camera` | Camera selection:<br /><br />- `0` — Primary camera<br />- `1` — Secondary camera | |
  | `file-path` | Path to the input media file.<br /><br />The media file should be placed in `$HOME/Downloads/qimsdk_samples/media` or provided using an absolute file path. | |
  | `rtsp-ip-port` | RTSP input URL. | |
  | `enable-usb-camera` | Enables or disables the USB camera.<br /><br />Supported values: `TRUE` or `FALSE` | |
  | `yolo-model-type` | YOLO model version:<br /><br />- `yolov5`<br />- `yolov8`<br />- `yolox`<br />- `yolonas`<br /><br />Default: `yolox` | |
  | `ml-framework` | Supported ML frameworks:<br /><br />- `snpe`<br />- `tflite`<br />- `qnn`<br /><br />Default: `tflite` | |
  | `model` | Path to the model file.<br /><br />Default model file:<br />`yolox_quantized.tflite`<br /><br />Other supported models:<br /><br />- YOLOv5 SNPE: `yolov5.dlc`<br />- YOLOv8 SNPE: `yolov8.dlc`<br />- YOLO NAS SNPE: `yolonas.dlc`<br />- YOLOX SNPE: `yolox.dlc`<br />- YOLOv5 TFLite: `yolov5.tflite`<br />- YOLOv8 TFLite: `yolov8_det_quantized.tflite`<br />- YOLOX TFLite: `yolox_quantized.tflite`<br />- YOLO NAS TFLite: `yolonas_quantized.tflite`<br />- YOLOv7 TFLite: `Yolo-v7-Quantized.tflite`<br />- YOLOv8 QNN: `yolov8_det_quantized.bin`<br />- YOLOX QNN: `yolox_quantized.bin`<br /><br />Default model directory: `$HOME/Downloads/qimsdk_samples/models` | |
  | `labels` | Path to the label file.<br /><br />Default label file:<br />`yolox.json`<br /><br />Other supported label files:<br /><br />- YOLOv5: `yolov5.json`<br />- YOLOv8: `yolov8.json`<br />- YOLOX: `yolox.json`<br />- YOLO NAS: `yolonas.json`<br />- YOLOv7: `yolov7.json`<br /><br />Default labels directory: `$HOME/Downloads/qimsdk_samples/labels` | |
  | `threshold` | Detection threshold from `0` to `100`.<br /><br />Default: `40` | |
  | `output-type` | Supported output sinks:<br /><br />- `waylandsink` — Display output<br />- `filesink` — MP4 file output<br />- `rtspsink` — RTSP stream output | |
  | `output-file` | Output filename or path.<br /><br />Default: `$HOME/Downloads/qimsdk_samples/media/output_object_detection.mp4` | |
  | `video-format` | USB camera video format.<br /><br />Supported formats:<br /><br />- `nv12`<br />- `yuy2`<br />- `mjpeg`<br /><br />Applicable only to the USB camera source. | |
  | `width` | USB camera video width. | |
  | `height` | USB camera video height. | |
  | `framerate` | USB camera frame rate. | |
  | `runtime` | Supported hardware runtimes:<br /><br />- `cpu`<br />- `gpu`<br />- `dsp`<br /><br />Default: `dsp` | |
  | `snpe-tensors` | JSON array containing the output tensors for the SNPE model. | |
  | `output-ip-address` | RTSP output server IP address.<br /><br />Default: `127.0.0.1` | |
  | `port` | RTSP output server port.<br /><br />Default: `8900` | |
</Accordion>

### Image Classification

The [**gst-ai-classification**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-classification/main.c) application is designed to identify the subject in an image. The use cases are implemented using the Qualcomm Neural Processing SDK, LiteRT or Qualcomm AI Engine Direct models.

The pipeline receives a video stream from a camera, file source, USB source, or RTSP, preprocesses it, and runs the inference on AI hardware. The results are either displayed on the screen, saved as an encoded MP4 file, or streamed over the RTSP server. For information about the plugins used in the pipeline flow, see [**Pipeline flow**](#classification-pipeline-flow)

<Note>
  The `onnx` runtime is not supported for this sample application in the `QLI GA 2.1` release.
</Note>

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-classification`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-classification/main.c)

<Accordion title="Try me">
  When the software image includes the **qticamsrc** plugin, the camera framework uses it by default. If absent, the framework switches to **libcamera** instead. Since Config #1 lacks support for **qticamsrc**, the system defaults to **libcamera**.

  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | IMX577 camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - | - |
  | Config #1 | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes |
  | Config #2 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |

  <Note>
    `camera` use-case is not supported on `IQ-2390` and `CQ-2390`.
  </Note>

  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | Qualcomm Neural Processing SDK | `inceptionv3.dlc` | `classification.json` |
  | LiteRT | `inception_v3_quantized.tflite` | `classification.json` |
  | Qualcomm AI Engine Direct | `inception_v3_quantized.bin` | `classification.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-classification --config-file=/etc/configs/config_classification.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config_classification.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-classification -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-classification --config-file=/path/to/config_classification.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config_classification.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config_classification.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config_classification.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config_classification.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config_classification.json "$XDG_CONFIG_HOME/qimsdk_samples/config_classification.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#classification-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-classification
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-classification application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#classification-config-json-description) for all fields.

  ```json theme={null}
  {
    "file-path": "<path-to-input-video-file>",
    "ml-framework": "<snpe or tflite or qnn framework>",
    "model": "<path-to-model-file>",
    "labels": "<path-to-label-file>",
    "threshold": <post processsing threshold, integer value from 1 to 100>,
    "runtime": "<dsp, gpu, cpu runtime>",
    "output-type": "waylandsink or filesink or rtspsink"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `file-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |
  | `output-file` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |

  <Note>
    For USB camera input, set the `video-format`, `resolution`, and `framerate` parameters in the config file
    to match the camera capabilities, see [Configure USB camera](https://dragonwingdocs.qualcomm.com/System/Interfaces/usb#configure-usb-camera).
  </Note>

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        Camera source, LiteRT model, DSP runtime

        ```json theme={null}
        {
          "camera": 0,
          "ml-framework": "tflite",
          "model": "inception_v3_quantized.tflite",
          "labels": "classification.json",
          "threshold": 40,
          "runtime": "dsp",
          "output-type": "waylandsink"
        }

        ```
      </Tab>

      <Tab title="Config #1">
        Camera source, LiteRT model, CPU runtime

        ```json theme={null}
        {
          "ml-framework": "tflite",
          "model": "inception_v3_quantized.tflite",
          "labels": "classification.json",
          "threshold": 40,
          "runtime": "cpu",
          "output-type": "waylandsink"
        }
        ```

        <Note>
          Config #1 supports only LiteRT models and the CPU runtime.
        </Note>
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  The classified object label and confidence score are overlaid on the video and displayed on the local display

  <img alt="Expected output for gst-ai-classification application" />

  #### Pipeline Flow

  The following table lists the plugins used in the classification pipeline:

  | Plugin | Description |
  | :- | :- |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | • Captures the live stream from camera.<br />• Uses `tee` to split the stream for inferencing. |
  | `filesrc` | • Captures the video stream using `filesrc`, followed by `qtdemux`, which demultiplexes the stream.<br />• Uses `tee` to split the stream for inferencing. |
  | `rtspsrc` | • Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction.<br />• Uses `tee` to split the stream for inferencing. |
  | `v4l2src` | • Captures the live stream from USB camera.<br />• Uses `tee` to split the stream for inferencing. |
  | `h264parse` | Parses the H.264 video bitstream. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | Hardware-decodes H.264 video to raw frames. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data:<br />   • Color conversion<br />   • Scaling (up or down)<br />   • Normalization<br />3. Converts the preprocessed video stream to a tensor stream on its source pad.<br />The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimlsnpe`](../plugin-reference/qtimlsnpe)<br />[`qtimltflite`](../plugin-reference/qtimltflite)<br />[`qtimlqnn`](../plugin-reference/qtimlqnn) | 1. After the inference runtime receives the tensor stream on its sink pad, it runs inference using the provided model.<br />2. Produces a tensor stream with the inference results on its source pad. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | Handles inference results from any classification model.<br />1. Applies a threshold to the chosen number of results.<br />2. Loads the MobileNet-softmax postprocessing module.<br />3. Produces results as video frames with classification labels.<br />4. Sends these processed frames to the sink pad of `qtivcomposer`. |
  | [`qtivcomposer`](../plugin-reference/qtivcomposer) | 1. Composes frames with contents from its sink pads.<br />2. Pushes the GStreamer buffers containing these composed frames to its source pad. |
  | [`waylandsink`](../plugin-reference/waylandsink) | 1. Submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |
  | `filesink` | Receives the video stream on sink pad and saves it as an H.264-encoded MP4 file. |
  | [`qtirtspbin`](../plugin-reference/qtirtspbin) | 1. Serves as a network sink.<br />2. Transmits UDP packets to the network. |

  #### Config JSON Field Description

  | Field | Values / Description |
  | - | - |
  | `ml-framework` | Supported ML frameworks:<br /><br />- `snpe` — Qualcomm Neural Processing SDK<br />- `tflite` — LiteRT<br />- `qnn` — Qualcomm AI Engine Direct<br /><br />Default: `snpe` |
  | `runtime` | Supported hardware runtimes:<br /><br />- `cpu`<br />- `gpu`<br />- `dsp`<br /><br />Default: `dsp` |
  | `file-path` | Path to the input media file.<br /><br />Default media directory: `$HOME/Downloads/qimsdk_samples/media` |
  | `rtsp-ip-port` | RTSP input URL. |
  | `camera` | Camera selection:<br /><br />- `0` — Primary camera<br />- `1` — Secondary camera |
  | `enable-usb-camera` | Enables or disables the USB camera.<br /><br />Supported values: `TRUE` or `FALSE` |
  | `model` | Path to the model file.<br /><br />Default model file:<br />`inception_v3_quantized.tflite`<br /><br />Other supported models:<br /><br />- SNPE: `inceptionv3.dlc`<br />- TFLite: `inception_v3_quantized.tflite`<br />- QNN: `inception_v3_quantized.bin`<br /><br />Default model directory: `$HOME/Downloads/qimsdk_samples/models` |
  | `labels` | Path to the label file.<br /><br />Default: `classification.json`<br /><br />Default labels directory: `$HOME/Downloads/qimsdk_samples/labels` |
  | `output-type` | Supported output sinks:<br /><br />- `waylandsink` — Display output<br />- `filesink` — MP4 file output<br />- `rtspsink` — RTSP stream output |
  | `output-file` | Output filename or path.<br /><br />Default: `$HOME/Downloads/qimsdk_samples/media/output_classification.mp4` |
  | `output-ip-address` | RTSP output server IP address.<br /><br />Default: `127.0.0.1` |
  | `port` | RTSP output server port.<br /><br />Default: `8900` |
  | `video-format` | USB camera video format.<br /><br />Supported formats:<br /><br />- `nv12`<br />- `yuy2`<br />- `mjpeg` |
  | `width` | USB camera video width. |
  | `height` | USB camera video height. |
  | `framerate` | USB camera frame rate. |
  | `threshold` | Detection threshold from `0` to `100`.<br /><br />Default: `40` |
</Accordion>

### Face Detection

The [**gst-ai-face-detection**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-face-detection/main.c) application collects the live video input from a camera, file, or an RTSP stream and uses the Qualcomm AI Engine direct and LiteRT face detection models to produce a preview with the overlaid AI model output on the HDMI display.

The following figure shows the pipeline, which receives the input, preprocesses it, runs inferences on AI hardware, and displays the results on the screen. For information about the plugins used in the pipeline flow, see [**Pipeline flow**](#face-detection-pipeline-flow).

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-face-detection`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-face-detection/main.c)

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - |
  | Config #1 | Yes | Yes | No | No | No | Yes | No |
  | Config #2 | Yes | Yes | No | Yes | No | Yes | No |

  <Note>
    `camera` use-case is not supported on `IQ-2390` and `CQ-2390`.
  </Note>

  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | `face_det_lite_quantized.tflite` | `face_detection.json` |
  | Qualcomm AI Engine Direct | `face_det_lite_quantized.bin` | `face_detection.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-face-detection --config-file=/etc/configs/config_face_detection.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config_face_detection.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-face-detection -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-face-detection --config-file=/path/to/config_face_detection.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config_face_detection.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config_face_detection.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config_face_detection.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config_face_detection.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config_face_detection.json "$XDG_CONFIG_HOME/qimsdk_samples/config_face_detection.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#face-detection-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-face-detection
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-face-detection application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#face-detection-config-json-description) for all fields.

  ```json theme={null}
  {
    "file-path": "<input-video-path>",
    "ml-framework": "<tflite or qnn framework>",
    "model": "<path-to-model-file",
    "labels": "<path-to-label-file",
    "threshold": <post-processing threshold, integer value from 1-100>,
    "runtime": "<cpu, gpu or dsp runtime>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `file-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        File source, LiteRT model, DSP runtime

        ```json theme={null}
        {
          "file-path": "video.mp4",
          "ml-framework": "tflite",
          "model":"face_det_lite_quantized.tflite",
          "labels": "face_detection.json",
          "threshold": 51,
          "runtime": "dsp"
        }
        ```
      </Tab>

      <Tab title="Config #1">
        File source, LiteRT model, CPU runtime

        ```json theme={null}
        {
          "file-path": "video.mp4",
          "ml-framework": "tflite",
          "model":"face_det_lite_quantized.tflite",
          "labels": "face_detection.json",
          "threshold": 51,
          "runtime": "cpu"
        }
        ```

        <Note>
          Config #1 supports only LiteRT models and the CPU runtime.
        </Note>
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  <img alt="Output Image Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the face detection pipeline:

  | Plugin | Description |
  | :- | :- |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | • Captures the live stream from camera.<br />• Uses `tee` to split the stream for inferencing. |
  | `rtspsrc` | • Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction.<br />• Uses `tee` to split the stream for inferencing. |
  | `h264parse` | • Parses the H.264 video bitstream. |
  | [`v4l2h264dec`](/plugin-reference/v4l2h264dec) | • Hardware-decodes H.264 video to raw frames. |
  | [`qtimlvconverter`](/plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data:<br />   • Color conversion<br />   • Scaling (up or down)<br />   • Normalization<br />3. Converts the preprocessed video stream to a tensor stream.<br />The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimltflite`](/plugin-reference/qtimltflite)<br />[`qtimlqnn`](/plugin-reference/qtimlqnn) | 1. After the inference runtime receives the tensor stream on its sink pad, it runs inference using the provided model.<br />2. Produces a tensor stream with the inference results on its source pad. |
  | [`qtimlpostprocess`](/plugin-reference/qtimlpostprocess) | 1. Handles inference results from any face detection model.<br />2. Applies a threshold to the chosen number of results. |
  | [`qtimetamux`](/plugin-reference/qtimetamux) | 1. Receives string-based postprocessing output text with video frame and multiplexes it. |
  | [`qtivoverlay`](/plugin-reference/qtivoverlay) | 1. Receives the multiplexed stream.<br />2. Overlays the bounding boxes on the stream. |
  | [`waylandsink`](/plugin-reference/waylandsink) | 1. Submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |

  #### Config JSON Field Description

  | Field | Values/descriptions |
  | - | - |
  | `camera` | Select `0` for the primary camera or `1` for the secondary camera. |
  | `file-path` | File source path.<br /><br />Default media directory: `$HOME/Downloads/qimsdk_samples/media` |
  | `rtsp-ip-port` | RTSP input URL. |
  | `ml-framework` | Supported ML frameworks:<br /><br />- `tflite` (LiteRT)<br />- `qnn` (Qualcomm AI Engine Direct) |
  | `model` | Optional model path that overrides the default model path.<br /><br />Default model: `face_det_lite_quantized.tflite`<br /><br />Other supported models:<br /><br />- TFLite default: `face_det_lite_quantized.tflite`<br />- QNN default: `face_det_lite_quantized.bin`<br /><br />Default models directory: `$HOME/Downloads/qimsdk_samples/models` |
  | `labels` | Optional labels path that overrides the default labels path.<br /><br />Default: `face_detection.json`<br /><br />Default labels directory: `$HOME/Downloads/qimsdk_samples/labels` |
  | `threshold` | Detection threshold from `0` to `100`.<br /><br />Default: `51` |
  | `runtime` | Supported hardware runtimes:<br /><br />- `cpu`<br />- `gpu`<br />- `dsp`<br /><br />Default: `dsp` |
</Accordion>

***

### Semantic Segmentation

The [**gst-ai-segmentation**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-segmentation/main.c) application allows you to divide an image into different and meaningful parts or segments and assign a label to each homogeneous segment based on the similarity of the attributes. The application uses Qualcomm Neural Processing SDK runtime, Qualcomm AI Engine direct runtime, and LiteRT for image segmentation.

The following figure shows the pipeline, which receives the input from a live camera feed, file, or an RTSP stream, preprocesses the video data, runs inferences using AI hardware, and displays the segmented data on the screen. For information about the plugins used in the pipeline flow, see [**Pipeline flow**](#segmentation-pipeline-flow).

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-segmentation`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-segmentation/main.c)

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - |
  | Config #1 | Yes | Yes | No | No | No | Yes | No |
  | Config #2 | Yes | Yes | No | Yes | No | Yes | No |

  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | Qualcomm Neural Processing SDK | `deeplabv3_plus_mobilenet.dlc` | `deeplabv3_resnet50.json` |
  | LiteRT | `deeplabv3_plus_mobilenet_quantized.tflite` | `deeplabv3_resnet50.json` |
  | Qualcomm AI Engine Direct | `deeplabv3_plus_mobilenet_quantized.bin` | `deeplabv3_resnet50.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-segmentation --config-file=/etc/configs/config_segmentation.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config_segmentation.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-segmentation -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-segmentation --config-file=/path/to/config_segmentation.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config_segmentation.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config_segmentation.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config_segmentation.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config_segmentation.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config_segmentation.json "$XDG_CONFIG_HOME/qimsdk_samples/config_segmentation.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#segmentation-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-segmentation
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-segmentation application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#segmentation-config-json-description) for all fields.

  ```json theme={null}
  {
    "file-path": "<input-video-path>",
    "ml-framework": "<snpe, tflite, or qnn framework>",
    "model": "<path-to-model-file>",
    "labels": "<path-to-label-file>",
    "runtime": "<dsp, gpu, or cpu runtime>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `file-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        File source, LiteRT model, DSP runtime

        ```json theme={null}
        {
         "file-path": "video.mp4",
         "ml-framework": "tflite",
         "model": "deeplabv3_plus_mobilenet_quantized.tflite",
         "labels": "deeplabv3_resnet50.json",
         "runtime": "dsp"
        }
        ```
      </Tab>

      <Tab title="Config #1">
        File source, LiteRT model, CPU runtime

        ```json theme={null}
        {
          "file-path": "video.mp4",
          "ml-framework": "tflite",
          "model": "deeplabv3_plus_mobilenet_quantized.tflite",
          "labels": "deeplabv3_resnet50.json",
          "runtime": "cpu"
        }
        ```

        <Note>
          Config #1 supports only LiteRT models and the CPU runtime.
        </Note>
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  The segmented data is displayed on the local display.

  <img alt="Expected output for gst-ai-classification application" />

  #### Pipeline Flow

  The following table lists the plugins used in the segmentation pipeline:

  | Plugin | Description |
  | :- | :- |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | • Captures the live stream from camera.<br />• Uses `tee` to split the stream for concurrent display and ML inference. |
  | `filesrc` | • Captures the video stream using `filesrc`, followed by `qtdemux`, which demultiplexes the stream.<br />• Uses `tee` to split the stream for processing. |
  | `rtspsrc` | • Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction.<br />• Uses `tee` to split the stream for processing. |
  | `h264parse` | • Parses the H.264 video bitstream to ensure downstream elements can handle the payload. |
  | [`v4l2h264dec`](/plugin-reference/v4l2h264dec) | • Hardware-accelerated decoder that converts H.264 video into raw frames. |
  | [`qtimlvconverter`](/plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data. This preprocessing is done when the model expects floating-point values as input:<br />   • Color conversion<br />   • Scaling (up or down)<br />   • Normalization<br />3. Converts the preprocessed video stream to a tensor stream on its source pad.<br />The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimlsnpe`](/plugin-reference/qtimlsnpe)<br />[`qtimltflite`](/plugin-reference/qtimltflite)<br />[`qtimlqnn`](/plugin-reference/qtimlqnn) | 1. After the inference runtime receives the tensor stream on its sink pad, it runs inference using the provided model.<br />2. Produces a tensor stream with the inference results on its source pad. |
  | [`qtimlpostprocess`](/plugin-reference/qtimlpostprocess) | 1. Converts the inference tensors received on its sink pad into video formats that multimedia plugins use for further processing. |
  | [`qtivcomposer`](/plugin-reference/qtivcomposer) | 1. Composes frames with contents from its sink pads.<br />2. Pushes the GStreamer buffers containing these composed frames to its source pad. |
  | [`waylandsink`](/plugin-reference/waylandsink) | 1. Submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |

  #### Config JSON Field Description

  | Field | Values / Descriptions |
  | - | - |
  | `ml-framework` | Supported ML frameworks:<br /><br />- `tflite` — LiteRT<br />- `qnn` — Qualcomm AI Engine Direct<br /><br />Default: `tflite` |
  | `runtime` | Supported hardware runtimes:<br /><br />- `cpu`<br />- `gpu`<br />- `dsp`<br /><br />Default: `dsp` |
  | `Input source` | Supported input sources:<br /><br />- `file-path`<br />- `rtsp-ip-port`<br />- `camera`<br /><br />Default input source: `camera` |
  | `file-path` | Path to the input media file.<br /><br />Default: `$HOME/Downloads/qimsdk_samples/media` |
  | `rtsp-ip-port` | RTSP input URL. |
  | `camera` | Camera selection:<br /><br />- `0` — Primary camera<br />- `1` — Secondary camera |
  | `model` | Optional model path that overrides the default model path.<br /><br />Default model: `deeplabv3_plus_mobilenet_quantized.tflite`<br /><br />Default model directory: `$HOME/Downloads/qimsdk_samples/models` |
  | `labels` | Optional labels path that overrides the default labels path.<br /><br />Default labels file: `deeplabv3_resnet50.json`<br /><br />Default labels directory: `$HOME/Downloads/qimsdk_samples/labels` |
</Accordion>

### Pose Detection

The [**gst-ai-pose-detection**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-pose-detection/main.c) application allows you to detect the body pose of the subject in an image or video. The use case processes input streams from a camera, file, or an RTSP source and uses LiteRT and Qualcomm AI Engine direct models for pose detection. The results are either displayed on the screen, saved as an encoded MP4 file, or streamed over the RTSP server.

The following figure shows the pipeline, which receives the input from a live camera feed, file, USB source, or an RTSP stream, preprocesses it, conducts inference on AI hardware, and generates the output.

This process allows for real-time pose detection and visualization of human poses. For information about the plugins used in the pipeline flow, see [**Pipeline flow**](#pose-pipeline-flow).

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-pose-detection`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-pose-detection/main.c)

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - |
  | Config #1 | Yes | Yes | Yes | No | Yes | Yes | Yes |
  | Config #2 | Yes | Yes | Yes | Yes | Yes | Yes | Yes |

  <Note>
    `camera` use-case is not supported on `IQ-2390` and `CQ-2390`.
  </Note>

  #### Sample Model and Label Files

  | Runtime | Model file | Label files |
  | - | - | - |
  | LiteRT | `hrnet_pose_quantized.tflite` | `hrnet_pose.json`, `hrnet_settings.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-pose-detection --config-file=/etc/configs/config_pose.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config_pose.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-pose-detection -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-pose-detection --config-file=/path/to/config_pose.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config_pose.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config_pose.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config_pose.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config_pose.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config_pose.json "$XDG_CONFIG_HOME/qimsdk_samples/config_pose.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#pose-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-pose-detection
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-pose-detection application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#pose-config-json-description) for all fields.

  ```json theme={null}
  {
    "file-path": "<input-video-path>",
    "ml-framework": "<tflite or qnn framework>",
    "model": "<path-to-model-file>",
    "labels": "<path-to-label-file>",
    "pose-settings-path": "<path-to-hrnet_settings-file>",
    "output-type": "waylandsink or filesink or rtspsink",
    "runtime": "<cpu, gpu or dsp runtime>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `file-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |
  | `output-file` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |

  <Note>
    For USB camera input, set the `video-format`, `resolution`, and `framerate`
    parameters in the config file to match the camera capabilities, see [Configure USB camera](https://dragonwingdocs.qualcomm.com/System/Interfaces/usb#configure-usb-camera).
  </Note>

  <Note>
    To change the threshold, you must configure the confidence value in the
    `hrnet_settings.json` file.
  </Note>

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        File source, LiteRT model, DSP runtime

        ```json theme={null}
        {
          "file-path": "video.mp4",
          "ml-framework": "tflite",
          "model": "hrnet_pose_quantized.tflite",
          "labels": "hrnet_pose.json",
          "pose-settings-path":"hrnet_settings.json",
          "runtime": "dsp",
          "output-type": "waylandsink"
        }
        ```
      </Tab>

      <Tab title="Config #1">
        File source, LiteRT model, CPU runtime

        ```json theme={null}
        {
          "file-path": "video.mp4",
          "ml-framework": "tflite",
          "model": "hrnet_pose_quantized.tflite",
          "labels": "hrnet_pose.json",
          "pose-settings-path":"hrnet_settings.json",
          "runtime": "cpu",
          "output-type": "waylandsink"
        }
        ```

        <Note>
          Config #1 supports only LiteRT models and the CPU runtime.
        </Note>
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  The displayed output shows the detected pose of the objects.

  <img alt="Pipeline Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the pose detection pipeline:

  | Plugin | Description |
  | :- | :- |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | • Captures the live stream from camera.<br />• Uses `tee` to split the stream for concurrent display and ML inference. |
  | `filesrc` | • Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing.<br />• Uses `tee` to split the stream for processing. |
  | `rtspsrc` | • Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction.<br />• Uses `tee` to split the stream for processing. |
  | `v4l2src` | • Captures the live stream from USB camera.<br />• Uses `tee` to split the stream for processing. |
  | `h264parse` | • Parses the H.264 video bitstream to ensure downstream elements can handle the payload. |
  | [`v4l2h264dec`](/plugin-reference/v4l2h264dec) | • Hardware-accelerated decoder that converts H.264 video into raw frames. |
  | [`qtimlvconverter`](/plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data:<br />   • Color conversion<br />   • Scaling (up or down)<br />   • Normalization<br />3. Converts the preprocessed video stream to a tensor stream.<br />The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimltflite`](/plugin-reference/qtimltflite)<br />[`qtimlqnn`](/plugin-reference/qtimlqnn) | 1. Uses the HRNet model for pose detection.<br />2. The application runs on the external delegate to execute the model using the Hexagon Tensor Processor.<br />3. After the inference runtime receives the tensor stream on its sink pad, it does the following:<br />   • Runs the inference.<br />   • Produces a tensor stream containing the inference results on its source pad.<br />   • Manages the inference results from the pose detection model. |
  | [`qtimlpostprocess`](/plugin-reference/qtimlpostprocess) | 1. Applies a threshold to the chosen number of results.<br />2. Loads corresponding modules for various pose detection models.<br />3. In this specific use case, `qtimlpostprocess` does the following:<br />   • Loads the HRNet module.<br />   • Produces results in the form of video frames with drawn poses.<br />   • Sends the results to the sink pad of `qtivcomposer` for further processing or display. |
  | [`qtivcomposer`](/plugin-reference/qtivcomposer) | 1. Composes frames with contents from its sink pads.<br />2. Pushes the GStreamer buffers containing these composed frames to its source pad. |
  | [`waylandsink`](/plugin-reference/waylandsink) | 1. Submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |
  | `filesink` | 1. Receives the video stream on its sink pad and saves it as an H.264-encoded MP4 file. |
  | [`qtirtspbin`](/plugin-reference/qtirtspbin) | 1. Serves as a network sink.<br />2. Transmits UDP packets to the network. |

  #### Config JSON Field Description

  | Field | Values / Description |
  | - | - |
  | `ml-framework` | Supported ML frameworks:<br /><br />- `tflite` — LiteRT<br />- `qnn` — Qualcomm AI Engine Direct<br /><br />Default: `tflite` |
  | `runtime` | Supported hardware runtimes:<br /><br />- `cpu`<br />- `gpu`<br />- `dsp`<br /><br />Default: `dsp` |
  | `file-path` | Path to the input media file.<br /><br />Default media directory: `$HOME/Downloads/qimsdk_samples/media` |
  | `rtsp-ip-port` | RTSP input URL. |
  | `camera` | Camera selection:<br /><br />- `0` — Primary camera<br />- `1` — Secondary camera |
  | `enable-usb-camera` | Enables or disables the USB camera.<br /><br />Supported values: `TRUE` or `FALSE` |
  | `model` | Path to the model file.<br /><br />Default model file:<br /> `hrnet_pose_quantized.tflite`<br /><br />Other supported models:<br /><br />- TFLite: `hrnet_pose_quantized.tflite`<br />- QNN: `hrnet_pose_quantized.bin`<br /><br />Default model directory: `$HOME/Downloads/qimsdk_samples/models` |
  | `labels` | Path to the labels file.<br /><br />Default: `hrnet_pose.json`<br /><br />Default labels directory: `$HOME/Downloads/qimsdk_samples/labels` |
  | `pose-settings-path` | Path to the pose settings file.<br /><br />Default: `hrnet_settings.json`<br /><br />Default settings directory: `$HOME/Downloads/qimsdk_samples/labels` |
  | `output-type` | Supported output sinks:<br /><br />- `waylandsink` — Display output<br />- `filesink` — MP4 file output<br />- `rtspsink` — RTSP stream output |
  | `output-file` | Output filename or path.<br /><br />Default: `$HOME/Downloads/qimsdk_samples/media/output_pose.mp4` |
  | `output-ip-address` | RTSP output server IP address.<br /><br />Default: `127.0.0.1` |
  | `port` | RTSP output server port.<br /><br />Default: `8900` |
  | `video-format` | USB camera video format.<br /><br />Supported formats:<br /><br />- `nv12`<br />- `yuy2`<br />- `mjpeg`<br /><br />Applicable only to the USB camera source. |
  | `width` | USB camera video width. |
  | `height` | USB camera video height. |
  | `framerate` | USB camera frame rate. |

  <Note>
    For better accuracy and detection results, use the
    **gst-ai-daisychain-detection-pose** application.
  </Note>
</Accordion>

### Mono Depth

The [**gst-ai-monodepth**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-monodepth/main.c) application allows you to infer depth of a source feed from a live camera stream, file, or an RTSP stream.

The following figure shows the pipeline, which captures feed from the source, preprocesses it, and runs inferences using the AI hardware. For information about the plugins used in the pipeline, see [**Pipeline flow**](#monodepth-pipeline-flow).

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-monodepth`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-monodepth/main.c)

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - |
  | Config #1 | Yes | Yes | No | No | No | Yes | No |
  | Config #2 | Yes | Yes | No | Yes | No | Yes | No |

  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | Qualcomm Neural Processing SDK | `midasv2.dlc` | `monodepth.json` |
  | LiteRT | `midas_quantized.tflite` | `monodepth.json` |
  | Qualcomm AI Engine Direct | `midas_quantized.bin` | `monodepth.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-monodepth --config-file=/etc/configs/config_monodepth.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config_monodepth.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-monodepth -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-monodepth --config-file=/path/to/config_monodepth.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config_monodepth.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config_monodepth.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config_monodepth.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config_monodepth.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config_monodepth.json "$XDG_CONFIG_HOME/qimsdk_samples/config_monodepth.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#monodepth-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-monodepth
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-monodepth application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#monodepth-config-json-description) for all fields.

  ```json theme={null}
  {
    "file-path": "<input video path>",
    "ml-framework": "<snpe, tflite, or qnn framework>",
    "model": "<path-to-model-file>",
    "labels": "<path-to-label-file>",
    "runtime": "<dsp, gpu, or cpu runtime>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `file-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        File source, LiteRT model, DSP runtime

        ```json theme={null}
        {
          "file-path": "video.mp4",
          "ml-framework": "tflite",
          "model": "midas_quantized.tflite",
          "labels": "monodepth.json",
          "runtime": "dsp"
        }
        ```
      </Tab>

      <Tab title="Config #1">
        File source, LiteRT model, CPU runtime

        ```json theme={null}
        {
          "file-path": "video.mp4",
          "ml-framework": "tflite",
          "model": "midas_quantized.tflite",
          "labels": "monodepth.json",
          "runtime": "cpu"
        }
        ```

        <Note>
          Config #1 supports only LiteRT models and the CPU runtime.
        </Note>
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  The overlaid model output stream is shown side by side with the live feed.

  <img alt="Pipeline Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the monodepth pipeline:

  | Plugin | Description |
  | :- | :- |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | • Captures the live stream from camera.<br />• Uses `tee` to split the stream for concurrent display and ML inference. |
  | `filesrc` | • Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing.<br />• Uses `tee` to split the stream for processing. |
  | `rtspsrc` | • Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction.<br />• Uses `tee` to split the stream for processing. |
  | `h264parse` | • Parses the H.264 video bitstream to ensure downstream elements can handle the payload. |
  | [`v4l2h264dec`](/plugin-reference/v4l2h264dec) | • Hardware-accelerated decoder that converts H.264 video into raw frames. |
  | [`qtimlvconverter`](/plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data. This preprocessing is done when the model expects floating-point values as input:<br />   • Color conversion<br />   • Scaling (up or down)<br />   • Normalization<br />3. Converts the preprocessed video stream to a tensor stream on its source pad.<br />The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimlsnpe`](/plugin-reference/qtimlsnpe)<br />[`qtimltflite`](/plugin-reference/qtimltflite)<br />[`qtimlqnn`](/plugin-reference/qtimlqnn) | Uses the **Midasv2** model for monodepth calculation.<br />1. The inference runtime receives the tensor stream on its sink pad.<br />2. The runtime executes the inference.<br />3. Produces a tensor stream with the inference results on its source pad. |
  | [`qtimlpostprocess`](/plugin-reference/qtimlpostprocess) | 1. Converts the inference tensors received on its sink pad into video formats that multimedia plugins use for further processing. |
  | [`qtivtransform`](/plugin-reference/qtivtransform) | 1. Converts the buffers on its source pad to formats compatible with composition on `waylandsink`. |
  | [`waylandsink`](/plugin-reference/waylandsink) | 1. Submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |

  #### Config JSON Field Description

  | Field | Values / Descriptions |
  | - | - |
  | `ml-framework` | Supported ML frameworks:<br /><br />- `snpe` — Qualcomm Neural Processing SDK<br />- `tflite` — LiteRT<br />- `qnn` — Qualcomm AI Engine Direct<br /><br />Default: `snpe` |
  | `runtime` | Supported hardware runtimes:<br /><br />- `cpu`<br />- `gpu`<br />- `dsp`<br /><br />Default: `dsp` |
  | `file-path` | Path to the input media file.<br /><br />Default media directory: `$HOME/Downloads/qimsdk_samples/media` |
  | `rtsp-ip-port` | RTSP input URL. |
  | `camera` | Camera selection:<br /><br />- `0` — Primary camera<br />- `1` — Secondary camera |
  | `model` | Path to the model file.<br /><br />Default model file:<br />`midas_quantized.tflite`<br /><br />Other supported models:<br /><br />- SNPE DLC: `midasv2.dlc`<br />- TFLite: `midas_quantized.tflite`<br />- QNN: `midas_quantized.bin`<br /><br />Default model directory: `$HOME/Downloads/qimsdk_samples/models` |
  | `labels` | Path to the label file.<br /><br />Default: `monodepth.json`<br /><br />Default labels directory: `$HOME/Downloads/qimsdk_samples/labels` |
</Accordion>

### Super Resolution

The [**gst-ai-superresolution**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-superresolution/main.c) application allows you to generate high resolution video frames from low-resolution input.

The following figures shows the pipeline, which receives a video stream from a file source as input, processes it through the super resolution module using LiteRT, and displays the output.

<Note>
  This application isn't supported in `Config #1` of `Qualcomm Linux` because CPU runtime is not supported.
</Note>

For information about the plugins used in the pipeline, see [**Pipeline flow**](#superresolution-pipeline-flow).

<img alt="Pipeline Diagram" />

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-superresolution`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-superresolution/main.c)

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - |
  | Config #2 | Yes | No | No | No | Yes | Yes | No |

  #### Sample Model Files

  | Runtime | Model file |
  | - | - |
  | LiteRT | `quicksrnetsmall_quantized.tflite` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-superresolution --config-file=/etc/configs/config-superresolution.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config-superresolution.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-superresolution -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-superresolution --config-file=/path/to/config-superresolution.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config-superresolution.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config-superresolution.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config-superresolution.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config-superresolution.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config-superresolution.json "$XDG_CONFIG_HOME/qimsdk_samples/config-superresolution.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#superresolution-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-superresolution
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-superresolution application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#superresolution-config-json-description) for all fields.

  ```json theme={null}
  {
    "input-file-path": "<input-video-path>",
    "model": "<path-to-model-file>",
    "output-file-path": "<path-to-output-video>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `input-file-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `output-file-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |

  <Note>
    The video super‑resolution application requires an input video resolution of
    128 × 128.
  </Note>

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        File source, LiteRT model, DSP runtime

        ```json theme={null}
        {
          "input-file-path": "video.mp4",
          "model": "quicksrnetsmall_quantized.tflite"
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  The output is displayed on an HDMI monitor.

  <img alt="Pipeline Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the superresolution pipeline:

  | Plugin | Description |
  | :- | :- |
  | `filesrc` | • Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing.<br />• Uses `tee` to split the stream for processing. |
  | `h264parse` | • Parses the H.264 video bitstream to ensure downstream elements can handle the payload. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | • Hardware-accelerated decoder that converts H.264 video into raw frames. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data. This preprocessing is done when the model expects floating-point values as input:<br />   • Color conversion<br />   • Scaling (up or down)<br />   • Normalization<br />3. Converts the preprocessed video stream to a tensor stream on its source pad.<br />The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | • Runs on LiteRT and uses the `quicksrnetsmall_quantized` model for super resolution.<br />1. The inference runtime receives the tensor stream on its sink pad.<br />2. The runtime executes the inference.<br />3. Produces a tensor stream with the inference results on its source pad. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | • Handles inference results from any super resolution model.<br />1. Loads the SRNet module.<br />2. Produces results as high-resolution video frames.<br />3. Sends the processed frames to the sink pad of `qtivcomposer`. |
  | [`qtivcomposer`](../plugin-reference/qtivcomposer) | 1. Composes frames with contents from its sink pads.<br />2. Pushes the GStreamer buffers containing these composed frames to its source pad. |
  | [`waylandsink`](../plugin-reference/waylandsink) | 1. Submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |

  #### Config JSON Field Description

  | Field | Values / Description |
  | - | - |
  | `input-file-path` | Path to the input media file.<br /><br />Default: `video.mp4`<br /><br />Default media directory: `$HOME/Downloads/qimsdk_samples/media`<br /><br />Alternatively, provide an absolute file path. |
  | `model` | Path to the super-resolution model file.<br /><br />Default: `quicksrnetsmall_quantized.tflite`<br /><br />Default model directory: `$HOME/Downloads/qimsdk_samples/models`<br /><br />Alternatively, provide an absolute file path. |
  | `output-file-path` | Output file path.<br /><br />For a relative path, the output is saved under `$HOME/Downloads/qimsdk_samples/media`.<br /><br />Alternatively, provide an absolute file path.<br /><br />If not specified, display output is selected. |
</Accordion>

***

### AI Event Encoder

The [**gst-ai-event-encoder**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-event-encoder/main.c) application receives the live video stream input from camera, file, or RTSP source. When a human enters the video frame the application preprocesses the video, runs inferences on the AI hardware, and encodes the video. The encoding stops 5 seconds after the human moves away from the frame and restarts when anyone enters the frame.

The following figures show the event detection and recording pipelines for event encoder application. For information about the plugins used in the pipeline flow, see [**Pipeline flow**](#event-encoder-pipeline-flow).

<img alt="Pipeline Diagram" />

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-event-encoder`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-event-encoder/main.c)

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - |
  | Config #1 | Yes | Yes | No | No | No | Yes | No |
  | Config #2 | Yes | Yes | No | Yes | No | Yes | No |

  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | `yolox_quantized.tflite` | `yolox.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-event-encoder --config-file=/etc/configs/config-event-encoder.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config-event-encoder.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-event-encoder -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-event-encoder --config-file=/path/to/config-event-encoder.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config-event-encoder.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config-event-encoder.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config-event-encoder.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config-event-encoder.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config-event-encoder.json "$XDG_CONFIG_HOME/qimsdk_samples/config-event-encoder.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#event-encoder-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-event-encoder
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-event-encoder application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#event-encoder-config-json-description) for all fields.

  ```json theme={null}
  {
    "file-path": "<path to video file>",
    "model": "<path to model file>",
    "labels": "<path to label file>",
    "threshold": <integer between 1 and 100>,
    "runtime": "<cpu, gpu, or dsp runtime>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `file-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        File source, LiteRT model, DSP runtime

        ```json theme={null}
        {
          "file-path": "video.mp4",
          "model": "yolox_quantized.tflite",
          "labels": "yolox.json",
          "threshold": 40,
          "runtime": "dsp"
        }
        ```
      </Tab>

      <Tab title="Config #1">
        File source, LiteRT model, CPU runtime

        ```json theme={null}
        {
          "file-path": "video.mp4",
          "model": "yolox_quantized.tflite",
          "labels": "yolox.json",
          "threshold": 40,
          "runtime": "cpu"
        }
        ```

        <Note>
          Config #1 supports only LiteRT models and the CPU runtime.
        </Note>
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  The output is saved as an MP4 file within `${HOME}/Downloads/qimsdk_samples/media` folder as `output-1.mp4`, `output-2.mp4`, and so on.

  <img alt="Output Image Diagram Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the event encoder pipeline:

  | Plugin | Description |
  | :- | :- |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | • Captures the live stream from camera.<br />• Uses `tee` to split the stream for concurrent display and ML inference. |
  | `filesrc` | • Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing.<br />• Uses `tee` to split the stream for processing. |
  | `rtspsrc` | • Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction.<br />• Uses `tee` to split the stream for processing. |
  | `h264parse` | • Parses the H.264 video bitstream to ensure downstream elements can handle the payload. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | • Hardware-accelerated decoder that converts H.264 video into raw frames. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data:<br />   • Color conversion<br />   • Scaling (up or down)<br />   • Normalization<br />3. Converts the preprocessed video stream to a tensor stream.<br />The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | 1. After the inference runtime receives the tensor stream on its sink pad, it executes the inference.<br />2. Produces a tensor stream with the inference results on its source pad. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | **Instance 1 (Detection Overlay)**:<br />   • Applies a threshold to the chosen number of results.<br />   • Loads the YOLOv8 module.<br />   • Produces video frames with only bounding boxes for object overlay.<br />   • Sends processed frames to the sink pad of `qtivcomposer`.<br /><br />**Instance 2 (Metadata Generation)**:<br />   • Produces output in text format (bounding box coordinates and labels).<br />   • Connects to an `appsink` plugin where metadata is read, parsed, and logged.<br />   • Uses bounding box information to count the number of humans in each frame. |
  | [`qtivcomposer`](../plugin-reference/qtivcomposer) | 1. Composes frames with contents from its sink pads.<br />2. Pushes the GStreamer buffers containing these composed frames to its source pad. |
  | [`waylandsink`](../plugin-reference/waylandsink) | 1. Submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |

  #### Config JSON Field Description

  | Field | Values / Descriptions |
  | - | - |
  | `runtime` | Supported hardware runtimes:<br /><br />- `cpu`<br />- `gpu`<br />- `dsp`<br /><br />Default: `dsp` |
  | `file-path` | Path to the input media file.<br /><br />Default media directory: `$HOME/Downloads/qimsdk_samples/media`<br /><br />Alternatively, provide an absolute file path. |
  | `rtsp-ip-port` | RTSP input URL. |
  | `camera` | Camera selection:<br /><br />- `0` — Primary camera<br />- `1` — Secondary camera |
  | `model` | Optional model path that overrides the default model path.<br /><br />Default: `yolox_quantized.tflite`<br /><br />Default model directory: `$HOME/Downloads/qimsdk_samples/models`<br /><br />Alternatively, provide an absolute file path. |
  | `labels` | Optional labels path that overrides the default labels path.<br /><br />Default: `yolox.json`<br /><br />Default labels directory: `$HOME/Downloads/qimsdk_samples/labels`<br /><br />Alternatively, provide an absolute file path. |
  | `threshold` | Detection threshold from `0` to `100`.<br /><br />Default: `40` |
</Accordion>

### Metadata Parser

The [**gst-ai-metadata-parser-example**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-metadata-parser-example/main.c) application receives the live video stream input from camera, file, or RTSP source, and passes the stream to the YOLO models for object detection and preview. The overlaid AI model output, including labels and bounding boxes, is displayed on an HDMI display. The extracted metadata is logged to the console and used to count the number of humans in the frame.

The following figure shows the pipeline for metadata parsing. For information about the plugins used in the pipeline flow, see [**Pipeline flow**](#metadata-parser-pipeline-flow).

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-metadata-parser-example`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-metadata-parser-example/main.c)

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - |
  | Config #1 | Yes | Yes | No | No | No | Yes | No |
  | Config #2 | Yes | Yes | No | Yes | No | Yes | No |

  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | `yolox_quantized.tflite` | `yolox.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-metadata-parser-example --config-file=/etc/configs/config-metadata-parser.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config-metadata-parser.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-metadata-parser-example -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-metadata-parser-example --config-file=/path/to/config-metadata-parser.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config-metadata-parser.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config-metadata-parser.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config-metadata-parser.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config-metadata-parser.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config-metadata-parser.json "$XDG_CONFIG_HOME/qimsdk_samples/config-metadata-parser.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#meta-data-parser-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-metadata-parser-example
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-metadata-parser-example application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#metadata-parser-config-json-description) for all fields.

  ```json theme={null}
  {
    "file-path": "<path to video file>",
    "model": "<path to model file>",
    "labels": "<path to label file>",
    "threshold": <integer between 1 and 100>,
    "runtime": "<cpu, gpu, or dsp runtime>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `file-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        File source, LiteRT model, DSP runtime

        ```json theme={null}
        {
          "file-path": "video.mp4",
          "model": "yolox_quantized.tflite",
          "labels": "yolox.json",
          "threshold": 40,
          "runtime": "dsp"
        }
        ```
      </Tab>

      <Tab title="Config #1">
        File source, LiteRT model, CPU runtime

        ```json theme={null}
        {
          "file-path": "video.mp4",
          "model": "yolox_quantized.tflite",
          "labels": "yolox.json",
          "threshold": 40,
          "runtime": "cpu"
        }
        ```

        <Note>
          Config #1 supports only LiteRT models and the CPU runtime.
        </Note>
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  <img alt="Pipeline Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the metadata parser pipeline:

  | Plugin | Description |
  | :- | :- |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | • Captures the live stream from camera.<br />• Uses `tee` to split the stream into two for inferencing and composing. |
  | `filesrc` | • Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing.<br />• Uses `tee` to split the stream into two for inferencing and composing. |
  | `rtspsrc` | • Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction.<br />• Uses `tee` to split the stream into two for inferencing and composing. |
  | `h264parse` | • Parses the H.264 video. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | • Decodes the video. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data:<br />   • Color conversion<br />   • Scaling (up or down)<br />   • Normalization<br />3. Converts the preprocessed video stream to a tensor stream on its source pad. The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | • After the inference runtime receives the tensor stream on its sink pad, it runs the inference.<br />• Produces a tensor stream with the inference results on its source pad. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | **Instance 1 (Object Detection)**:<br />   • Applies a threshold to the chosen number of results.<br />   • Loads the YOLOv8 module.<br />   • Produces video frames with only bounding boxes that can be overlaid on objects.<br />   • Sends these processed frames to the sink pad of `qtivcomposer`.<br /><br />**Instance 2 (Human Counting)**:<br />   • Produces the output in a text format (bounding box coordinates and labels).<br />   • This output is connected to `appsink` plugin where the metadata is read, parsed, and logged.<br />   • The bounding box information is used to count the number of humans in each frame. |
  | [`qtivcomposer`](../plugin-reference/qtivcomposer) | 1. Composes frames with contents from its sink pads.<br />2. Pushes the GStreamer buffers containing these composed frames to its source pad. |
  | [`waylandsink`](../plugin-reference/waylandsink) | 1. `waylandsink` submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |

  #### Config JSON Field Description

  | Field | Values / Descriptions |
  | - | - |
  | `runtime` | Supported hardware runtimes:<br /><br />- `cpu`<br />- `gpu`<br />- `dsp`<br /><br />Default: `dsp` |
  | `file-path` | Path to the input media file.<br /><br />Default media directory: `$HOME/Downloads/qimsdk_samples/media`<br /><br />Alternatively, provide an absolute file path. |
  | `rtsp-ip-port` | RTSP input URL. |
  | `camera` | Camera selection:<br /><br />- `0` — Primary camera<br />- `1` — Secondary camera |
  | `model` | Path to the model file.<br /><br />Default: `yolox_quantized.tflite`<br /><br />Default model directory: `$HOME/Downloads/qimsdk_samples/models`<br /><br />Alternatively, provide an absolute file path. |
  | `labels` | Path to the label file.<br /><br />Default: `yolox.json`<br /><br />Default labels directory: `$HOME/Downloads/qimsdk_samples/labels`<br /><br />Alternatively, provide an absolute file path. |
  | `threshold` | Detection threshold from `0` to `100`.<br /><br />Default: `40` |
</Accordion>

### AI USB Camera

The [**gst-ai-usb-camera-app**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-usb-camera-app/main.c) streams video from a USB webcam connected to the `Qualcomm EVK`. This webcam should be accessible as a `/dev/videoX` device. Additionally, you can perform object detection and preview the results.

You can choose to preview the output on Wayland, or encode to a video file, or live stream through the RTSP.

Alternatively, you can set `enable-object-detection` as `True` to perform object detection.

The following figures show a pipeline, which processes the input from the USB camera to generate various outputs.

For information about the plugins used in this pipeline, see [**Pipeline flow**](#usb-camera-app-pipeline-flow).

<img alt="Pipeline Diagram" />

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-usb-camera-app`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-usb-camera-app/main.c)

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - |
  | Config #1 | No | No | Yes | No | Yes | Yes | Yes |
  | Config #2 | No | No | Yes | No | Yes | Yes | Yes |

  #### Sample Model and Label Files

  | Runtime | Model Files | Label Files |
  | - | - | - |
  | `Qualcomm Neural Processing SDK` | `yolonas.dlc` <br /> `yolov5.dlc` <br /> `yolov8.dlc` | `yolonas.json` <br /> `yolov5.json` <br /> `yolov8.json` |
  | `LiteRT` | `yolov8_det_w8a8.tflite` <br /> `yolonas_quantized.tflite` <br /> `yolov5.tflite` <br /> `yolox_quantized.tflite` | `yolov8.json` <br /> `yolonas.json` <br /> `yolov5.json` <br /> `yolox.json` |
  | `Qualcomm AI Engine direct` | `yolov8_det_quantized.bin` | `yolov8.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-usb-camera-app --config-file=/etc/configs/config-usb-camera-app.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config-usb-camera-app.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-usb-camera-app -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-usb-camera-app --config-file=/path/to/config-usb-camera-app.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config-usb-camera-app.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config-usb-camera-app.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config-usb-camera-app.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config-usb-camera-app.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config-usb-camera-app.json "$XDG_CONFIG_HOME/qimsdk_samples/config-usb-camera-app.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#usb-camera-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-usb-camera-app
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-usb-camera-app application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#usb-camera-app-config-json-description) for all fields.

  ```json theme={null}
  {
    "width": "<Supported USB camera width>",
    "height": "<Supported USB camera Height>",
    "framerate": "<Supported USB camera fps>",
    "video-format": "<Supported USB camera video format like yuy2, mjpeg, nv12>",
    "output": "<output type like waylandsink, filesink or rtspsink>",
    "ip-address": "<Device IP address in case of rtsp streaming>",
    "port": "<Device port num in case of rtsp streaming>",
    "enable-object-detection": "<to enable the object-detection>",
    "ml-framework": "<snpe, or tflite or qnn framework>",
    "yolo-model-type": "<yolov8 or yolonas or yolov5 or yolox>",
    "model": "<Model Path>",
    "labels": "<Label Path>",
    "threshold": "<Post-processing threshold, integer value from 1-100>",
    "runtime": "<dsp, cpu or gpu runtime>",
    "snpe-tensors": "<Model output tensor name>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |
  | `output-file` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |

  <Note>
    For USB camera input, set the `video-format`, `resolution`, and `framerate` parameters in the config file
    to match the camera capabilities.
  </Note>

  <Note>
    You can run the `Yolo-NAS-Quantized.tflite` model using the YOLOv8 value in the `yolo-model-type` field.
  </Note>

  <Note>
    The `snpe-tensors` field applies only to the SNPE runtime. To retrieve the output tensor names for a DLC model, open the model in [**Netron**](https://netron.app/).
  </Note>

  <Note>
    When using DLC models from the AI Hub, the `snpe-tensors` field is optional.
  </Note>

  <Note>
    If the USB camera isn’t detected on the target device, Download the required firmware. See Download PCIe to USB controller firmware.
  </Note>

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        RTSP input, LiteRT, YOLOx model and DSP runtime

        ```json theme={null}
        {
          "width": 1920,
          "height": 1080,
          "framerate": 30,
          "output":"waylandsink",
          "video-format":"yuy2",
          "model":"yolox_quantized.tflite",
          "labels":"yolox.json",
          "output-file":"output.mp4",
          "ip-address":"127.0.0.1",
          "port":"8900",
          "enable-object-detection": "TRUE",
          "ml-framework": "tflite",
          "yolo-model-type": "yolox",
          "threshold": 75,
          "runtime": "dsp"
        }
        ```
      </Tab>

      <Tab title="Config #1">
        RTSP input, LiteRT, YOLOx model and CPU runtime

        ```json theme={null}
        {
          "width": 1920,
          "height": 1080,
          "framerate": 30,
          "output":"waylandsink",
          "video-format":"yuy2",
          "model":"yolox_quantized.tflite",
          "labels":"yolox.json",
          "output-file":"output.mp4",
          "ip-address":"127.0.0.1",
          "port":"8900",
          "enable-object-detection": "TRUE",
          "ml-framework": "tflite",
          "yolo-model-type": "yolox",
          "threshold": 75,
          "runtime": "cpu"
        }
        ```

        <Note>
          Config #1 supports only LiteRT models and the CPU runtime.
        </Note>
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  <img alt="Output Diagram" />

  #### Pipeline flow

  The following table lists the plugins used in AI USB camera pipelines:

  | Pipeline | Description |
  | - | - |
  | Dump the USB camera to a filesink | • USB camera captures the live camera stream.<br /><br />• [`qtivtransform`](../plugin-reference/qtivtransform) transforms the stream data.<br /><br />• `capsfilter` is applied to enforce constraints on the raw video data.<br /><br />• `filesink` is used to dump the data into a file. |
  | Video encoding | • USB camera captures the live camera stream.<br /><br />• [`qtivtransform`](../plugin-reference/qtivtransform) transforms the stream data.<br /><br />• `capsfilter` is applied to enforce constraints on the raw video data.<br /><br />• [`v4l2h264enc`](../plugin-reference/v4l2h264enc) is used to encode the video using the H.264 format.<br /><br />• `h264parse` is used to parse the video.<br /><br />• `mp4mux` is used to multiplex the video into an MP4 container.<br /><br />• `filesink` is used to write the video to a file. |
  | RTSP streaming | • USB camera captures the live camera stream.<br /><br />• [`qtivtransform`](../plugin-reference/qtivtransform) transforms the stream data.<br /><br />• `capsfilter` is applied to enforce constraints on the raw video data.<br /><br />• [`v4l2h264enc`](../plugin-reference/v4l2h264enc) is used to encode the video using the H.264 format.<br /><br />• `h264parse` is used to parse the video.<br /><br />• [`qtirtspbin`](../plugin-reference/qtirtspbin) is used to load the stream to RTSP. |
  | USB camera and object detection on RTSP | • USB camera captures the live camera stream.<br /><br />• `capsfilter` is applied to enforce constraints on the raw video data.<br /><br />• `tee` is used to split the stream for inferencing.<br /><br />• [`qtivtransform`](../plugin-reference/qtivtransform) transforms the stream data.<br /><br />• [`qtimlvconverter`](../plugin-reference/qtimlvconverter) performs preprocessing and converts the video stream to a tensor stream, which is used for inferencing.<br /><br />• [`qtimlsnpe`](../plugin-reference/qtimlsnpe), [`qtimltflite`](../plugin-reference/qtimltflite), or [`qtimlqnn`](../plugin-reference/qtimlqnn) run the inference on the stream.<br /><br />• [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) handles the inference results from any object detection model and produces video frames.<br /><br />• [`qtivcomposer`](../plugin-reference/qtivcomposer) composes the video frames and shares them with [`qtirtspbin`](../plugin-reference/qtirtspbin).<br /><br />• [`qtirtspbin`](../plugin-reference/qtirtspbin) submits the composed video stream to Weston, which renders it on the local display. |
  | USB camera and object detection on wayland | • USB camera captures the live camera stream.<br /><br />• `capsfilter` is applied to enforce constraints on the raw video data.<br /><br />• `tee` is used to split the stream for inferencing.<br /><br />• [`qtivtransform`](../plugin-reference/qtivtransform) transforms the stream data.<br /><br />• [`qtimlvconverter`](../plugin-reference/qtimlvconverter) performs preprocessing and converts the video stream to a tensor stream, which is used for inferencing.<br /><br />• [`qtimlsnpe`](../plugin-reference/qtimlsnpe), [`qtimltflite`](../plugin-reference/qtimltflite), or [`qtimlqnn`](../plugin-reference/qtimlqnn) run the inference on the stream.<br /><br />• [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) handles the inference results from any object detection model and produces video frames.<br /><br />• [`qtivcomposer`](../plugin-reference/qtivcomposer) composes the video frames and shares them with [`waylandsink`](../plugin-reference/waylandsink).<br /><br />• [`waylandsink`](../plugin-reference/waylandsink) submits the composed video stream to Weston, which renders it on the local display. |
  | Object detection using USB camera and file encode | • USB camera captures the live camera stream.<br /><br />• `capsfilter` is applied to enforce constraints on the raw video data.<br /><br />• `tee` is used to split the stream for inferencing.<br /><br />• [`qtivtransform`](../plugin-reference/qtivtransform) transforms the stream data.<br /><br />• [`qtimlvconverter`](../plugin-reference/qtimlvconverter) performs preprocessing and converts the video stream to a tensor stream, which is used for inferencing.<br /><br />• [`qtimlsnpe`](../plugin-reference/qtimlsnpe), [`qtimltflite`](../plugin-reference/qtimltflite), or [`qtimlqnn`](../plugin-reference/qtimlqnn) run the inference on the stream.<br /><br />• [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) handles the inference results from any object detection model and produces video frames.<br /><br />• [`qtivcomposer`](../plugin-reference/qtivcomposer) composes the video frames and shares them with `filesink`.<br /><br />• `filesink` writes the composed video stream to a file. |

  #### Config JSON field description

  | Field | Values / Descriptions |
  | - | - |
  | `width` | USB camera resolution width. |
  | `height` | USB camera resolution height. |
  | `framerate` | USB camera frame rate. |
  | `video-format` | USB camera video format.<br /><br />Supported formats:<br /><br />- `nv12`<br />- `yuy2`<br />- `mjpeg` |
  | `output` | Supported output sinks:<br /><br />- `waylandsink` — Display output<br />- `filesink` — MP4 file output<br />- `rtspsink` — RTSP stream output |
  | `output-file` | Output filename or path.<br /><br />Default: `$HOME/Downloads/qimsdk_samples/media/output.mp4`<br /><br />Alternatively, provide an absolute file path. |
  | `ip-address` | RTSP output server IP address.<br /><br />Default: `127.0.0.1` |
  | `port` | RTSP output server port.<br /><br />Default: `8900` |
  | `enable-object-detection` | Enables or disables object detection.<br /><br />Supported values: `TRUE` or `FALSE` |
  | `yolo-model-type` | YOLO model version:<br /><br />- `yolov5`<br />- `yolov8`<br />- `yolox`<br />- `yolonas`<br /><br />YOLOv7 TFLite models use `yolov8` as the `yolo-model-type`. |
  | `ml-framework` | Supported ML frameworks:<br /><br />- `snpe` — Qualcomm Neural Processing SDK<br />- `tflite` — LiteRT<br />- `qnn` — Qualcomm AI Engine Direct<br /><br />Default: `snpe` |
  | `model` | Optional model path that overrides the default model path.<br /><br />Default model file:<br />`yolox_quantized.tflite`<br /><br />Other supported models:<br /><br />- YOLOv5 SNPE DLC: `yolov5.dlc`<br />- YOLOv8 SNPE DLC: `yolov8.dlc`<br />- YOLO NAS SNPE DLC: `yolonas.dlc`<br />- YOLOv5 TFLite: `yolov5.tflite`<br />- YOLOv8 TFLite: `yolov8_det_quantized.tflite`<br />- YOLO NAS TFLite: `yolonas_quantized.tflite`<br />- YOLOv7 TFLite: `yolov7_quantized.tflite`<br />- YOLOX TFLite: `yolox_quantized.tflite`<br />- YOLOv8 QNN: `yolov8_det_quantized.bin`<br /><br />Default model directory: `$HOME/Downloads/qimsdk_samples/models`<br /><br />Alternatively, provide an absolute file path. |
  | `labels` | Optional labels path that overrides the default labels path.<br /><br />Default label file:<br />`yolox.json`<br /><br />Other label files:<br /><br />- YOLOv5: `yolov5.json`<br />- YOLOv8: `yolov8.json`<br />- YOLOX: `yolox.json`<br />- YOLO NAS: `yolonas.json`<br />- YOLOv7: `yolov7.json`<br /><br />Default labels directory: `$HOME/Downloads/qimsdk_samples/labels`<br /><br />Alternatively, provide an absolute file path. |
  | `threshold` | Detection threshold from `0` to `100`.<br /><br />Default: `40` |
  | `runtime` | Supported hardware runtimes:<br /><br />- `cpu`<br />- `gpu`<br />- `dsp`<br /><br />Default: `dsp` |
  | `snpe-tensors` | JSON array specifying the output tensors for the SNPE model. |
</Accordion>

***

## AI Audio Applications

### Audio Classification

The [**gst-ai-audio-classification**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-audio-classification/main.c) application shows audio classification using input from either a file source or a microphone. It displays both the classification results and a video preview.

The following figure shows the pipeline, which gets the input from a file or a microphone, preprocesses it, and runs inferences on AI hardware. The results are displayed on the screen.

For information about the plugins used in the pipeline flow, see [**Pipeline flow**](#audio-classification-pipeline-flow).

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-audio-classification`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-audio-classification/main.c)

<Accordion title="Try me">
  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | `yamnet.tflite` | `yamnet.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-audio-classification --config-file=/etc/configs/config-audio-classification.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config-audio-classification.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-audio-classification -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-audio-classification --config-file=/path/to/config-audio-classification.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config-audio-classification.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config-audio-classification.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config-audio-classification.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config-audio-classification.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config-audio-classification.json "$XDG_CONFIG_HOME/qimsdk_samples/config-audio-classification.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#audio-classification-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-audio-classification
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-audio-classification application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#audio-classification-config-json-description) for all fields.

  ```json theme={null}
  {
    "file-path": "<path to video file>",
    "model": "<path to model file>",
    "labels": "<path to label file>",
    "threshold": <integer between 1 and 100>,
    "runtime": "<cpu or gpu>",
    "codec": "<mp3 or flac>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `file-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        File source, LiteRT model, CPU runtime

        ```json theme={null}
        {
          "file-path": "video-mp3.mp4",
          "model": "yamnet.tflite",
          "labels": "yamnet.json",
          "runtime": "cpu",
          "threshold": 20,
          "codec": "mp3"
        }
        ```
      </Tab>

      <Tab title="Config #1">
        File source, LiteRT model, CPU runtime

        ```json theme={null}
        {
          "file-path": "video-mp3.mp4",
          "model": "yamnet.tflite",
          "labels": "yamnet.json",
          "runtime": "cpu",
          "threshold": 20,
          "codec": "mp3"
        }
        ```

        <Note>
          Config #1 supports only LiteRT models and the CPU runtime.
        </Note>
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  The output video and classified audio are played on the screen.

  <img alt="Output Image Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the audio classification pipeline:

  | Plugin | Description |
  | :- | :- |
  | `filesrc` | • Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing.<br />• Uses `tee` to split the stream for processing. |
  | `h264parse` | • Parses the H.264 video. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | • Decodes the video bitstream into raw frames. |
  | `mpegaudioparse` / `flacparse` | • Parses the audio bitstream (MP3 or FLAC) to ensure downstream elements can handle the payload. |
  | `mpg123audiodec` / `flacdec` | • Decodes the compressed audio (MP3 or FLAC) into raw audio buffers. |
  | `audioconvert` | • Converts raw audio buffers between various possible formats to ensure compatibility. |
  | `audioresample` | • Resamples the audio buffers to different sample rates as required by the model. |
  | [`pulsesrc`](../plugin-reference/pulsesrc) | • Reads the live audio stream from the microphone. |
  | `audiobuffersplit` | • Splits the incoming audio buffers into equal-sized chunks for consistent processing. |
  | qtimlaconverter\` | 1. Receives the audio stream on its sink pad.<br />2. Performs preprocessing on the audio stream data.<br />3. Converts the stream to a tensor stream for inferencing in the later stages of the pipeline. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | 1. Receives the tensor stream on its sink pad.<br />2. Performs inferencing using the **YAMNet** model.<br />3. Produces a tensor stream with the results on its source pad. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | • Uses the `yamnet` module to handle audio classification inference results:<br />   • Applies a threshold to the chosen number of results.<br />   • Creates a text overlay for the identified audio classes. |
  | [`qtivcomposer`](../plugin-reference/qtivcomposer) | • Combines the text overlay for classification results and the video preview into a single composed frame. |
  | [`waylandsink`](../plugin-reference/waylandsink) | 1. Submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |

  #### Config JSON Field Description

  | Field | Values / Descriptions |
  | - | - |
  | `file-path` | Path to the input audio file.<br /><br />If not provided, `pulsesrc` is selected as the input source. |
  | `model` | Path to the model file.<br /><br />Default: `$HOME/Downloads/qimsdk_samples/models/yamnet.tflite` |
  | `labels` | Path to the labels file.<br /><br />Default: `$HOME/Downloads/qimsdk_samples/labels/yamnet.json` |
  | `threshold` | Classification threshold from `0` to `100`.<br /><br />Default: `40` |
  | `runtime` | Supported hardware runtimes:<br /><br />- `cpu`<br />- `gpu`<br /><br />Default: `cpu` |
  | `codec` | Audio codec for the input file:<br /><br />- `mp3`<br />- `flac`<br /><br />Default: `mp3` |
</Accordion>

## AI Multi-Model Applications

### Face Recognition

The [**gst-ai-face-recognition**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-face-recognition/main.c) application collects the live video input from a camera or an RTSP stream and shares this input for face detection, facial landmarking, and face recognition. It uses the `face_det_quantized` models for face detection, `facemap_3dmm_quantized` model for facial landmarking, and `face_attrib_net_quantized` model for face recognition labels.

The result is a preview of the overlaid AI model on the HDMI display.

<Note>
  This application isn't supported in `Config #1` of `Qualcomm Linux` because CPU runtime is not supported.
</Note>

The following figure shows the pipeline, which receives the input, preprocesses it, runs inferences on AI hardware, and displays the results on the screen.

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-face-recognition`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-face-recognition/main.c)

For information about the plugins used in the pipeline flow, see [**Pipeline flow**](#face-recognition-pipeline-flow).

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - |
  | Config #2 | No | Yes | No | Yes | No | Yes | No |

  <Note>
    `camera` use-case is not supported on `IQ-2390` and `CQ-2390`.
  </Note>

  #### Sample Model and Label Files

  | Runtime | Model files | Label files |
  | - | - | - |
  | Qualcomm AI Engine Direct & LiteRT | `face_det_lite_quantized.tflite`, `facemap_3dmm_quantized.tflite`, `face_attrib_net_quantized.tflite`, `face_det_lite_quantized.bin`, `facemap_3dmm_quantized.bin`, `face_attrib_net_quantized.bin` | `face_detection.json`, `face_recognition_settings.json`, `face_recognition.json`, `facemap_3dmm_settings.json` |

  #### Register a face for facial recognition

  Before running the gst-ai-face-recognition application, you can register a face for secure verification and authentication:

  <Steps>
    <Step>
      Ensure that you complete the [**Prerequisites**](#prerequisites).
    </Step>

    <Step>
      To register a face, use the following gst-pipeline on the target device shell:

      ```bash theme={null}
      gst-pipeline-app -e \
      qtimlvconverter name=stage_01_preproc mode=image-batch-non-cumulative \
      qtimltflite name=stage_01_inference model=${HOME}/Downloads/qimsdk_samples/models/face_det_lite_quantized.tflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
      external-delegate-options="QNNExternalDelegate,backend_type=htp;" \
      qtimlpostprocess name=stage_01_postproc settings="{\"confidence\": 40.0}" results=4 module=qfd labels=${HOME}/Downloads/qimsdk_samples/labels/face_detection.json \
      qtimlvconverter name=stage_03_preproc mode=roi-batch-cumulative \
      qtimltflite name=stage_03_inference model=${HOME}/Downloads/qimsdk_samples/models/face_attrib_net_quantized.tflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
      external-delegate-options="QNNExternalDelegate,backend_type=htp;" \
      qticamsrc video_0::type=video name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080 ! queue ! waylandsink fullscreen=true sync=false \
      camsrc.image_1 ! video/x-raw,width=1920,height=1080 ! qtivtransform ! video/x-raw,format=NV12 ! tee name=t_split_1 \
      t_split_1. ! queue ! metamux_1. \
      t_split_1. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! stage_01_inference. stage_01_inference. ! queue ! \
      stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! metamux_1. \
      qtimetamux name=metamux_1 ! queue ! tee name=t_split_3 \
      t_split_3. ! queue ! stage_03_preproc. stage_03_preproc. ! queue ! stage_03_inference. stage_03_inference. ! queue ! \
      multifilesink location=${HOME}/Downloads/qimsdk_samples/data/tensor_%d.bin sync=true async=false enable-last-sample=false
      ```
    </Step>

    <Step>
      To prepare for capturing a facial image, do the following:

      <Steps>
        <Step>
          Select the following options from the list. Choose the number corresponding to the option:

          * `PLAYING`: Move the pipeline to the Playing state.
          * `Plugin Mode` ➔ `camsrc` ➔ `capture-image`: Capture the image using a camera source.
        </Step>

        <Step>
          Using the live preview on the display, face the camera and ensure that the camera is pointed straight and there is only one person in the frame.
        </Step>

        <Step>
          In the terminal, enter 1 for the following values:

          * `GstImageCaptureMode` for `arg0`.
          * `guint` for `arg1`.
        </Step>
      </Steps>
    </Step>

    <Step>
      To capture all the sides of your face, select `capture-image` do the following for each side:

      <Steps>
        <Step>
          Left and right: Turn your head left by 40° while keeping the landmarks visible, then repeat steps 3 and 4. Turn your head right (by 40°) and repeat.
        </Step>

        <Step>
          Up and down: Raise your head by 30° while keeping the landmarks visible, then repeat steps 3 and 4. Lower your head (by 30°) and repeat.
        </Step>
      </Steps>
    </Step>

    <Step>
      To stop the pipeline, use `(b)Back` and `(q)Quit`.<br />
      After running the pipeline, five individual tensor bins are created (`tensor_0.bin` to `tensor_4.bin`) with facial properties recorded for each side of the face.
    </Step>

    <Step>
      On the target device, go to `${HOME}/Downloads/qimsdk_samples/data/`, find the tensor bins.

      ```bash theme={null}
      cd ${HOME}/Downloads/qimsdk_samples/data
      ```

      To merge the tensor bins with all the facial properties into a cohesive image, download and run the `facedb.py` script in the same directory as the tensor bins on the target device.

      <Steps>
        <Step>
          Download the `facedb.py` script:

          ```bash theme={null}
          curl -L -O https://raw.githubusercontent.com/quic/sample-apps-for-qualcomm-linux/refs/heads/main/qualcomm-linux/scripts/facedb.py
          ```
        </Step>

        <Step>
          Run the script. Note that `<Name of the person>` is case and style sensitive. Ensure that you use the same name consistently.

          ```bash theme={null}
          python3 ./facedb.py "<Name of the person>" 512 32 tensor_0.bin tensor_1.bin tensor_2.bin tensor_3.bin tensor_4.bin
          ```

          A `face.bin` binary is created.
        </Step>
      </Steps>
    </Step>

    <Step>
      Rename the generated` face.bin` file to `face{N}.bin`, replacing `{N}` with the next serial number—for example, `face0.bin`, `face1.bin` and so on.
    </Step>

    <Step>
      Use the following reference `face_recognition.json` file as an example for registering a new person in the database. The sample configuration demonstrates face registration for two individuals.

      ```json theme={null}
      [
        {"id": 0, "color": "0x00FF00FF", "label": "<Name of Person>"},
        {"id": 1, "color": "0xFFFF00FF", "label": "<Name of Person>"}
      ]
      ```

      <Note>
        Update the ID field according to the number in the list. If more faces are registered, add the structure in a new line within `face_recognition.json`.
      </Note>
    </Step>

    <Step>
      Use the following `face_recognition_settings.json` file as a reference when specifying the registered face database files:

      ```json theme={null}
      {
        "confidence": 51.0,
          "databases":[
            {"id":  0, "database": "/root/Downloads/qimsdk_samples/data/face0.bin"},
            {"id":  1, "database": "/root/Downloads/qimsdk_samples/data/face1.bin"}
          ]
      }
      ```
    </Step>
  </Steps>

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-face-recognition --config-file=/etc/configs/config-face-recognition.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config-face-recognition.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-face-recognition -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-face-recognition --config-file=/path/to/config-face-recognition.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config-face-recognition.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config-face-recognition.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config-face-recognition.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config-face-recognition.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config-face-recognition.json "$XDG_CONFIG_HOME/qimsdk_samples/config-face-recognition.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#face-recognition-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-face-recognition
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-face-recognition application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#face-recognition-config-json-description) for all fields.

  ```json theme={null}
  {
    "ml-framework": "<qnn or tflite>",
    "face-detection-model": "<path for face detection model file>",
    "face-landmark-model":"<path for face landmark model file>",
    "face-recognition-model":"<path for face recognition model file>",
    "face-detection-labels":"<path for face detection labels file>",
    "face-recognition-labels":"<path for face recognition labels file>",
    "face-recognition-settings": "<Path of face recognition settings>",
    "facemap-3dmm-settings": "<Path of facemap-3dmm settings>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `face-detection-model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `face-landmark-model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `face-recognition-model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `face-detection-labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |
  | `face-recognition-labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |
  | `face-recognition-settings` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |
  | `facemap-3dmm-settings` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        Camera source, LiteRT, and DSP runtime

        ```json theme={null}
        {
          "ml-framework":"tflite",
          "face-detection-model":"face_det_lite_quantized.tflite",
          "face-landmark-model":"facemap_3dmm_quantized.tflite",
          "face-recognition-model":"face_attrib_net_quantized.tflite",
          "face-detection-labels": "face_detection.json",
          "face-recognition-labels": "face_recognition.json",
          "face-recognition-settings": "face_recognition_settings.json",
          "facemap-3dmm-settings": "facemap_3dmm_settings.json"
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected output

  <img alt="Pipeline Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the face recognition pipeline:

  | Plugin | Description |
  | :- | :- |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | • Captures the live stream from camera.<br />• Uses `tee` to split the stream for inferencing. |
  | `filesrc` | • Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing.<br />• Uses `tee` to split the stream for inferencing. |
  | `rtspsrc` | • Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction.<br />• Uses `tee` to split the stream for inferencing. |
  | `h264parse` | • Parses the H.264 video bitstream. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | • Hardware-decodes H.264 video to raw frames. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data:<br />   • Color conversion<br />   • Scaling (up or down)<br />   • Normalization<br />3. Converts the preprocessed video stream to a tensor stream.<br />The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | 1. After the inference runtime receives the tensor stream on its sink pad, it runs the inference.<br />2. Produces a tensor stream with the inference results on its source pad. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (Pose-estimation) | • Uses the `lite-3dmm` module to perform facial pose recognition. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (Classification) | • Uses the `qfr` module to receive the stream from `qtimetamux` and classifies the face. |
  | [`qtimetamux`](../plugin-reference/qtimetamux) | 1. Receives the output of the face detection models from `qtimlpostprocess` and multiplexes it.<br />2. Receives the output of facial pose from `qtimlpostprocess` and multiplexes it. |
  | `tee` | • Splits the stream for inferencing. |
  | [`qtivoverlay`](../plugin-reference/qtivoverlay) | 1. Receives the multiplexed stream.<br />2. Overlays the bounding boxes on the stream. |
  | [`waylandsink`](../plugin-reference/waylandsink) | 1. Submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |

  #### Config JSON Field Description

  | Field | Values / Description |
  | - | - |
  | `camera` | Select the camera:<br /><br />- `0` – Primary camera<br />- `1` – Secondary camera |
  | `rtsp-ip-port` | Provide the RTSP input URI.<br /><br />Example: `rtsp://192.168.1.110:8554/live.mkv` |
  | `ml-framework` | Select the ML framework:<br /><br />- `tflite` – Execute the model using TFLite. **Default**<br />- `qnn` – Execute the model using QNN |
  | `tflite-face-detection-model` | Optional path to the TFLite face detection model. Overrides the default path.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />An absolute file path can also be provided. |
  | `tflite-face-landmark-model` | Optional path to the TFLite face landmark model. Overrides the default path.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />An absolute file path can also be provided. |
  | `tflite-face-recognition-model` | Optional path to the TFLite face recognition model. Overrides the default path.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />An absolute file path can also be provided. |
  | `qnn-face-detection-model` | Optional path to the QNN face detection model. Overrides the default path.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />An absolute file path can also be provided. |
  | `qnn-face-landmark-model` | Optional path to the QNN face landmark model. Overrides the default path.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />An absolute file path can also be provided. |
  | `qnn-face-recognition-model` | Optional path to the QNN face recognition model. Overrides the default path.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />An absolute file path can also be provided. |
  | `face-detection-labels` | Optional path to the face detection labels file. Overrides the default path.<br /><br />Label files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />An absolute file path can also be provided. |
  | `face-recognition-labels` | Optional path to the face recognition labels file. Overrides the default path.<br /><br />Label files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />An absolute file path can also be provided. |
  | `facemap-3dmm-settings` | Optional path to the face map settings file. Overrides the default path.<br /><br />Settings files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />An absolute file path can also be provided. |
  | `face-recognition-settings` | Optional path to the face recognition settings file. Overrides the default path.<br /><br />Settings files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />An absolute file path can also be provided. |
</Accordion>

***

### Daisychain Detection + Classification

The [**gst-ai-daisychain-detection-classification**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-daisychain-detection-classification/main.c) application allows you to perform cascaded object detection and classification with a camera, file source, or RTSP stream. The use case involves detecting objects and classifying the detected objects.

The following figures show the pipeline workflow, which captures the video stream from the source, preprocesses it, and runs inferences using AI hardware. The results are either displayed on the screen, saved as an encoded MP4 file, or streamed over the RTSP server. For information about the plugins used in this pipeline, see [**Pipeline flow**](#daisychain-classification-pipeline-flow).

<img alt="Pipeline Diagram" />

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-daisychain-detection-classification`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-daisychain-detection-classification/main.c)

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - |
  | Config #1 | Yes | Yes | Yes | No | Yes | Yes | Yes |
  | Config #2 | Yes | Yes | Yes | Yes | Yes | Yes | Yes |

  #### Sample Model and Label Files

  | Runtime | Model files | Label files |
  | - | - | - |
  | LiteRT | detection: `yolox_quantized.tflite`,<br />classification: `inception_v3_quantized.tflite` | `yolox.json`, `classification.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-daisychain-detection-classification --config-file=/etc/configs/config_daisychain_detection_classification.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config_daisychain_detection_classification.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-daisychain-detection-classification -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-daisychain-detection-classification --config-file=/path/to/config_daisychain_detection_classification.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config_daisychain_detection_classification.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config_daisychain_detection_classification.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config_daisychain_detection_classification.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config_daisychain_detection_classification.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config_daisychain_detection_classification.json "$XDG_CONFIG_HOME/qimsdk_samples/config_daisychain_detection_classification.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#daisychain-classification-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-daisychain-detection-classification
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-daisychain-detection-classification application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#daisychain-classification-config-json-description) for all fields.

  ```json theme={null}
  {
    "input-file": "<input-video-path>",
    "detection-model": "<path-to-detection model>",
    "detection-labels": "<path-to-detection-labels>",
    "classification-model": "<path-to-classification-model>",
    "classification-labels": "<path-to-classification-labels>",
    "detection-runtime": "<Can be dsp or cpu or gpu>",
    "classification-runtime": "<Can be dsp or cpu or gpu>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `input-file` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `detection-model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `classification-model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `detection-labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |
  | `classification-labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |
  | `output-file` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |

  <Note>
    For USB camera input, set the `video-format`, `resolution`, and `framerate` parameters in the config file
    to match the camera capabilities, see [Configure USB camera](https://dragonwingdocs.qualcomm.com/System/Interfaces/usb#configure-usb-camera).
  </Note>

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        File source, LiteRT model, DSP runtime

        ```json theme={null}
        {
          "input-file": "video.mp4",
          "detection-model": "yolox_quantized.tflite",
          "detection-labels": "yolox.json",
          "classification-model": "inception_v3_quantized.tflite",
          "classification-labels": "classification.json",
          "detection-runtime": "dsp",
          "classification-runtime": "dsp"
        }
        ```
      </Tab>

      <Tab title="Config #1">
        File source, LiteRT model, CPU runtime

        ```json theme={null}
        {
          "input-file": "video.mp4",
          "detection-model": "yolox_quantized.tflite",
          "detection-labels": "yolox.json",
          "classification-model": "inception_v3_quantized.tflite",
          "classification-labels": "classification.json",
          "detection-runtime": "cpu",
          "classification-runtime": "cpu"
        }
        ```

        <Note>
          Config #1 supports only LiteRT models and the CPU runtime.
        </Note>
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  The cropped video frame is overlaid on the frame and displayed on a local device.

  <img alt="Pipeline Diagram" />

  <img alt="Pipeline Diagram" />

  <Note>
    The classification models trained on the Imagenet dataset don't contain the person class.
  </Note>

  #### Pipeline Flow

  The following table lists the plugins used in the daisychain detection and classification pipeline:

  | Plugin | Description |
  | :- | :- |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | • Captures the live stream from camera.<br />• Uses `tee` to split the stream for inferencing. |
  | `filesrc` | • Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing.<br />• Uses `tee` to split the stream for inferencing. |
  | `rtspsrc` | • Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction.<br />• Uses `tee` to split the stream for inferencing. |
  | `v4l2src` | • Captures the live stream from USB camera.<br />• Uses `tee` to split the stream for inferencing. |
  | `h264parse` | • Parses the H.264 video bitstream. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | • Hardware-decodes H.264 video to raw frames. |
  | [`qtimetamux`](../plugin-reference/qtimetamux) | • Multiplexes the stream. |
  | [`qtivsplit`](../plugin-reference/qtivsplit) | • Crops the full frame into smaller frames based on the detected bounding boxes (maximum 4). |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data:<br />   • Color conversion<br />   • Scaling (up or down)<br />   • Normalization<br />3. Converts the preprocessed video stream to a tensor stream.<br />The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | 1. After the inference runtime receives the tensor stream on its sink pad, it runs the inference.<br />2. Produces a tensor stream with the inference results on its source pad. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (Detection) | • Handles inference results from any object detection model:<br />   • Applies a threshold to the chosen number of results.<br />   • Loads the YOLOv8 module.<br />   • Produces video frames with only bounding boxes that can be overlaid on objects.<br />   • Produces video frames with only bounding boxes that can be cropped. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (Classification) | • Processes results on the cropped frame:<br />   • Applies the threshold to the chosen number of results on the cropped frame.<br />   • Loads the MobileNet-softmax module.<br />   • Produces results as video frames with classification labels.<br />   • Sends them to the sink pad of `qtivcomposer`. |
  | [`qtivcomposer`](../plugin-reference/qtivcomposer) | 1. Composes frames with contents from its sink pads.<br />2. Pushes the GStreamer buffers containing these composed frames to its source pad. |
  | `filesink` | • Receives the video stream on its sink pad and saves it as an H.264-encoded MP4 file. |
  | [`qtirtspbin`](../plugin-reference/qtirtspbin) | 1. Serves as a network sink.<br />2. Transmits UDP packets to the network. |
  | [`waylandsink`](../plugin-reference/waylandsink) | 1. Submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |

  #### Config JSON Field Description

  | Field | Values / Description |
  | - | - |
  | `input-file` | Path to the input media file.<br /><br />The media file should be placed in:<br />`$HOME/Downloads/qimsdk_samples/media`<br /><br />Alternatively, provide an absolute file path. |
  | `rtsp-ip-port` | Provide the RTSP input URI.<br /><br />Example: `rtsp://192.168.1.110:8554/live.mkv`<br /><br />If neither `input-file` nor `rtsp-ip-port` is provided, camera input is selected. |
  | `detection-model` | Path to the YOLOX detection model file.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />Alternatively, provide an absolute file path. |
  | `detection-labels` | Path to the YOLOX labels file.<br /><br />Label files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />Alternatively, provide an absolute file path. |
  | `classification-model` | Path to the classification model file.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />Alternatively, provide an absolute file path. |
  | `classification-labels` | Path to the classification labels file.<br /><br />Label files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />Alternatively, provide an absolute file path. |
  | `enable-usb-camera` | Enables or disables the USB camera input.<br /><br />Accepted values:<br />- `TRUE`<br />- `FALSE` |
  | `width` | USB camera resolution width. |
  | `height` | USB camera resolution height. |
  | `framerate` | USB camera frame rate. |
  | `video-format` | USB video format.<br /><br />Supported values:<br />- `nv12`<br />- `yuy2`<br />- `mjpeg` |
  | `output-type` | Select the output type.<br /><br />Supported values:<br />- `waylandsink`<br />- `filesink`<br />- `rtspsink` |
  | `output-file` | Path to the output file.<br /><br />Output files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/media`<br /><br />Alternatively, provide an absolute file path. |
  | `output-ip-address` | IP address for the RTSP output.<br /><br />Example: `127.0.0.1`<br />Default IP address: `127.0.0.1` |
  | `port` | Port for the RTSP output.<br /><br />Example: `8900`<br />Default port: `8900` |
  | `classification-runtime` | Runtime used for classification.<br /><br />Supported values:<br />- `cpu`<br />- `gpu`<br />- `dsp` |
  | `detection-runtime` | Runtime used for detection.<br /><br />Supported values:<br />- `cpu`<br />- `gpu`<br />- `dsp` |
</Accordion>

### Daisychain Detection + Pose

The [**gst-ai-daisychain-detection-pose**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-daisychain-detection-pose/main.c) application allows you to perform cascaded object detection and pose detection with a camera, file source, or an RTSP stream. The use cases involve detecting objects and estimating the body poses of the subject in an image or a video.

The following figure show the application workflow, which receives the source, postprocesses it, and runs inferences on AI hardware. The results are either displayed on the screen, saved as an encoded MP4 file, or streamed over the RTSP server.

For information about the plugins used in the pipeline flow, see [**Pipeline flow**](#daisychain-pose-pipeline-flow).

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-daisychain-detection-pose`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-daisychain-detection-pose/main.c)

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - |
  | Config #1 | Yes | Yes | Yes | No | Yes | Yes | Yes |
  | Config #2 | Yes | Yes | Yes | Yes | Yes | Yes | Yes |

  #### Sample Model and Label Files

  | Runtime | Model files | Label files |
  | - | - | - |
  | LiteRT | detection: `yolox_quantized.tflite`,<br />pose: `hrnet_pose_quantized.tflite` | detection: `yolox.json`, pose: `hrnet_pose.json`, pose: `hrnet_settings.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-daisychain-detection-pose --config-file=/etc/configs/config-daisychain-detection-pose.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config-daisychain-detection-pose.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-daisychain-detection-pose -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-daisychain-detection-pose --config-file=/path/to/config-daisychain-detection-pose.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config-daisychain-detection-pose.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config-daisychain-detection-pose.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config-daisychain-detection-pose.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config-daisychain-detection-pose.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config-daisychain-detection-pose.json "$XDG_CONFIG_HOME/qimsdk_samples/config-daisychain-detection-pose.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#daisychain-pose-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-daisychain-detection-pose
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-daisychain-detection-pose application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#daisychain-detection-config-json-description) for all fields.

  ```json theme={null}
  {
    "input-file": "<input-video-path>",
    "rtsp-ip-port": "<RTSP-IP-port-address>",
    "detection-model": "<path-to-detection model>",
    "detection-labels": "<path-to-detection-labels>",
    "detection-runtime": "<It can take dsp, gpu or cpu as runtime. Runs detection model on given runtime>",
    "pose-runtime": "<It can take dsp, gpu or cpu as runtime. Runs pose model on given runtime>",
    "output-file": "<output-file-path>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `input-file` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `detection-model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `pose-model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `detection-labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |
  | `pose-labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |
  | `pose-settings-path` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |
  | `output-file` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |

  <Note>
    For `QCS6490`, if `file-path` and `rtsp-ip-port` are not present in the configuration file, then the camera input is selected.
  </Note>

  <Note>
    For USB camera input, set the `video-format`, `resolution`, and `framerate` parameters in the config file
    to match the camera capabilities, see [Configure USB camera](https://dragonwingdocs.qualcomm.com/System/Interfaces/usb#configure-usb-camera).
  </Note>

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        File source, LiteRT model, DSP runtime

        ```json theme={null}
        {
         "input-file": "video.mp4",
         "pose-runtime":"dsp",
         "detection-runtime":"dsp",
         "detection-model": "yolox_quantized.tflite",
         "detection-labels": "yolox.json",
         "pose-model": "hrnet_pose_quantized.tflite",
         "pose-labels": "hrnet_pose.json",
         "pose-settings-path":"hrnet_settings.json"
        }
        ```
      </Tab>

      <Tab title="Config #1">
        File source, LiteRT model, CPU runtime

        ```json theme={null}
        {
         "input-file": "video.mp4",
         "pose-runtime":"cpu",
         "detection-runtime":"cpu",
         "detection-model": "yolox_quantized.tflite",
         "detection-labels": "yolox.json",
         "pose-model": "hrnet_pose_quantized.tflite",
         "pose-labels": "hrnet_pose.json",
         "pose-settings-path":"hrnet_settings.json"
        }
        ```

        <Note>
          Config #1 supports only LiteRT models and the CPU runtime.
        </Note>
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  The cropped video frame is overlaid on the frame and displayed on a local device.

  <img alt="Pipeline Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the daisychain detection and pose pipeline:

  | Plugin | Description |
  | :- | :- |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | • Captures the live stream from camera.<br />• Uses `tee` to split the stream for inferencing. |
  | `filesrc` | • Captures the video stream using `filesrc`, followed by `qtdemux` for demultiplexing.<br />• Uses `tee` to split the stream for inferencing. |
  | `rtspsrc` | • Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction.<br />• Uses `tee` to split the stream for inferencing. |
  | `v4l2src` | • Captures the live stream from USB camera.<br />• Uses `tee` to split the stream for inferencing. |
  | `h264parse` | • Parses the H.264 video bitstream. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | • Hardware-decodes H.264 video to raw frames. |
  | [`qtimetamux`](../plugin-reference/qtimetamux) | • Multiplexes the stream. |
  | [`qtivsplit`](../plugin-reference/qtivsplit) | • Crops the full frame into smaller frames based on the detected bounding boxes (maximum 4). |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data:<br />   • Color conversion<br />   • Scaling (up or down)<br />   • Normalization<br />3. Converts the preprocessed video stream to a tensor stream.<br />The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | 1. After the inference runtime receives the tensor stream on its sink pad, it runs the inference.<br />2. Produces a tensor stream with the inference results on its source pad. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (Detection) | • Handles inference results from any object detection model:<br />   • Applies a threshold to the chosen number of results.<br />   • Loads the YOLOv8 module.<br />   • Produces video frames with only bounding boxes that can be overlaid on objects.<br />   • Produces video frames with only bounding boxes that can be cropped. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (Pose) | • Applies a threshold to the chosen number of results.<br />• Loads corresponding modules for various pose detection models.<br />In this specific use case, `qtimlpostprocess` does the following:<br />1. Loads the HRNet module.<br />2. Produces results in the form of video frames with drawn poses.<br />3. Sends the results to the sink pad of `qtivcomposer` for further processing or display. |
  | [`qtivcomposer`](../plugin-reference/qtivcomposer) | 1. Composes frames with contents from its sink pads.<br />2. Pushes the GStreamer buffers containing these composed frames to its source pad. |
  | `filesink` | • Receives the video stream on sink pad and saves it as an H.264-encoded MP4 file. |
  | [`qtirtspbin`](../plugin-reference/qtirtspbin) | 1. Serves as a network sink.<br />2. Transmits UDP packets to the network. |
  | [`waylandsink`](../plugin-reference/waylandsink) | 1. Submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |

  #### Config JSON Field Description

  | Field | Values / Description |
  | - | - |
  | `input-file` | Path to the input media file.<br /><br />The media file should be placed in:<br />`$HOME/Downloads/qimsdk_samples/media`<br /><br />Alternatively, provide an absolute file path. |
  | `rtsp-ip-port` | Provide the RTSP input URI.<br /><br />Example: `rtsp://192.168.1.110:8554/live.mkv`<br /><br />If neither `input-file` nor `rtsp-ip-port` is provided, camera input is selected. |
  | `detection-model` | Path to the YOLOX detection model file.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />Alternatively, provide an absolute file path. |
  | `detection-labels` | Path to the YOLOX labels file.<br /><br />Label files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />Alternatively, provide an absolute file path. |
  | `pose-model` | Path to the pose detection model file.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />Alternatively, provide an absolute file path. |
  | `pose-labels` | Path to the pose detection labels file.<br /><br />Label files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />Alternatively, provide an absolute file path. |
  | `pose-settings-path` | Path to the pose settings file.<br /><br />Settings files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />Alternatively, provide an absolute file path. |
  | `output-file` | Path to the output file.<br /><br />For a relative path, the output is saved under:<br />`$HOME/Downloads/qimsdk_samples/media`<br /><br />Alternatively, provide an absolute file path.<br /><br />If this field is not specified, display output is selected. |
  | `enable-usb-camera` | Enables or disables the USB camera input.<br /><br />Accepted values:<br />- `TRUE`<br />- `FALSE` |
  | `width` | USB camera resolution width. |
  | `height` | USB camera resolution height. |
  | `framerate` | USB camera frame rate. |
  | `video-format` | USB video format.<br /><br />Supported values:<br />- `nv12`<br />- `yuy2`<br />- `mjpeg` |
  | `output-ip-address` | IP address for the RTSP output.<br /><br />Example: `127.0.0.1` |
  | `port` | Port for the RTSP output.<br /><br />Example: `8900` |
  | `pose-runtime` | Runtime used for pose detection.<br /><br />Supported values:<br />- `cpu`<br />- `gpu`<br />- `dsp` |
  | `detection-runtime` | Runtime used for detection.<br /><br />Supported values:<br />- `cpu`<br />- `gpu`<br />- `dsp` |
</Accordion>

### Multistream Inference

The [**gst-ai-multistream-inference**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-multistream-inference/main.c) application shows AI inference (object detection and classification) on up to 32 input streams coming from camera, file, or RTSP stream.

The following figure shows the pipeline, which receives several input streams, preprocesses them, runs AI inferences, combines the streams, and merges them all into a single video output.

The maximum number of input streams supported on each SoC as verified on 1080P and 720P are follows:

* QCS6490–8

* Dragonwing IQ-8275–16

* Dragonwing IQ-9075–32
  <Note>
    This application isn't supported in `Config #1` of `Qualcomm Linux` because CPU runtime is not supported.
  </Note>
  The output is displayed on an HDMI display, saved as an H.264 encoded MP4
  file, or converted into an RTSP stream.

For information about the plugins used in this pipeline, see [**Pipeline flow**](#multistream-inference-pipeline-flow).

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-multistream-inference`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-multistream-inference/main.c)

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - |
  | Config #2 | Yes | Yes | No | Yes | Yes | Yes | Yes |

  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | detection: `yolox_quantized.tflite`<br />classification: `inception_v3_quantized.tflite` | detection: `yolox.json`<br />classification: `classification.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-multistream-inference --config-file=/etc/configs/config-multistream-inference.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config-multistream-inference.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-multistream-inference -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-multistream-inference --config-file=/path/to/config-multistream-inference.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config-multistream-inference.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config-multistream-inference.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config-multistream-inference.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config-multistream-inference.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config-multistream-inference.json "$XDG_CONFIG_HOME/qimsdk_samples/config-multistream-inference.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#multistream-inference-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-multistream-inference
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-multistream-inference application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#multistream-inference-config-json-description) for all fields.

  ```json theme={null}
  {
    "input-file-path": ["<input file 1>", "<input file 2>"],
    "input-rtsp-path": ["<rtsp stream 1>", "<rtsp stream 2 >"],
    "input-type": "<h264 or h265>",
    "model": "<path to model file>",
    "labels": "<path to label file>",
    "output-display": "<0 or 1>",
    "output-file-path": "<output file path>",
    "output-ip-address": "<ip address of test kit>",
    "output-port-number": "<port number over which rtsp stream can be listened>",
    "use-case": "<0 or 1>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `input-file-path` | `$HOME/Downloads/qimsdk_samples/medias/<file-name>` |
  | `model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |
  | `output-file-path` | `$HOME/Downloads/qimsdk_samples/medias/<file-name>` |

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        Object Detection on 8 H.264 file inputs, LiteRT model, DSP runtime

        ```json theme={null}
        {
          "input-file-path":
            [
              "video.mp4",
              "video.mp4",
              "video.mp4",
              "video.mp4",
              "video.mp4",
              "video.mp4",
              "video.mp4",
              "video.mp4"
            ],
          "model": "yolox_quantized.tflite",
          "labels": "yolox.json",
          "input-type": "h264",
          "output-display": 1,
          "use-case": 0
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  <img alt="Pipeline Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the multistream inference pipeline:

  | Plugin | Description |
  | :- | :- |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | • Captures the live stream from camera.<br />• Uses `tee` to split the stream for inferencing. |
  | `filesrc` | • Captures the video stream using `filesrc`, followed by `qtdemux`, which demultiplexes the stream.<br />• Uses `tee` to split the stream for inferencing. |
  | `rtspsrc` | • Captures the RTSP stream using `rtspsrc`, followed by `rtph264depay` for video extraction.<br />• Uses `tee` to split the stream for inferencing. |
  | `h264parse` | • Parses the H.264 video. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | • Decodes the video. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data. This preprocessing is done when the model expects floating-point values as input:<br />   • Color conversion<br />   • Scaling (up or down)<br />   • Normalization<br />3. Converts the preprocessed video stream to a tensor stream on its source pad.<br />The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | 1. After the inference runtime receives the tensor stream on its sink pad, it runs the inference.<br />2. Produces a tensor stream with the inference results on its source pad. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | Handles the inference results from any object detection, classification, pose detection, and segmentation model.<br /><br />**Detection use case:**<br />   • Applies a threshold to the chosen number of results.<br />   • Loads the YOLOv8 module.<br />   • Produces video frames with only bounding boxes that can be overlaid on objects, sending them to the sink pad of `qtivcomposer`.<br /><br />**Classification use case:**<br />   • Applies the threshold to the chosen number of results.<br />   • Loads the MobileNet-softmax module.<br />   • Produces results as video frames with classification labels, sending them to the sink pad of `qtivcomposer`. |
  | [`qtivcomposer`](../plugin-reference/qtivcomposer) | 1. Composes frames with contents from its sink pads.<br />2. Pushes the GStreamer buffers containing these composed frames to its source pad. |
  | [`waylandsink`](../plugin-reference/waylandsink) | 1. `waylandsink` submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |
  | `filesink` | 1. Receives the video stream on its sink pad.<br />2. Saves the stream as a H.264-encoded MP4 file. |
  | [`qtirtspbin`](../plugin-reference/qtirtspbin) | 1. Serves as a network sink.<br />2. Transmits UDP packets to the network. |

  #### Config JSON Field Description

  | Field | Values / Description |
  | - | - |
  | `num-camera` | Number of camera streams.<br /><br />Supported values:<br />- `1`<br />- `2`<br /><br />Maximum number of camera streams: `2` |
  | `camera-id` | Camera ID used as the input source.<br /><br />Supported values:<br />- `0`<br />- `1`<br /><br />If no other input is selected, camera `0` is used by default.<br /><br />This parameter is ignored when `num-camera` is set to `2`. |
  | `input-file-path` | JSON array containing the input media file paths.<br /><br />Example:<br />`["video1.mp4", "video2.mp4"]`<br /><br />Relative paths are resolved under:<br />`$HOME/Downloads/qimsdk_samples/media`<br /><br />Absolute file paths are also supported.<br /><br />Maximum number of input files: `32` |
  | `input-rtsp-path` | JSON array containing the input RTSP stream paths.<br /><br />Example:<br />`["rtsp://127.0.0.1:8554/live1.mkv", "rtsp://127.0.0.1:8554/live2.mkv"]`<br /><br />Maximum number of RTSP input streams: `32` |
  | `model` | Path to the model file.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />Alternatively, provide an absolute file path. |
  | `labels` | Path to the labels file.<br /><br />Label files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />Alternatively, provide an absolute file path. |
  | `output-file-path` | Path to save the H.264-encoded output file.<br /><br />Relative paths are resolved under:<br />`$HOME/Downloads/qimsdk_samples/media`<br /><br />Absolute file paths are also supported. |
  | `output-ip-address` | IP address on which the RTSP server listens.<br /><br />Example: `127.0.0.1`<br /><br />Adding this parameter enables RTSP stream output. |
  | `output-port-number` | Port number on which the RTSP server listens.<br /><br />Example: `8554`<br /><br />Adding this parameter enables RTSP stream output. |
  | `output-display` | Enables or disables output on the Wayland display.<br /><br />Set the value to `true` to enable Wayland display output. |
  | `use-case` | Select the application use case.<br /><br />Supported values:<br />- `0` – Detection<br />- `1` – Classification |
  | `input-type` | Encoding type for file or RTSP input.<br /><br />Supported values:<br />- `h264`<br />- `h265`<br /><br />The default encoding type is H.264. |
</Accordion>

***

### Multi-Stream Batch Inference

The [**gst-ai-multistream-batch-inference**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-multistream-batch-inference/main.c) application shows batched AI inference (object detection and segmentation) on up to 24 input streams from video files.

The following figure shows the pipeline, which receives several input streams, preprocesses them, runs AI inferences, combines the streams with inference, and merges them into a single video output.

The maximum number of input streams supported on each SoC are follows:

QCS6490–8

Dragonwing IQ-8275–4

Dragonwing IQ-9075–4

The output is displayed either on an HDMI display or saved as an H.264 encoded MP4 file.

For information about the plugins used in this pipeline, see [**Pipeline flow**](#multistream-batch-inference-pipeline-flow).

<Note>
  This application isn't supported in `Config #1` of `Qualcomm Linux` because CPU runtime is not supported.
</Note>

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-multistream-batch-inference`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-multistream-batch-inference/main.c)

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - |
  | Config #2 | Yes | No | No | No | Yes | Yes | Yes |

  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | segmentation: `deeplabv3_plus_mobilenet_quantized.tflite`<br />detection: `yolov8_det_w8a8.tflite` | segmentation:`deeplabv3_resnet50.json`<br />detection: `yolov8.json` |
  | Qualcomm AI Engine direct | segmentation: `deeplabv3_plus_mobilenet_quantized.bin`<br />detection: `yolov8_det_quantized.bin` | segmentation:`deeplabv3_resnet50.json`<br />detection: `yolov8.json` |
  | Qualcomm Neural Processing SDK | segmentation: `deeplabv3_plus_mobilenet_quantized.dlc`<br />detection: `yolov8_det_quantized.dlc` | segmentation:`deeplabv3_resnet50.json`<br />detection: `yolov8.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-multistream-batch-inference --config-file=/etc/configs/config-multistream-batch-inference.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config-multistream-batch-inference.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-multistream-batch-inference -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-multistream-batch-inference --config-file=/path/to/config-multistream-batch-inference.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config-multistream-batch-inference.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config-multistream-batch-inference.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config-multistream-batch-inference.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config-multistream-batch-inference.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config-multistream-batch-inference.json "$XDG_CONFIG_HOME/qimsdk_samples/config-multistream-batch-inference.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#multistream-batch-inference-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-multistream-batch-inference
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-multistream-batch-inference application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#multistream-batch-inference-config-json-description) for all fields.

  ```json theme={null}
  {
    "output-type": "wayland or filesink",
    "out-file":"<Path to output file if output-type is filesink>",
    "pipeline-info":[
      {
        "id": "<Batch id takes value from 0 to 5>",
        "Input type": "<Input source type like file>",
        "input-file-path": [
          {
          "<stream[i]: Comma separated input file/RTSP path>"
          }
        ],
        "mlframework": "<tflite or snpe or qnn>",
        "model-path": "<path-to-model-file>",
        "labels": "<path-to-label-file>",
        "post processing plugin": "qtimlvdetection"
      }
    ]
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `input-file-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `model-path` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |
  | `out-file` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |

  <Note>
    For 16 and 24 streams, add the required elements in the `pipeline-info` parameter. The `id` parameter takes the values from 0 to 5 for each added batch.
  </Note>

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        File source, LiteRT model, DSP runtime

        ```json theme={null}
        {
          "output-type":"wayland",
          "pipeline-info":[
            {
              "id":0,
              "input-type":"file",
              "input-file-path":[
                {
                    "stream-0":"video.mp4",
                    "stream-1":"video.mp4",
                    "stream-2":"video.mp4",
                    "stream-3":"video.mp4"
                }
              ],
              "mlframework":"tflite",
              "model-path":"yolov8_det_w8a8.tflite",
              "labels-path":"yolov8.json",
              "post-process-plugin": "qtimlvdetection"
            }
          ]
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  <img alt="Pipeline Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the multistream batch inference pipeline:

  | Plugin | Description |
  | - | - |
  | `filesrc` | 1. Captures the video stream using `filesrc`.<br />2. `qtdemux` demultiplexes the stream.<br />3. Uses `tee` to split the stream for inferencing. |
  | `h264parse` | Parses the H.264 video. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | Decodes the video. |
  | [`qtibatch`](../plugin-reference/qtibatch) | 1. Reads input from the streams on its sink pad.<br />2. Batches the streams for preprocessing. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data when the model expects floating-point values as input:<br />   • Color conversion<br />   • Scaling (up or down)<br />   • Normalization<br />3. The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | 1. After the inference runtime receives the tensor stream on its sink pad, it runs the inference.<br />2. Produces a tensor stream with the inference results on its source pad. |
  | [`qtimldemux`](../plugin-reference/qtimldemux) | 1. Demultiplexes the batched output.<br />2. Splits the output corresponding to the input streams. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | Converts the inference tensors received on the sink pad into video formats that the multimedia plugins can use for further processing. |
  | [`qtivcomposer`](../plugin-reference/qtivcomposer) | 1. Composes frames with contents from its sink pads.<br />2. Pushes the GStreamer buffers containing these composed frames to its source pad. |
  | [`waylandsink`](../plugin-reference/waylandsink) | 1. Submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |
  | `filesink` | Takes the video stream received on its sink pad and saves it as an H.264-encoded MP4 file. |

  #### Config JSON Field Description

  | Field | Values / Description |
  | - | - |
  | `input-type` | Specifies the input type.<br /><br />The application accepts file input. |
  | `input-file-path` | Path to the input media file.<br /><br />Relative paths are resolved under:<br />`$HOME/Downloads/qimsdk_samples/media`<br /><br />Absolute file paths are also supported. |
  | `model-path` | Path to the model file.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />Absolute file paths are also supported. |
  | `labels-path` | Path to the labels file.<br /><br />Label files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />Absolute file paths are also supported. |
  | `post-process-plugin` | Select the post-processing plugin.<br /><br />Supported values:<br />- `qtimlvsegmentation`<br />- `qtimlvdetection` |
  | `mlframework` | Select the ML framework.<br /><br />Supported values:<br />- `tflite`<br />- `snpe`<br />- `qnn` |
  | `snpe-tensors` | JSON array containing the output tensors for the SNPE model.<br /><br />Example:<br />`["boxes", "scores", "class_idx"]` |
  | `output-type` | Select the output type.<br /><br />Supported values:<br />- `wayland`<br />- `filesink` |
  | `out-file` | Path to the output file.<br /><br />Relative paths are resolved under:<br />`$HOME/Downloads/qimsdk_samples/media`<br /><br />Absolute file paths are also supported. |
</Accordion>

***

### Multi input/output object detection

The [**gst-ai-multi-input-output-object-detection**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-multi-input-output-object-detection/main.c) application allows you to perform object detection, object classification, pose detection, and image segmentation on an input stream from different sources such as a camera, a file, or an RTSP network. The use cases implement the LiteRT models for object detection, image segmentation, classification, and pose detection.

The following figure shows the pipeline workflow, which captures video streams for inferencing from different sources such as camera, file, or RTSP. For information about the plugins used in the pipeline, see [**Pipeline flow**](#multi-input-output-object-detection-pipeline-flow).

<Note>
  This application isn't supported in `Config #1` of `Qualcomm Linux` because CPU runtime is not supported.
</Note>

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-multi-input-output-object-detection`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-multi-input-output-object-detection/main.c)

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - |
  | Config #2 | Yes | Yes | No | Yes | Yes | Yes | Yes |

  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | `yolov5.tflite` | `yolov5.json` |

  #### Prerequisites

  <Note>
    Update the following commands according to the Python version in your Linux
    host computer.
  </Note>

  * Create the Python 3.8 virtual environment:

  ```bash theme={null}
  sudo apt-get install python3.8 
  ```

  ```bash theme={null}
  sudo apt-get install python3.8-venv 
  ```

  ```bash theme={null}
  python3.8 -m venv py3.8 
  ```

  ```bash theme={null}
  source py3.8/bin/activate 
  ```

  * Generate the `yolov5.tflite` model:

  ```bash theme={null}
  git clone https://github.com/ultralytics/yolov5.git
  ```

  ```bash theme={null}
  cd yolov5
  ```

  ```bash theme={null}
  python -m pip install -r requirements.txt tensorflow-cpu
  ```

  ```bash theme={null}
  python export.py --weights yolov5m.pt --img 320 --include tflite --int8 --data data/coco128.yaml
  ```

  * In the terminal of the host computer, run the following command to push the model to the target device:

  ```bash theme={null}
  scp yolov5m-int8.tflite root@<ip address of the device>:~/Downloads/qimsdk_samples/models/yolov5.tflite
  ```

  If any model isn't available after downloading the script file, you can download the model from [**IoT– Qualcomm AI Hub**](https://aihub.qualcomm.com/iot/models).

  * In the terminal of the host computer, run the following command to push the model files to the target device:

  ```bash theme={null}
  scp <model_filename> root@< address of target device>:~/Downloads/qimsdk_samples/models
  ```

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-multi-input-output-object-detection --config-file=/etc/configs/config-multi-input-output-object-detection.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config-multi-input-output-object-detection.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-multi-input-output-object-detection -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-multi-input-output-object-detection --config-file=/path/to/config-multi-input-output-object-detection.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config-multi-input-output-object-detection.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config-multi-input-output-object-detection.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config-multi-input-output-object-detection.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config-multi-input-output-object-detection.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config-multi-input-output-object-detection.json "$XDG_CONFIG_HOME/qimsdk_samples/config-multi-input-output-object-detection.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#multi-input-output-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-multi-input-output-object-detection
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-multi-input-output-object-detection application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#multi-input-output-object-detection-config-json-description) for all fields.

  ```json theme={null}
  {
    "num-camera": "<number-of-cameras>",
    "camera-id": "<specific-camera-id>",
    "input-file-path": "<JSON-array-of-input-file>",
    "input-rtsp-path": "<JSON-array-of-RTSP-URLs>",
    "model": "<path-to-model-file>",
    "labels": "<path-to-label-file>",
    "output-file-path": "<path-to-output-file>",
    "output-ip-address": "<ip-address-of-output>",
    "output-port-number": "<port number over which rtsp stream can be listened>",
    "output-display": "<true or false>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `input-file-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `labels` | `$HOME/Downloads/qimsdk_samples/labels/<file-name>` |
  | `output-file-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |

  <Note>
    Ensure that the total number of input streams from the camera, RTSP, and file source doesn't exceed 6.
  </Note>

  <Note>
    For QCS6490, if `file-path` and `rtsp-ip-port` are not present in the configuration file, then the camera input is selected.
  </Note>

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        File source, LiteRT model, DSP runtime

        ```json theme={null}
        {
          "input-file-path":
              [
                  "video1.mp4",
                  "video2.mp4"
              ],
          "model": "yolov5.tflite",
          "labels": "yolov5.json",
          "output-display": true
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  Based on the use case, the results are either displayed on an HDMI screen, saved as an H.264 encoded MP4 file, or streamed over the RTSP server.

  <img alt="Pipeline Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the multi input output object detection pipeline:

  | Plugin | Description |
  | - | - |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | • Captures the live stream from the camera.<br />• Uses `tee` to split the stream for inferencing. |
  | `filesrc` | • Captures the video stream using `filesrc`.<br />• Followed by `qtdemux`, which demultiplexes the stream.<br />• Uses `tee` to split the stream for inferencing. |
  | `rtspsrc` | • Captures the RTSP stream using `rtspsrc`.<br />• Followed by `rtph264depay` for video extraction.<br />• Uses `tee` to split the stream for inferencing. |
  | `h264parse` | Parses the H.264 video. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | Decodes the video. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | • Receives the video stream on its sink pad.<br />• Performs the following preprocessing on the stream data when the model expects floating-point values as input:<br />  • Color conversion<br />  • Scaling (up or down)<br />  • Normalization<br />• The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | • Runs on LiteRT and uses the `yolov5.tflite` model for object detection.<br />• After the inference runtime receives the tensor stream on its sink pad, it runs the inference.<br />• Produces a tensor stream with the inference results on its source pad. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | • Converts the inference tensors that it receives on its sink pad into video formats that the multimedia plugins can process later. |
  | [`qtivcomposer`](../plugin-reference/qtivcomposer) | • Composes frames with contents from its sink pads.<br />• Pushes the GStreamer buffers containing these composed frames to its source pad. |
  | [`waylandsink`](../plugin-reference/waylandsink) | • `waylandsink` submits the video stream received on its sink pad to the Wayland compositor.<br />• Renders the video stream on a local display. |
  | `filesink` | Takes the video stream that it receives on its sink pad and saves it as an H.264-encoded MP4 file. |
  | [`qtirtspbin`](../plugin-reference/qtirtspbin) | • Serves as a network sink.<br />• Transmits UDP packets to the network. |

  #### Config JSON Field Description

  | Field | Values / Description |
  | - | - |
  | `num-camera` | Number of camera streams.<br /><br />Supported values:<br />- `1`<br />- `2`<br /><br />Maximum number of camera streams: `2` |
  | `camera-id` | Camera ID used as the input source.<br /><br />Supported values:<br />- `0`<br />- `1`<br /><br />Camera `0` is selected by default if no other input is specified.<br /><br />This parameter is ignored when `num-camera` is set to `2`. |
  | `input-file-path` | JSON array containing the input file paths.<br /><br />Example:<br />`["video1.mp4", "video2.mp4"]`<br /><br />Relative paths are resolved under:<br />`$HOME/Downloads/qimsdk_samples/media`<br /><br />Absolute file paths are also supported.<br /><br />Maximum number of input files: `6` |
  | `input-rtsp-path` | JSON array containing the input RTSP stream paths.<br /><br />Example:<br />`["rtsp://127.0.0.1:8554/live1.mkv", "rtsp://127.0.0.1:8554/live2.mkv"]`<br /><br />Maximum number of RTSP input streams: `6` |
  | `model` | Path to the model file.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />Alternatively, provide an absolute file path. |
  | `labels` | Path to the labels file.<br /><br />Label files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />Alternatively, provide an absolute file path. |
  | `output-file-path` | Path to save the H.264-encoded output file.<br /><br />Relative paths are resolved under:<br />`$HOME/Downloads/qimsdk_samples/media`<br /><br />Absolute file paths are also supported. |
  | `output-ip-address` | IP address on which the RTSP server listens.<br /><br />Example: `127.0.0.1`<br /><br />Adding this parameter enables RTSP stream output. |
  | `output-port-number` | Port number on which the RTSP server listens.<br /><br />Example: `8554`<br /><br />Adding this parameter enables RTSP stream output. |
  | `output-display` | Enables or disables output on the Wayland display.<br /><br />Set the value to `true` to enable Wayland display output.<br /><br />If no output is selected, Wayland output is selected by default. |
</Accordion>

***

### Parallel Inferencing

The [**gst-ai-parallel-inference**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-parallel-inference/main.c) application allows you to perform object detection, object classification, pose detection, and image segmentation on an input stream from different sources such as a camera, a file, or an RTSP network. The use cases implement the LiteRT models for object detection, image segmentation, classification, and pose detection.

The following figure shows the pipeline, which receives input streams from a camera, file, or an RTSP stream, performs the parallel inferencing for the four use cases, and displays the results side by side on the screen.

<Note>
  This application isn't supported in `Config #1` of `Qualcomm Linux` because CPU runtime is not supported.
</Note>

For information about the plugins used in this pipeline, see [**Pipeline flow**](#parallel-inference-pipeline-flow).

<img alt="Pipeline Diagram" />

**Application:** [`gst-ai-parallel-inference`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-ai-parallel-inference/main.c)

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output |
  | - | - | - | - | - | - | - | - |
  | Config #2 | Yes | Yes | No | Yes | No | Yes | No |

  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | detection: `yolox_quantized.tflite`<br />classification: `inception_v3_quantized.tflite`<br />segmentation: `deeplabv3_plus_mobilenet_quantized.tflite`<br />pose: `hrnet_pose_quantized.tflite` | detection: `yolox.json`<br />classification: `classification.json`<br />segmentation: `deeplabv3_resnet50.json`<br />pose: `hrnet_pose.json`, `hrnet_settings.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device.

      <Note>
        Sample configuration files are available in `/etc/configs`. You can use the sample configuration or create a custom configuration file.
      </Note>
    </Step>

    <Step title="Run the application with the sample configuration file">
      To run the application with the sample configuration file, use:

      ```bash theme={null}
      gst-ai-parallel-inference --config-file=/etc/configs/config-parallel-inference.json
      ```

      This command runs the application using the sample configuration in `/etc/configs/config-parallel-inference.json`.

      To view the available command-line options, run:

      ```bash theme={null}
      gst-ai-parallel-inference -h
      ```

      Press **CTRL + C** to stop the execution.
    </Step>

    <Step title="Run the application with a custom configuration file">
      You can use a custom configuration file by specifying its path with `--config-file`:

      ```bash theme={null}
      gst-ai-parallel-inference --config-file=/path/to/config-parallel-inference.json
      ```

      If `--config-file` is specified, the application uses that file. Otherwise, it searches for `config-parallel-inference.json` in the following order:

      | Priority | Location | Description |
      | :-: | :- | :- |
      | 1 | The path provided with `--config-file`. | Uses the configuration file specified on the command line. |
      | 2 | `$XDG_CONFIG_HOME/qimsdk_samples/config-parallel-inference.json` | Uses the user-defined configuration directory when the environment variable is set. |
      | 3 | `$HOME/.config/qimsdk_samples/config-parallel-inference.json` | Uses the default user configuration directory when no file is found in `XDG_CONFIG_HOME`. |
      | 4 | `/etc/configs/config-parallel-inference.json` | Uses the system-wide configuration file as the final fallback. |

      To create a user-specific configuration, copy the sample file to the user configuration directory:

      ```bash theme={null}
      export XDG_CONFIG_HOME="$HOME/.config"
      mkdir -p "$XDG_CONFIG_HOME/qimsdk_samples"
      cp /etc/configs/config-parallel-inference.json "$XDG_CONFIG_HOME/qimsdk_samples/config-parallel-inference.json"
      ```

      <Note>
        Modify the copied configuration file to specify the required parameters. The custom configuration file must follow the schema defined in [`configurations`](#parallel-inference-configurations).
      </Note>

      Run the application:

      ```bash theme={null}
      gst-ai-parallel-inference
      ```
    </Step>
  </Steps>

  #### Configurations

  The gst-ai-parallel-inference application uses the following structure for its configuration file. Update its properties to match your model, input stream, and output. See [**Config JSON Field Description**](#parallel-inference-config-json-description) for all fields.

  ```json theme={null}
  {
    "camera": "<camera-id>",
    "file-path": "<input-video-path>",
    "rtsp-ip-port": "<RTSP-IP-Port-address>",
    "detection-model": "<path-to-detection model>",
    "detection-labels": "<path-to-detection-labels>",
    "pose-model": "<path-to-pose-model>",
    "pose-labels": "<path-to-pose-label>",
    "pose-settings-path": "<path-to-pose-settings-file>",
    "segmentation-model": "<path-to-segmentation-model>",
    "segmentation-labels": "<path-to-segmentation-labels>",
    "classification-model": "<path-to-classification-model>",
    "classification-labels": "<path-to-classification-labels>"
  }
  ```

  The application uses `$HOME/Downloads/qimsdk_samples` as its artifact directory. When a relative path or filename is specified, it is resolved against the corresponding subdirectory below. Alternatively, absolute paths are used as provided.

  | Configuration field | Resolution of a relative path |
  | :- | :- |
  | `file-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `detection-model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `pose-model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `segmentation-model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `classification-model` | `$HOME/Downloads/qimsdk_samples/models/<file-name>` |
  | `detection-labels` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `pose-labels` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `pose-settings-path` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `segmentation-labels` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |
  | `classification-labels` | `$HOME/Downloads/qimsdk_samples/media/<file-name>` |

  <Note>
    For QCS6490, if `file-path` and `rtsp-ip-port` are not present in the configuration file, then the camera input is selected.
  </Note>

  <Accordion title="Available configurations">
    <Tabs>
      <Tab title="Config #2">
        File source, LiteRT model, DSP runtime

        ```json theme={null}
        {
            "file-path": "video.mp4",
            "detection-model": "yolox_quantized.tflite",
            "detection-labels": "yolox.json",
            "pose-model": "hrnet_pose_quantized.tflite",
            "pose-labels": "hrnet_pose.json",
            "pose-settings-path": "hrnet_settings.json",
            "segmentation-model": "deeplabv3_plus_mobilenet_quantized.tflite",
            "segmentation-labels": "deeplabv3_resnet50.json",
            "classification-model": "inception_v3_quantized.tflite",
            "classification-labels": "classification.json"
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  #### Expected Output

  After performing the four parallel inferences, the results are displayed side by side on the screen.

  <img alt="Pipeline Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the parallel inference pipeline:

  | Plugin | Description |
  | - | - |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | • Captures the live stream from the camera.<br />• Uses `tee` to split the stream for inferencing. |
  | `filesrc` | • Captures the video stream using `filesrc`.<br />• Followed by `qtdemux`, which demultiplexes the stream.<br />• Uses `tee` to split the stream for inferencing. |
  | `rtspsrc` | • Captures the RTSP stream using `rtspsrc`.<br />• Followed by `rtph264depay` for video extraction.<br />• Uses `tee` to split the stream for inferencing. |
  | `h264parse` | Parses the H.264 video. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | Decodes the video. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data when the model expects floating-point values as input:<br />  • Color conversion<br />  • Scaling (up or down)<br />  • Normalization<br />3. The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | • After the inference runtime receives the tensor stream on its sink pad, it runs the inference.<br />• Produces a tensor stream with the inference results on its source pad. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) — detection | a. Receives the inference tensors from the object detection model.<br />b. Converts the inference tensors on its sink pad into formats such as video or text that the multimedia plugins can process later.<br />c. Applies the threshold to the chosen number of results.<br />d. Loads the corresponding modules for detection models.<br />In this use case, `qtimlpostprocess` does the following:<br />  • Loads the YOLOv8 submodule.<br />  • Produces results as structures of text.<br />  • Sends them to the sink pad of `qtimetamux`. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) — classification | a. Receives the inference tensors from the classification model.<br />b. Converts the inference tensors on its sink pad into formats such as video or text that the multimedia plugins can process later.<br />c. Applies the threshold to the chosen number of results.<br />d. Loads the corresponding modules for classification models.<br />In this use case, `qtimlpostprocess` does the following:<br />  • Loads the submodule of the model.<br />  • Produces results as video frames with classification labels.<br />  • Sends them to the sink pad of `qtivcomposer`. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) — segmentation | a. Receives the inference tensors on its sink pad.<br />b. Converts the inference tensors into video formats that the multimedia plugins can process later.<br />c. Produces the semantic segmentations for the frame.<br />d. Loads the corresponding modules for the segmentation models.<br />In this use case, `qtimlpostprocess` does the following:<br />  • Loads the deeplab-argmax submodule.<br />  • Produces video frames with segmentation masks.<br />  • Sends them to the sink pad of `qtivcomposer`. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) — pose | a. Receives the inference tensors on its sink pad.<br />b. Converts the inference tensors into video formats that the multimedia plugins can process later.<br />c. Applies the threshold to the chosen number of results.<br />d. Loads the corresponding modules for various pose estimation models.<br />In this use case, `qtimlpostprocess` does the following:<br />  • Loads the HRNet module.<br />  • Produces results as video frames with poses drawn.<br />  • Sends them to the sink pad of `qtivcomposer`. |
  | [`qtivcomposer`](../plugin-reference/qtivcomposer) | 1. Composes frames with contents from its sink pads.<br />2. Pushes the GStreamer buffers containing these composed frames to its source pad. |
  | [`waylandsink`](../plugin-reference/waylandsink) | 1. Submits the video stream received on its sink pad to Weston.<br />2. Weston renders the video stream on a local display. |

  #### Config JSON Field Description

  | Field | Values / Description |
  | - | - |
  | `camera` | Select the camera:<br /><br />- `0` – Primary camera<br />- `1` – Secondary camera |
  | `file-path` | Path to the input media file.<br /><br />The media file should be placed in:<br />`$HOME/Downloads/qimsdk_samples/media`<br /><br />Alternatively, provide an absolute file path. |
  | `rtsp-ip-port` | Provide the RTSP input URI.<br /><br />Example: `rtsp://192.168.1.110:8554/live.mkv` |
  | `detection-model` | Path to the object detection model.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />Alternatively, provide an absolute file path. |
  | `detection-labels` | Path to the object detection labels.<br /><br />Label files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />Alternatively, provide an absolute file path. |
  | `pose-model` | Path to the pose detection model.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />Alternatively, provide an absolute file path. |
  | `pose-labels` | Path to the pose detection labels.<br /><br />Label files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />Alternatively, provide an absolute file path. |
  | `segmentation-model` | Path to the segmentation model.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />Alternatively, provide an absolute file path. |
  | `segmentation-labels` | Path to the segmentation labels.<br /><br />Label files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />Alternatively, provide an absolute file path. |
  | `classification-model` | Path to the classification model.<br /><br />Model files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/models`<br /><br />Alternatively, provide an absolute file path. |
  | `classification-labels` | Path to the classification labels.<br /><br />Label files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />Alternatively, provide an absolute file path. |
  | `pose-settings-path` | Path to the pose settings file.<br /><br />Settings files should be placed in:<br />`$HOME/Downloads/qimsdk_samples/labels`<br /><br />Alternatively, provide an absolute file path. |
</Accordion>

***

### Gesture Recognition

The [**gst-gesture-recognition**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-gesture-recognition/main.c) provides four-stage cascading pipeline that performs palm detection, hand landmark estimation, gesture embedding, and gesture classification on a live camera stream using ROI-based metadata propagation.

<Note>
  This application is not supported on the `QLI GA 2.1` release for the `IQ-2390` and `CQ-2390` platforms.
</Note>

For information about the plugins used in the pipeline flow, see [**Pipeline flow**](#gesture-recognition-pipeline-flow)

<img alt="Pipeline Diagram" />

**Application:** [`gst-gesture-recognition`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-sample-apps/gst-gesture-recognition/main.c)

<Accordion title="Try me">
  #### Input and Output Capabilities

  | Config | File src | RTSP | USB camera | MIPI camera | File output | Display | RTSP output | Web RTC |
  | - | - | - | - | - | - | - | - | - |
  | Config #2 | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes |

  <Note>
    `camera` use-case is not supported on `IQ-2390` and `CQ-2390`.
  </Note>

  #### Sample Model and Label Files

  The gesture recognition pipeline uses the following floating-point LiteRT models and associated metadata files:

  | Pipeline stage | Model file | Label and settings files |
  | - | - | - |
  | Palm detection | `hand_detector.tflite` | `palmd_labels.json`<br />`palmd_settings.json` |
  | Hand landmark estimation | `hand_landmark.tflite` | `hlandmark_labels.json`<br />`hlandmark_settings.json` |
  | Gesture embedding | `gesture_embedder.tflite` | |
  | Gesture classification | `canned_gesture_classifier.tflite` | `gesture_labels.json` |

  <Note>
    All AI models use floating-point precision.
  </Note>

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      * Complete the [`Prerequisites`](#prerequisites) to copy the sample models, labels, media files, and configuration files to the target device. The application reads models from `$HOME/models` and labels from `$HOME/labels` by default.
      * Enable the GPU delegate before running the application, as it supports execution on GPU only.
    </Step>

    <Step title="Run the application">
      To run the application with the ISP camera and display the annotated stream:

      ```bash theme={null}
      gst-gesture-recognition --input-type=isp --input-config=0 --width=1280 --height=720
      ```

      <Note>
        Display output is enabled by default but can be disabled with `--no-display`
      </Note>

      To use a USB camera:

      ```bash theme={null}
      gst-gesture-recognition \
        --input-type=usb \
        --input-config=/dev/video0 \
        --width=1280 \
        --height=720 \
        --framerate=30
      ```

      To process a video file and save the annotated output:

      ```bash theme={null}
      gst-gesture-recognition \
        --input-type=file \
        --input-config=${HOME}/Downloads/qimsdk_samples/media/video.mp4 \
        --output-type=file \
        --output-config=output.mp4
      ```

      To view all supported options, run:

      ```bash theme={null}
      gst-gesture-recognition --help
      ```

      Press **CTRL + C** to stop the execution.
    </Step>
  </Steps>

  #### Expected Output

  The application detects hands, estimates keypoints, and recognizes gestures. Results are overlaid on each frame and rendered on the display.

  <img alt="Pipeline Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the gesture recognition pipeline:

  | Plugin | Description |
  | - | - |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | Captures live video from the ISP camera as the pipeline source. |
  | tee | Splits the stream for palm detection and downstream ROI-based stages. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | Preprocesses full frames (Stage 1) and ROI-cropped patches (Stage 2) for inference. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | Runs palm detection, hand landmark, gesture embedder, and gesture classifier inference sequentially. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | Post-processes each stage's tensors (palm ROIs, landmarks, gesture labels). |
  | qtimetatransform | Transforms ROI palm-detection metadata into cropped regions for the landmark stage. |
  | [`qtimetamux`](../plugin-reference/qtimetamux) | Merges video and metadata/text streams, attaching inference results as GST buffer metadata. |
  | [`qtivoverlay`](../plugin-reference/qtivoverlay) | Overlays inference results (labels, bounding boxes, keypoints) onto the video frame using CL. |
  | [`waylandsink`](../plugin-reference/waylandsink) | Renders the final annotated video stream to a local display via Weston. |

  #### Config JSON Field Description

  `gst-gesture-recognition` does not use a JSON configuration file. Configure the application with the following command-line options:

  | Option | Values and default |
  | - | - |
  | `--input-type` | Input source: `usb`, `isp`, `rtsp`, or `file`. Defaults to `isp`. |
  | `--input-config` | Input-specific configuration. Use `/dev/videoX` for `usb`, a camera ID for `isp`, an RTSP URL for `rtsp`, or a file path for `file`. Defaults to `/dev/video0` for USB and `0` for ISP. RTSP and file inputs require a value. |
  | `--output-type` | Output destination: `none`, `file`, `rtsp`, or `webrtc`. Defaults to `none`. |
  | `--output-config` | Output-specific configuration. Use an output file path for `file`, a port for `rtsp`, or a WebSocket signaling URL for `webrtc`. Defaults to `output.mp4`, `8554`, and `ws://127.0.0.1:8443`, respectively. |
  | `--model-base-path` | Base directory containing the required `models` and `labels` directories. Defaults to `$HOME`. |
  | `--no-display` | Disables on-screen display output. |
  | `--width`, `--height`, `--framerate` | Raw video properties. Defaults are `1920`, `1080`, and `30`, respectively. |
  | `--webrtc-id` | Local WebRTC signaling ID. Defaults to `1010`. |
</Accordion>

***

### Hardware benchmarking application

The hardware benchmarking application monitors the device hardware usage for a defined set of sample applications to capture metrics such as CPU/GPU/NPU usage and device thermals. These metrics explain the resource usage and throttling, which help to tune your AI use cases according to the requirements.

The following figure shows the pipeline, which processes the input from a set of USB cameras to generate various outputs.

<Note>
  This application isn't supported in the `QLI GA 2.1` release.
</Note>

<img alt="Pipeline Diagram" />

For information about the plugins used in this pipeline, see [**Pipeline flow**](#hardware-benchmarking-pipeline-flow).

<Accordion title="Try me">
  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | • inception\_v3\_quantized.tflite<br />• deeplabv3\_plus\_mobilenet\_quantized.tflite<br />• hrnet\_pose\_quantized.tflite<br />• midas\_quantized.tflite<br />• yolox\_quantized.tflite | • classification.json<br />• deeplabv3\_resnet50.json<br />• hrnet\_pose.json<br />• monodepth.json<br />• yolox.json |

  #### Setup the target device

  <Steps>
    <Step>
      To access the target device from your Linux host computer, set up SSH. For instructions, see [Sign in using SSH](#prerequisites).

      <Note>
        If SSH is already set up, you can skip this step.
      </Note>
    </Step>

    <Step>
      Use the HDMI port to connect the display to the device. For instructions, see Set up HDMI display.

      If you face issues with display, see [Troubleshoot display issues](https://dragonwingdocs.qualcomm.com/Technologies/Display/troubleshoot-display-issues).
    </Step>

    <Step>
      Connect two USB cameras and a mouse to the target device.
    </Step>

    <Step>
      Install the Qualcomm® Profiler on the Linux host computer. For installation instructions see [**Qualcomm Profiler**](https://dragonwingdocs.qualcomm.com/System/Performance/analyze-performance-with-tools#qualcomm-profiler-cli).

      After connecting the device to the PC, run `InstallerLE` from the following locations:

      * For Linux:

      ```bash theme={null}
      cd /opt/qcom/Shared/QualcommProfiler/API/target-le
      ./InstallerLE
      ```

      * For Windows:

      ```bash theme={null}
      cd “C:\Program Files (x86)\Qualcomm\Shared\QualcommProfiler\API\target-le”
      .\InstallerLE.exe
      ```
    </Step>
  </Steps>

  #### Run the application on the target device

  <Steps>
    <Step>
      Clone the repository for the demo application and push it to the target device:

      ```bash theme={null}
      git clone https://github.com/Avnet/QCS6490-Vision-AI-Demo.git
      cd QCS6490-Vision-AI-Demo
      git checkout QLI_2.0
      scp -r ../QCS6490-Vision-AI-Demo root@${IP_ADDRESS_OF_TARGET_DEVICE}:/opt
      ```
    </Step>

    <Step>
      Sign in to the target device over SSH and run the script to set up the resources for hardware benchmarking application:

      ```bash theme={null}
      cd /opt/QCS6490-Vision-AI-Demo && bash install.sh
      ```
    </Step>

    <Step>
      Start the application:

      ```bash theme={null}
      bash launch_visionai_with_env.sh
      ```

      <img alt="Pipeline Diagram" />
    </Step>

    <Step>
      Select the preferred sample applications from the **Camera 1** and **Camera 2** drop-downs. The system thermal and hardware usage details appear at the bottom of the screen.

      You may run different sample applications to check the output and understand the hardware utilization.

      * Example 1: Choose the **Camera** option from **Camera 1** and **Camera 2** drop-down lists to observe the preview streams on the screen.
      * Example 2: Choose any sample application from **Camera 1** and **Camera 2** drop-down lists to observe the AI inferencing camera streams on the screen.

        <img alt="Pipeline Diagram" />

      <Note>
        For more information and features of the application, select the Info icon.
      </Note>
    </Step>

    <Step>
      Select the **Exit** icon to close the application.
    </Step>
  </Steps>

  #### Pipeline Flow

  The following table lists the plugins used in the hardware benchmarking pipeline:

  | Plugin | Description |
  | - | - |
  | [`v4l2src`](../plugin-reference/v4l2src) | • Captures the live stream from the USB camera.<br />• Uses `tee` to split the stream for inferencing. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sink pad.<br />2. Performs the following preprocessing on the stream data when the model expects floating-point values as input:<br />  • Color conversion<br />  • Scaling (up or down)<br />  • Normalization<br />3. The tensor stream is used for inferencing in the later stages of the pipeline. |
  | [`qtimltflite`](../plugin-reference/qtimltflite)<br />[`qtimltsnpe`](../plugin-reference/qtimltsnpe)<br />[`qtimlqnn`](../plugin-reference/qtimlqnn) | • After the inference runtime receives the tensor stream on its sink pad, it runs the inference.<br />• Produces a tensor stream with the inference results on its source pad. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) — detection | a. Receives the inference tensors from the object detection model.<br />b. Converts the inference tensors on its sink pad into formats such as video or text that the multimedia plugins can process later.<br />c. Applies the threshold to the chosen number of results.<br />d. Loads the corresponding modules for detection models.<br />In this use case, `qtimlpostprocess` does the following:<br />  • Loads the YOLO (`YOLOv5`, `YOLOv8`, `YOLOX`, or `YOLO-NAS`) submodule.<br />  • Produces results as structures of text.<br />  • Sends them to the sink pad of `qtimetamux`. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) — classification | a. Receives the inference tensors from the classification model.<br />b. Converts the inference tensors on its sink pad into formats such as video or text that the multimedia plugins can process later.<br />c. Applies the threshold to the chosen number of results.<br />d. Loads the corresponding modules for classification models.<br />In this use case, `qtimlpostprocess` does the following:<br />  • Loads the submodule of the model.<br />  • Produces results as video frames with classification labels.<br />  • Sends them to the sink pad of `qtivcomposer`. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) — segmentation | a. Receives the inference tensors on its sink pad.<br />b. Converts the inference tensors into video formats that the multimedia plugins can process later.<br />c. Produces the semantic segmentations for the frame.<br />d. Loads the corresponding modules for the segmentation models.<br />In this use case, `qtimlpostprocess` does the following:<br />  • Loads the `deeplab-argmax` submodule.<br />  • Produces video frames with segmentation masks.<br />  • Sends them to the sink pad of `qtivcomposer`. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) — pose | a. Receives the inference tensors on its sink pad.<br />b. Converts the inference tensors into video formats that the multimedia plugins can process later.<br />c. Applies the threshold to the chosen number of results.<br />d. Loads the corresponding modules for various pose estimation models.<br />In this use case, `qtimlpostprocess` does the following:<br />  • Loads the `HRNet` module.<br />  • Produces results as video frames with poses drawn.<br />  • Sends them to the sink pad of `qtivcomposer`. |
</Accordion>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Model not available after running the download script?">
    If any model isn’t available after downloading the script file, you can download the model manually from [IoT — Qualcomm AI Hub](https://aihub.qualcomm.com)
    and push it to the target device:

    ```bash theme={null}
    scp <model filename> root@<ip addr of the target device>:~/Downloads/qimsdk_samples/models
    ```

    For example:

    ```bash theme={null}
    scp mobilenet_v2_quantized.tflite root@<ip addr of the target device>:~/Downloads/qimsdk_samples/models
    ```
  </Accordion>

  <Accordion title="Running sample applications from the UART shell">
    Remount the file system with read/write permissions:

    For Qualcomm Linux:

    ```bash theme={null}
    mount -o remount,rw /usr
    ```
  </Accordion>

  <Accordion title="Cannot locate the qticamsrc plugin?">
    If you cannot locate the qticamsrc plugin, ensure that the camera server is
    running and clear the GStreamer cache using the following commands:

    ```bash theme={null}
    ps -ef | grep cam-server 
    ```

    ```bash theme={null}
    rm ~/.cache/gstreamer-1.0/registry.aarch64.bin 
    ```
  </Accordion>

  <Accordion title="GStreamer logging and debugging">
    To enable basic GStreamer logging, run the following before launching the
    application:

    ```bash theme={null}
    export GST_DEBUG=2 
    ```

    To increase verbosity for specific
    plugins, use a comma-separated list with log levels (1–7):

    ```bash theme={null}
    export GST_DEBUG=3,qticamsrc:5,qtimlvconverter:5,qtimltflite:5 
    ```

    To redirect logs to a file for offline analysis:

    ```bash theme={null}
    export GST_DEBUG=3 
    export GST_DEBUG_FILE=/tmp/gst_classification_debug.log 
    ```
  </Accordion>
</AccordionGroup>


# Multimedia
Source: https://imsdkdocs.qualcomm.com/sample-application/multimedia-sample-applications

Ready-to-run C++ applications demonstrating QIM SDK camera, video, and audio capabilities

The multimedia sample applications provide custom use cases for camera, video, audio, graphics, and display capabilities of Qualcomm Linux.

The following tables list all available GStreamer C/C++ multimedia applications and their platform support. Select the appropriate configuration tab for your setup.

<Tabs>
  <Tab title="Config #1">
    | Application | Source code | Description | QCS6490 | IQ-8275 | IQ-9075 | IQ-615 | IQx7181 | IQx5121 | IQ-2390 |
    | - | - | - | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
    | **Camera** | | | | | | | | | |
    | Single camera streaming | [`gst-camera-single-stream-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-single-stream-example) | Single camera: preview, encode, YUV dump, or RTSP. | ✓ | ✓ | ✓ | ✓ | ✓ | × | × |
    | Multi-camera streaming | [`gst-multi-camera-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-multi-camera-example) | Concurrent streaming from two cameras. | × | × | × | TBD | × | × | × |
    | Multistream | [`gst-multi-stream-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-multi-stream-example) | Single camera duplicated into two streams via tee. | ✓ | ✓ | ✓ | ✓ | × | × | × |
    | Live snapshot | [`gst-snapshot-stream-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-snapshot-stream-example) | Preview + JPEG snapshot capture simultaneously. | × | × | × | × | × | × | × |
    | RTMP camera streaming | [`gst-rtmp-stream-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-rtmp-stream-example) | ISP / RTSP camera feed converted to RTMP stream. | × | × | × | × | × | × | × |
    | Activate/deactivate streams at runtime | [`gst-activate-deactivate-streams-runtime-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-activate-deactivate-streams-runtime) | Change resolution on-the-fly without sensor restart. | × | × | × | × | × | × | × |
    | Add/remove streams at runtime | [`gst-add-remove-streams-runtime`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-add-remove-streams-runtime) | Dynamically add and remove streams. | × | × | × | × | × | × | × |
    | Camera frame processing | [`gst-appsink-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-appsink-example) | Apply transforms to camera frames before encoding. | ✓ | ✓ | ✓ | ✓ | ✓ | × | × |
    | Camera metadata | [`gst-camera-metadata-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-metadata-example) | Capture scene mode, white balance, and bit rate metadata. | × | × | × | × | × | × | × |
    | Camera stream configuration | [`gst-add-streams-as-bundle-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-add-streams-as-bundle-example) | Configure camera streams as a bundle. | × | × | × | × | × | × | × |
    | Camera burst capture | [`gst-camera-burst-capture-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-burst-capture-example) | Burst-mode snapshot capture. | × | × | × | × | × | × | × |
    | Camera switch | [`gst-camera-switch-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-switch-example) | Switch between two cameras without stopping pipeline. | × | × | × | × | × | × | × |
    | SHDR / LDC / EIS | [`gst-camera-shdr-ldc-eis-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-shdr-ldc-eis-example) | Super HDR, lens distortion correction, and image stabilization. | × | × | × | × | × | × | × |
    | Decode JPEG images | [`gst-jpg-decode-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-jpg-decode-example) | Decode JPEG images and preview on screen. | ✓ | ✓ | ✓ | ✓ | × | × | × |
    | FastCV Padding Strip | [`gst-fastcv-padding-strip-example`](https://github.com/qualcomm/gst-plugins-imsdk/tree/main/gst-sample-apps/gst-fastcv-padding-strip-example) | Padding strpiped and Dumped to a file | × | × | × | × | × | × | × |
    | GstD-camera-single-stream-example | \[`GstD-camera-single-stream-example`] ([https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/GstD-camera-single-stream-example](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/GstD-camera-single-stream-example)) | Single camera using GstD: preview, encode or YUV dump. | ✓ | ✓ | ✓ | ✓ | ✓ | × | × |
    | **Video** | | | | | | | | | |
    | Video Wall | [`gst-concurrent-videoplay-composition`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-concurrent-videoplay-composition) | Concurrent video playback composed on display. | ✓ | ✓ | ✓ | ✓ | × | ✓ | ✓ |
    | Smart codec | [`gst-smartcodec-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-smartcodec-example) | Reduce bandwidth / storage for camera input. | × | × | × | × | × | × | × |
    | Multiformat encode/decode | [`gst-videocodec-concurrent-playback`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-videocodec-concurrent-playback) | Concurrent decode and playback of multiple formats. | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
    | Audio-video playback | [`gst-audio-video-playback`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-audio-video-playback) | Simultaneous audio and video decode and playback. | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
    | Video transcoding | [`gst-video-transcode-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-video-transcode-example) | Transcode AVC↔HEVC. | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
    | Video playback | [`gst-video-playback-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-video-playback-example) | Decode and play back a video file. | ✓ | ✓ | ✓ | ✓ | × | × | × |
    | Record and preview transformed video | [`gst-transform-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-transform-example) | Rotate, flip, and scale a video stream. | ✓ | ✓ | ✓ | ✓ | ✓ | × | × |
    | Video composition using Weston | [`gst-weston-composition-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-weston-composition-example) | Compose live camera and offline file sources. | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
    | Encode audio and video | [`gst-audio-video-encode`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-audio-video-encode) | Encode audio and video to a file. | × | × | × | × | × | × | TBD |
    | **Audio** | | | | | | | | | |
    | Audio playback | [`gst-audio-decode-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-audio-decode-example) | Decode and play back WAV, MP3, or FLAC. | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | TBD |
    | Audio recording | [`gst-audio-encode-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-audio-encode-example) | Record and encode audio to FLAC or WAV. | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | TBD |
  </Tab>

  <Tab title="Config #2">
    | Application | Source code | Description | QCS6490 | IQ-8275 | IQ-9075 | IQ-615 | IQx7181 | IQx5121 | IQ-2390 |
    | - | - | - | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
    | **Camera** | | | | | | | | | |
    | Single camera streaming | [`gst-camera-single-stream-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-single-stream-example) | Single camera: preview, encode, YUV dump, or RTSP. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | Multi-camera streaming | [`gst-multi-camera-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-multi-camera-example) | Concurrent streaming from two cameras. | ✓ | ✓ | ✓ | × | ✓ | TBD | TBD |
    | Multistream | [`gst-multi-stream-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-multi-stream-example) | Single camera duplicated into two streams via tee. | ✓ | ✓ | ✓ | × | ✓ | TBD | TBD |
    | Live snapshot | [`gst-snapshot-stream-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-snapshot-stream-example) | Preview + JPEG snapshot capture simultaneously. | ✓ | × | × | × | ✓ | TBD | TBD |
    | RTMP camera streaming | [`gst-rtmp-stream-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-rtmp-stream-example) | ISP / RTSP camera feed converted to RTMP stream. | ✓ | ✓ | ✓ | × | ✓ | TBD | TBD |
    | Activate/deactivate streams at runtime | [`gst-activate-deactivate-streams-runtime-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-activate-deactivate-streams-runtime) | Change resolution on-the-fly without sensor restart. | ✓ | ✓ | ✓ | × | ✓ | TBD | TBD |
    | Add/remove streams at runtime | [`gst-add-remove-streams-runtime`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-add-remove-streams-runtime) | Dynamically add and remove streams. | ✓ | ✓ | ✓ | × | ✓ | TBD | TBD |
    | Camera frame processing | [`gst-appsink-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-appsink-example) | Apply transforms to camera frames before encoding. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | Camera metadata | [`gst-camera-metadata-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-metadata-example) | Capture scene mode, white balance, and bit rate metadata. | ✓ | ✓ | ✓ | × | ✓ | TBD | TBD |
    | Camera stream configuration | [`gst-add-streams-as-bundle-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-add-streams-as-bundle-example) | Configure camera streams as a bundle. | ✓ | ✓ | ✓ | × | ✓ | TBD | TBD |
    | Camera burst capture | [`gst-camera-burst-capture-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-burst-capture-example) | Burst-mode snapshot capture. | ✓ | × | × | × | ✓ | TBD | TBD |
    | Camera switch | [`gst-camera-switch-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-switch-example) | Switch between two cameras without stopping pipeline. | ✓ | ✓ | ✓ | × | ✓ | TBD | TBD |
    | SHDR / LDC / EIS | [`gst-camera-shdr-ldc-eis-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-shdr-ldc-eis-example) | Super HDR, lens distortion correction, and image stabilization. | ✓ | × | × | × | × | TBD | TBD |
    | Decode JPEG images | [`gst-jpg-decode-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-jpg-decode-example) | Decode JPEG images and preview on screen. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | FatsCV Padding Strip | [`gst-fastcv-padding-strip-example`](https://github.com/qualcomm/gst-plugins-imsdk/tree/main/gst-sample-apps/gst-fastcv-padding-strip-example) | Padding strpiped and Dumped to a file | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | GstD-camera-single-stream-example | \[`GstD-camera-single-stream-example`] ([https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/GstD-camera-single-stream-example](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/GstD-camera-single-stream-example)) | Single camera using GstD: preview, encode or YUV dump. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | **Video** | | | | | | | | | |
    | Video Wall | [`gst-concurrent-videoplay-composition`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-concurrent-videoplay-composition) | Concurrent video playback composed on display. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | Smart codec | [`gst-smartcodec-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-smartcodec-example) | Reduce bandwidth / storage for camera input. | ✓ | ✓ | ✓ | × | ✓ | TBD | TBD |
    | Multiformat encode/decode | [`gst-videocodec-concurrent-playback`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-videocodec-concurrent-playback) | Concurrent decode and playback of multiple formats. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | Audio-video playback | [`gst-audio-video-playback`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-audio-video-playback) | Simultaneous audio and video decode and playback. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | Video transcoding | [`gst-video-transcode-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-video-transcode-example) | Transcode AVC↔HEVC. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | Video playback | [`gst-video-playback-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-video-playback-example) | Decode and play back a video file. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | Record and preview transformed video | [`gst-transform-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-transform-example) | Rotate, flip, and scale a video stream. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | Video composition using Weston | [`gst-weston-composition-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-weston-composition-example) | Compose live camera and offline file sources. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | Encode audio and video | [`gst-audio-video-encode`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-audio-video-encode) | Encode audio and video to a file. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | **Audio** | | | | | | | | | |
    | Audio playback | [`gst-audio-decode-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-audio-decode-example) | Decode and play back WAV, MP3, or FLAC. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | Audio recording | [`gst-audio-encode-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-audio-encode-example) | Record and encode audio to FLAC or WAV. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
  </Tab>
</Tabs>

***

## Config 1: Supported Camera Slots by Platform

<Tabs>
  <Tab title="QCS6490">
    <Note>
      Connect the IMX577 MIPI camera sensor to the CAM3 slot of the RB3 device.
    </Note>
  </Tab>

  <Tab title="IQ-9075-EVK">
    <Note>
      Connect the IMX577 MIPI camera sensor to the CAM1 slot of the IQ-9075-EVK device.
    </Note>
  </Tab>

  <Tab title="IQ-8275-EVK">
    <Note>
      Connect the IMX577 MIPI camera sensor to the CAM1 slot of the IQ-8275-EVK device.
    </Note>
  </Tab>

  <Tab title="IQ615">
    <Note>
      Connect the IMX577 MIPI camera sensor to the CAM1 slot of the IQ615 device.
    </Note>
  </Tab>

  <Tab title="IQx7181">
    <Note>
      Connect the IMX577 MIPI camera sensor to the CAM1 (CSI1 22-pin) slot of the IQx7181 device.
    </Note>
  </Tab>
</Tabs>

***

<Note>
  For camera and video sample applications that require a YUV player, see the [Sourceforge YUV player project](https://sourceforge.net/projects/raw-yuvplayer/).
</Note>

## Prerequisites

<Accordion title="Try me">
  <Note>
    **Note:** Some of the steps in the pre-requisties will be removed from future releases once the necessary fixes are mainlined.
  </Note>

  <Steps>
    <Step title="Set up Wi-Fi">
      Connect to the Wireless Access Point (Wi-Fi Router):

      ```bash theme={null}
      nmcli dev wifi connect <WiFi-SSID> password <WiFi-password>
      ```

      Check the connection and device status:

      ```bash theme={null}
      nmcli -p device
      ```

      **Login to the target device**

      Locate the IP address of the device according to the type of network connection,
      using the UART console on the Linux host:

      For Ethernet:

      ```bash theme={null}
      ip address show eth2
      ```

      For Wi-Fi:

      ```bash theme={null}
      ip address show wlp1s0
      ```

      Use the IP address from the Linux host to establish an SSH connection to the device:

      ```bash theme={null}
      ssh root@<ip-address>
      ```

      Example:

      ```bash theme={null}
      ssh root@192.168.0.222
      ```

      Connect to the SSH shell using the following password:

      ```bash theme={null}
      oelinux123
      ```
    </Step>

    <Step title="Set up Wi-Fi">
      Connect to the Wireless Access Point (Wi-Fi Router):

      ```bash theme={null}
      nmcli dev wifi connect <WiFi-SSID> password <WiFi-password>
      ```

      Check the connection and device status:

      ```bash theme={null}
      nmcli -p device
      ```

      **Login to the target device**

      Locate the IP address of the device according to the type of network connection,
      using the UART console on the Linux host:

      For Ethernet:

      ```bash theme={null}
      ip address show eth2
      ```

      For Wi-Fi:

      ```bash theme={null}
      ip address show wlp1s0
      ```

      Use the IP address from the Linux host to establish an SSH connection to the device:

      ```bash theme={null}
      ssh root@<ip-address>
      ```

      Example:

      ```bash theme={null}
      ssh root@192.168.0.222
      ```

      Connect to the SSH shell using the following password:

      ```bash theme={null}
      oelinux123
      ```
    </Step>

    <Step title="Download Models and Artifacts">
      On the target device, obtain the `download_artifacts.sh` script, set executable
      permissions, and run it to download the model, media, and label files:

      ```bash theme={null}
      cd /tmp/
      curl -L -O https://raw.githubusercontent.com/quic/sample-apps-for-qualcomm-linux/refs/heads/main/qualcomm-linux/scripts/download_artifacts.sh
      chmod +x download_artifacts.sh
      ./download_artifacts.sh
      ```
    </Step>

    <Step title="Enable qticamsrc">
      In the terminal of the target device, run the following command to enable the `qticamsrc` on Config #2:

      ```bash theme={null}
      echo -n "camx" > /var/data
      efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -w -f /var/data
      efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -p
      sync
      reboot
      ```
    </Step>

    <Step title="Enable libcamera">
      For Dragonwing RB3 Gen 2 Development Kit, enable the libcamera using the following instructions:

      <Note>
        **Note:** The libcamera plugin supports only the IMX577 camera sensor. Connect the IMX577 sensor before enabling libcamera.
      </Note>

      In the terminal of the target device, enable the bootloader mode using the
      following command:

      ```bash theme={null}
      reboot bootloader
      ```

      Once the device enters bootloader mode, flash the Vision Kit CDT file from
      the extracted folder:

      ```bash theme={null}
      fastboot flash cdt cdt_vision_kit.bin
      ```

      <Note>
        **Note:** You can obtain the `cdt_vision_kit.bin` file from [Qualcomm multimedia proprietary image](https://artifacts.codelinaro.org/ui/native/qli-ci/flashable-binaries/meta-qcom/rb3gen2-core-kit/qli-2.0-rc3-qcom-multimedia-proprietary-image.zip) at `images/rb3gen2-core-kit/qcom-multimedia-proprietary-image-rb3gen2-core-kit`:
      </Note>

      Reboot the device:

      ```bash theme={null}
      fastboot reboot
      ```
    </Step>

    <Step title="Enable Audio and GPU Delegate">
      In the terminal of the target device, run the following command to enable audio:

      ```bash theme={null}
      systemctl stop pipewire wireplumber pipewire.socket pipewire-manager.socket
      chmod 777 /dev/dma_heap/system
      adsprpcd audiopd &
      systemctl start pipewire wireplumber
      wpctl status
      ```

      To set the default devices for sink and source, get the device numbers from
      `wpctl status` and run the following command:

      ```bash theme={null}
      wpctl set-default <device ID>
      ```

      In the terminal of the target device, run the following command to enable the
      GPU delegate and backend:

      ```bash theme={null}
      ln -sf /usr/lib/libOpenCL.so.1 /usr/lib/libOpenCL.so
      export OCL_ICD_FILENAMES=/usr/lib/libOpenCL_adreno.so.1
      ```
    </Step>

    <Step title="Known Issue: Camera basic use-case on IQ8">
      To resolve this issue, apply the following workaround.

      For prebuilt users: Run the command below on the device shell before launching the Camera use case:

      ```bash theme={null}
      cp /usr/share/qcom/sa8775p/Qualcomm/SA8775P-RIDE/dsp/cdsp/libbitml_nsp_73nb_skel.so /usr/share/qcom/qcs8300/Qualcomm/IQ8275-EVK/dsp/cdsp/.
      ```
    </Step>
  </Steps>
</Accordion>

## Camera Applications

### Single Camera Streaming

**Description:** The [**gst-camera-single-stream-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-single-stream-example) application demonstrates single camera use cases with possible outputs: live camera preview on display, store video encoder output, dump the camera YUV, and live RTSP streaming.

* Live camera preview on display
* Encode MP4 saved to file
* Encode YUV saved to file
* RTSP use case
* **Usage Example:** For example, you can use this application in a security system where the goal is to capture a video from one camera and stream it across a network. You can either view it or archive it for future review.
  The following figure shows the pipeline flow for various single camera stream use cases.For more information, see [pipeline flow.](#single-camera-pipeline-flow)
  <img alt="Single camera pipeline" />
  **Figure: gst-camera-single-stream-example pipeline**

<Note>
  **Note:** The **camera-source-bin** uses the **libcamerasrc** and **qtivtransform** plugins in Config #1 and the **qticamsrc** plugin in the Config #2 to generate the output stream.
</Note>

**Application:** [`gst-camera-single-stream-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-single-stream-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="In the terminal of the target device, select a use case and run the command">
      **Live camera use case:** View the live camera on Waylandsink:

      ```bash theme={null}
      gst-camera-single-stream-example -o 0 -w 1920 -h 1080
      ```

      **MP4 use case:** Dump the MP4 to the `$HOME/Downloads/qimsdk_samples/media/` folder:

      ```bash theme={null}
      gst-camera-single-stream-example -o 1 -w 1920 -h 1080
      ```

      **YUV use case:** Dump the YUV data to the `$HOME/Downloads/qimsdk_samples/media/` folder:

      ```bash theme={null}
      gst-camera-single-stream-example -o 2 -w 1920 -h 1080
      ```

      **RTSP use case:** View the RTSP stream (run the RTSP server) on the Linux host computer:

      1. Run the RTSP server in a separate console on the target device with udpsrc (you can run it in the background as a service):

      ```bash theme={null}
      gst-rtsp-server  -p 8900 -a <ip_address> -m /live "( udpsrc name=pay0 port=<port_num> caps=\"application/x-rtp,media=video,clock-rate=90000,encoding-name=H264,payload=96\" )"
      ```

      For Example:

      ```bash theme={null}
      gst-rtsp-server -p 8900 -a 10.92.182.47 -m /live "( udpsrc name=pay0 port=8554 caps=\"application/x-rtp,media=video,clock-rate=90000,encoding-name=H264,payload=96\" )"
      ```

      2. Then start single camera streaming:

      ```bash theme={null}
      gst-camera-single-stream-example -o 3 -w 1280 -h 720 -i <ip_address> -p <port_num>
      ```
    </Step>

    <Step title="In the terminal of the host computer, run one of the following commands to view the stream">
      **For VLC:**

      ```bash theme={null}
      vlc -vvv rtsp://<ip_address>:8900/live
      ```

      **For FFplay:**

      ```bash theme={null}
      ffplay rtsp://<ip_address>:8900/live
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-camera-single-stream-example --help
      ```
    </Step>

    <Step title="In the terminal of the host computer, run the following command to pull the files from the target device:">
      ```bash theme={null}
      scp root@<IP address of target device>:$HOME/Downloads/qimsdk_samples/media/<filename to pull> <destination directory>
      ```
    </Step>
  </Steps>

  ### Expected output

  The following table lists the expected output based on the selected output type.

  <img alt="Camera Preview Output" />

  **Figure2: Expected output for gst-camera-single-stream-example**

  **Table:Expected output for gst-camera-single-stream-example**

  | Output | Description |
  | - | - |
  | Display | Preview the stream. |
  | Dump the MP4 and YUV data | Save the output to a file at `$HOME/Downloads/qimsdk_samples/media/`. |
  | RTSP | View the output on a remote display. |

  ### Pipeline flow

  The following table lists the plugins used in the single camera streaming pipeline:

  <table>
    <thead>
      <tr>
        <th>Pipeline</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>Live camera preview on display</td>

        <td>
          <ol>
            <li>`qticamsrc` captures the camera live stream.</li>
            <li>`Capsfilter` is applied to enforce constraints on the raw video data.</li>
            <li>The data is sent to the Wayland display sink (`waylandsink`) for live preview.</li>
          </ol>
        </td>
      </tr>

      <tr>
        <td>Video encoding</td>

        <td>
          <ol>
            <li>`qticamsrc` captures the camera live stream.</li>
            <li>`Capsfilter` is applied to enforce constraints on the raw video data.</li>
            <li>The data is sent to the Wayland display sink (`waylandsink`) for live preview.</li>
            <li>`v4l2h264enc` is used to encode the video using the H.264 format.</li>
            <li>`H264parse` is used to parse the video.</li>
            <li>`Mp4mux` is used to multiplex the video into an MP4 container.</li>
            <li>`Filesink` is used to write the video to a file.</li>
          </ol>
        </td>
      </tr>

      <tr>
        <td>Dump the camera YUV to a filesink</td>

        <td>
          <ol>
            <li>`qticamsrc` captures the camera live stream.</li>
            <li>`Capsfilter` is applied to enforce constraints on the raw video data.</li>
            <li>`Filesink` is used to dump the YUV data into a file.</li>
          </ol>
        </td>
      </tr>

      <tr>
        <td>RTSP streaming</td>

        <td>
          <ol>
            <li>`qticamsrc` captures the camera live stream.</li>
            <li>`Capsfilter` is applied to enforce constraints on the raw video data.</li>
            <li>`v4l2h264enc` is used to encode the video using the H.264 format.</li>
            <li>`H264parse` is used to parse the video, rtph264pay is used to load the stream to RTSP, and then the stream is
            sent to a UDP sink for RTSP streaming.</li>
          </ol>
        </td>
      </tr>
    </tbody>
  </table>
</Accordion>

***

### Multi-Camera

**Description:** The [**gst-multi-camera-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-multi-camera-example) application allows you to stream from two camera sensors simultaneously. The application composes the camera feeds side by side to display on a screen or encodes and stores the video streams to files.

A few use cases that need many camera inputs are dash camera or stereo camera. You can use this application as a reference and build your use case. For example, a security system where the goal is to capture a video from several cameras. You can either view it or archive it for a future review.

The following figure shows the pipeline for several camera streams. For more information about the elements used in this pipeline, see [pipeline flow.](#multi-camera-pipeline-flow)

<img alt="Multi camera pipeline" />

**Figure: gst-multi-camera-stream-example pipeline**

**Application:** [`gst-multi-camera-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-multi-camera-example)

<Accordion title="Try me">
  ### Run the application

  <Note>
    **Note:** In the following commands, provide the necessary parameters, such as width, height, and output type. The width and height changes are applicable to the primary camera only.
  </Note>

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="In the terminal of the target device, select any of the following use cases and run the respective command:">
      **Waylandsink use case:** View the Waylandsink output:

      ```bash theme={null}
      gst-multi-camera-example -o 0 -w 1920 -h 1080
      ```

      **Encoder use case:** View the encoder output:

      ```bash theme={null}
      gst-multi-camera-example -o 1 -w 1920 -h 1080
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-multi-camera-example --help
      ```
    </Step>

    <Step title="In the terminal of the host computer, run the following command to pull the files from the target device:">
      ```bash theme={null}
      scp root@<IP address of target device>:$HOME/Downloads/qimsdk_samples/media/<filename to pull>
      ```
    </Step>
  </Steps>

  ### Expected Output

  The output is displayed on the screen and saved to a file.

  * If the output type is display, you can preview the stream on the display.

  * If the output type is video encoding, then the encoded files are saved at
    `$HOME/Downloads/qimsdk_samples/media/cam1_vid.mp4` and `$HOME/Downloads/qimsdk_samples/media/cam2_vid.mp4` for camera 1 and camera 2 respectively.

  <img alt="Camera Preview Output" />

  **Figure: Expected output for gst-multi-camera-example application–preview**

  ### Pipeline flow

  The following table lists the plugins used in the multi camera streaming pipeline:

  <table>
    <thead>
      <tr>
        <th>Pipeline</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>Preview on display</td>

        <td>
          <ol>
            <li>`qticamsrc` captures video from both camera0 and camera1.</li>
            <li>`Capsfilter` is applied to enforce constraints on the raw video data.</li>
            <li>`qtivcomposer` composites the video streams and sends the composited video data to Wayland display sink.</li>
            <li>`waylandsink` shows the live preview.</li>
          </ol>
        </td>
      </tr>

      <tr>
        <td>Encoder dump on the device</td>

        <td>
          <ol>
            <li>`qticamsrc` captures video from both camera0 and camera1.</li>
            <li>`Capsfilter` is applied to enforce constraints on the raw video data.</li>
            <li>`v4l2h264enc` is used to encode the video using the H.264 format.</li>
            <li>`H264parse` is used to parse the video.</li>
            <li>`Mp4mux` is used to multiplex the video into an MP4 container.</li>
            <li>`Filesink` is used to write the video to a file.</li>
          </ol>
        </td>
      </tr>
    </tbody>
  </table>
</Accordion>

***

### Camera Switch

**Description:** The [**gst-camera-switch-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-switch-example) application allows you to switch between two cameras.

The following figure shows the pipeline, which demonstrates the camera switch between the two cameras of the device without changing the state of the pipeline. The switching is done in the Playing state after every 5 seconds.

For more information about the plugins used in this pipeline, see [Pipeline flow.](#camera-switch-pipeline-flow)

<img alt="camera switch pipeline" />

**Figure: gst-camera-switch-example pipeline**

**Application:** [`gst-camera-switch-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-switch-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="In the terminal of the target device, run the respective command for any of the following use cases:">
      **Camera switch:**

      ```bash theme={null}
      gst-camera-switch-example <OPTION>
      ```

      **Display the stream after switching (default option):**

      ```bash theme={null}
      gst-camera-switch-example -d 1
      ```

      **Encode the stream after switching:**

      ```bash theme={null}
      gst-camera-switch-example -d 0
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-camera-switch-example --help
      ```
    </Step>

    <Step title="In the terminal of the host computer, run the following command to pull the files from the target device:">
      ```bash theme={null}
        scp root@<IP address of target device>:$HOME/Downloads/qimsdk_samples/media/<filename to pull>
      ```
    </Step>
  </Steps>

  ### Expected output

  The application generates the output as a preview or an encoded MP4 file.

  <img alt="Camera Preview Output" />

  **Figure: Expected output for gst-camera-switch-example application**

  ### Pipeline flow

  The following table lists the plugins used in the camera switch pipeline:

  <table>
    <thead>
      <tr>
        <th>Pipeline</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>Display preview</td>

        <td>
          <ol>
            <li>`qticamsrc` captures the camera live stream and uses capsfilter to create a branch for preview.</li>
            <li>The data is sent to the Wayland display sink (`waylandsink`) for preview.</li>
          </ol>
        </td>
      </tr>

      <tr>
        <td>Encode as MP4 file</td>

        <td>
          <ol>
            <li>`qticamsrc` captures the camera live stream and uses capsfilter to create a branch for encoding.</li>
            <li>The stream data is encoded.</li>
            <li>`H264parse` is used to parse the captured data.</li>
            <li>`Mp4mux` is used to multiplex the data.</li>
          </ol>
        </td>
      </tr>
    </tbody>
  </table>
</Accordion>

***

### Multi-Stream

**Description:** The [**gst-multi-stream-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-multi-stream-example) The gst-multi-stream-example application allows you to access several streams from a single camera and either display them simultaneously on a screen or save them in MP4 format.
For example, you can use this application for a camera security system where the video is captured from a camera. This video can be simultaneously streamed over a network, displayed on a monitor, and archived for future review.

The figure shows the pipeline, which is run with flags to specify width, height, stream count, and output file.
For more information, see [Pipeline flow.](#multi-stream-pipeline-flow)

<img alt="camera switch pipeline" />

**Figure: gst-multi-stream-example pipeline**

**Application:** [`gst-multi-stream-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-multi-stream-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="In the terminal of the target device, run the use case:">
      ```bash theme={null}
      gst-multi-stream-example -w 1920 -h 1080 -n 2
      ```
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-multi-stream-example --help
      ```
    </Step>

    <Step title="In the terminal of the host computer, run the following command to pull the files from the target device:">
      ```bash theme={null}
      scp root@<IP address of target device>:$HOME/Downloads/qimsdk_samples/media/<filename to pull>
      ```
    </Step>
  </Steps>

  ### Expected output

  The output streams are displayed simultaneously on the screen or saved to a file.

  <img alt="Camera Preview Output" />

  **Figure: Expected output for gst-multi-stream-example application - Preview**

  ### Pipeline flow

  The following table lists the plugins used in the multistream pipeline:

  <table>
    <thead>
      <tr>
        <th>Plugin</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>qticamsrc</td>

        <td>
          <ol>
            <li><code>qticamsrc</code> captures the video stream and feeds it into the capsfilter.</li>
          </ol>
        </td>
      </tr>

      <tr>
        <td>Capsfilter</td>

        <td>
          <ul>
            <li>Branches the source into two separate pipelines. In one pipeline, the output is sent to <code>waylandsink</code> for display.</li>

            <li>
              In the other pipeline, the video undergoes the following:

              <ol>
                <li>Encoded using <code>v4l2h264enc</code>.</li>
                <li>Parsed with <code>h264parse</code>.</li>
                <li>Multiplexed into MP4 format with <code>mp4mux</code>.</li>
                <li>Written to a file using <code>filesink</code>.</li>
              </ol>
            </li>
          </ul>
        </td>
      </tr>
    </tbody>
  </table>

  ### Known Issues

  * The default bit rate for the encoded video is configured to a low setting, leading to reduced quality at higher resolutions.
  * The application may not work due to v4l2h264enc encoder issue.
</Accordion>

***

### Wide dynamic range and electronic image stabilization

**Description:** The [**gst-camera-shdr-ldc-eis-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-shdr-ldc-eis-example) application shows the staggered high dynamic range (sHDR), lens distortion correction (LDC), and electronic image stabilization (EIS) camera imaging features. For more information, see Overview of camera imaging features.

<Note>
  **Note:** This sample application isn’t supported on Dragonwing IQ-8275 and Dragonwing IQ-9075.
</Note>

The following figure shows the pipeline, where the input is received from the camera and the output is displayed using Waylandsink. For more information about the pipeline, see [Pipeline flow.](#dynamic-range-pipeline-flow)

<img alt="wide range pipeline" />

* For sHDR, ensure that the Sensor mode for Digital Overlap mode is enabled and tuned.
* For LDC and EIS, ensure that the sensor and hardware are tuned.

  **Application:** [`gst-camera-shdr-ldc-eis-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-shdr-ldc-eis-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="In the terminal of the target device, run the respective command for any of the following use cases to display the output:">
      **sHDR**

      ```bash theme={null}
      gst-camera-shdr-ldc-eis-example -s 1 -w 1920 -h 1080
      ```

      <Note>
        **Note:** This use case isn’t supported on the Ubuntu Server.
      </Note>

      **LDC**

      ```bash theme={null}
      gst-camera-shdr-ldc-eis-example -l 1 -w 1920 -h 1080
      ```

      **EIS**

      ```bash theme={null}
      gst-camera-shdr-ldc-eis-example -e 1 -w 1920 -h 1080
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-camera-shdr-ldc-eis-example --help
      ```
    </Step>
  </Steps>

  ### Expected output

  * sHDR
    <img alt="without sHDR" />
    **Figure: Output without sHDR**
    <img alt="with sHDR" />
    **Figure: Output with sHDR**
  * LDC
    <img alt="without LDC" />
    **Figure: Output without LDC**
    <img alt="with LDC" />
    **Figure: Output with LDC**

  ### Pipeline flow

  The following table lists the plugins used in the gst-camera-shdr-ldc-eis-example pipeline:

  <table>
    <thead>
      <tr>
        <th>Plugin</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>qticamsrc</td>

        <td>
          <ol>
            Captures the camera feed and sends it to the capsfilter.
          </ol>
        </td>
      </tr>

      <tr>
        <td>Capsfilter</td>

        <td>
          <ol>
            Passes the feed for display.
          </ol>
        </td>
      </tr>

      <tr>
        <td>waylandsink</td>

        <td>
          <ol>
            Shows the output on the Wayland display.
          </ol>
        </td>
      </tr>
    </tbody>
  </table>

  ### Overview of camera imaging features

  The following table provides a brief description of the supported imaging features:

  **Table : Camera imaging features**

  <table>
    <thead>
      <tr>
        <th>Feature</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>sHDR</td>

        <td>
          <ol>
            * The video high dynamic range feature is supported on camera sensors that operate in the Digital Overlap mode.
            * The fusion of the low and high exposure frames is enabled in the processor GPU.
            * Use the sHDR flag to enable the sHDR mode.
          </ol>
        </td>
      </tr>

      <tr>
        <td>LDC</td>

        <td>
          <ol>
            * A GPU-based LDC algorithm is supported to rectify barrel distortion artifacts that are inherent with wide field of vision (FoV) lens.
            * Use the LDC flag to enable the LDC mode.
          </ol>
        </td>
      </tr>

      <tr>
        <td>EIS</td>

        <td>
          <ol>
            * A GPU-based EIS algorithm is supported.
            * Use the EIS flag to enable the EIS mode.
          </ol>
        </td>
      </tr>
    </tbody>
  </table>
</Accordion>

***

### Snapshot Stream

**Description:** The [**gst-snapshot-stream-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-snapshot-stream-example) application allows you to generate two streams: a preview stream for live display and a snapshot stream that captures images and saves them as JPEG files.
For the snapshot stream, you can determine the number of snapshots.

<Note>
  **Note:** This sample application isn’t supported on Dragonwing IQ-8275 and Dragonwing IQ-9075.
</Note>

The following figure shows the pipeline, which allows for simultaneous live preview and snapshot capture from the same video source.
For more information, see [Pipeline flow.](#snapshot-pipeline-flow)

<img alt="Snapshot pipeline" />

**Figure: gst-snapshot-stream-example pipeline**

**Application:** [`gst-snapshot-stream-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-snapshot-stream-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="In the terminal of the target device, use the following command to run the use case. Specify the application with parameters, such as camera width, height, and the maximum snapshot count for dumping JPEG files.">
      ```bash theme={null}
      gst-snapshot-stream-example -W 1280 -H 720 -w 3840 -h 2160 -c 5
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-snapshot-stream-example --help
      ```
    </Step>
  </Steps>

  ### Expected output

  The application generates an output for preview on the display and dumps the JPEG files in the respective location on the device.
  After the use case stops, the snapshot output files are available at the `$HOME/Downloads/qimsdk_samples/media/` path.

  ### Pipeline flow

  The following table lists the plugins used in the live snapshot pipeline:

  <table>
    <thead>
      <tr>
        <th>Plugin</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>qticamsrc</td>

        <td>
          <ol>
            <li><code>qticamsrc</code> captures the video stream and feeds it into the capsfilter.</li>
          </ol>
        </td>
      </tr>

      <tr>
        <td>Capsfilter</td>

        <td>
          <ul>
            <li>Branches the source into two separate pipelines. In one pipeline, the constraints on the raw video data are enforced, and then the video data is sent to `waylandsink` for live preview.</li>

            <li>
              In the other pipeline:

              <ol>
                <li>The frames are written to multiple files using multi filesink.</li>
                <li>The snapshots or frames from the video stream are saved to a disk.</li>
              </ol>
            </li>
          </ul>
        </td>
      </tr>
    </tbody>
  </table>
</Accordion>

***

### Camera frame processing

**Description:** The [gst-appsink-example](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-appsink-example) application allows you to extract the buffer samples from a camera pipeline. These buffers can configure the camera resolution with the options such as width and height.

For example, the appsink plugin can be used for a multimedia application that processes the video frame samples in real-time. Filters or transformations can be applied to the data before playing it back.

The following figure shows the pipeline, which configures the camera resolution using parameters such as width and height. When appsink emits a signal indicating the availability of a new sample in the pipeline, the application extracts the buffer from this sample.

For information about the plugins used in this pipeline, see [Pipeline flow.](#appsink-pipeline)

<img alt="Appsink pipeline : gst-appsink-example pipeline" />

**Figure : gst-appsink-example pipeline**

**Application:** [`gst-appsink-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-appsink-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites.](#prerequisites)
    </Step>

    <Step title="In the terminal of the target device, run the use case">
      ```bash theme={null}
      gst-appsink-example -w 1920 -h 1080
      ```
    </Step>

    <Step title="To stop the running use case">
      Press `CTRL + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-appsink-example --help
      ```
    </Step>
  </Steps>

  ### Expected output

  The user interface displays the following message:

  ```text theme={null}
  Received a buffer, doing some processing.
  ```

  <img alt="Appsink output : Expected output for gst-appsink-example application" />

  **Figure : Expected output for gst-appsink-example application**

  ### Pipeline flow

  The following table lists the plugins used in the camera frame processing pipeline:

  | Plugin | Description |
  | - | - |
  | `qticamsrc` | Captures the camera feed and sends it into the capsfilter. |
  | `Capsfilter` | Ensures that the video stream is in the correct format. |
  | `Appsink` | Provides the video data as an output to an application. |
</Accordion>

***

### Camera metadata

**Description:** The [gst-camera-metadata-example](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-metadata-example) application starts with a menu that allows you to choose between multiple properties of the camera plugin such as `video-metadata`, `static-metadata`, and `image-metadata`.

A practical use case for this application is handling camera metadata such as white balance, International Organization for Standardization (ISO) sensitivity, and exposure to tune the camera.

The following figure shows the execution of appsink when a new sample is available in the pipeline. The application is run from the command-line with specific options for custom pipeline, display type, and timestamp location.

For information about the plugins used in this pipeline, see [Pipeline flow.](#camera-metadata-pipeline)

<img alt="camera metadata : gst-camera-metadata-example pipeline" />

**Figure : gst-camera-metadata-example pipeline**

**Application:** [`gst-camera-metadata-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-metadata-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites.](#prerequisites)
    </Step>

    <Step title="In the terminal of the target device, run the following use case:">
      Camera metadata use case with output on Waylandsink:

      ```bash theme={null}
      gst-camera-metadata-example
      ```

      Camera metadata use case without Waylandsink:

      ```bash theme={null}
      gst-camera-metadata-example -d 0
      ```

      Dump metadata of saturation:

      ```bash theme={null}
      gst-camera-metadata-example -d 0 -r "org.codeaurora.qcamera3. saturation use_saturation"
      ```

      Dump metadata of exposureTime:

      ```bash theme={null}
      gst-camera-metadata-example -d 0 -r "spectra.sensor exposureTime"
      ```
    </Step>

    <Step title="To stop the use case">
      Press `CTRL + C`.
    </Step>

    <Step
      title="In the terminal of the target device, run the following command to display the available help options:

"
    >
      ```bash theme={null}
      gst-camera-metadata-example --help
      ```
    </Step>
  </Steps>

  ### Expected output

  * During the execution, the application requires the command-line options to output the camera metadata and options available to set or get the vendor tags.
  * On successful execution of the application, you can find and dump the respective metadata information.

  <img alt="camera metadata output : Expected output for gst-camera-metadata-example application" />

  **Figure : Expected output for gst-camera-metadata-example application**

  ### Pipeline flow

  The following table lists the plugins used in the camera metadata pipeline:

  | Plugin | Description |
  | - | - |
  | `qticamsrc` | Captures the camera feed and sends it into the capsfilter. |
  | `Capsfilter` | Passes the feed to waylandsink for display. If display option isn’t provided, passes the feed to appsink. |

  The application extracts the buffer from the sample for further processing. During execution, the application expects an argument on which the following options should be displayed on the console.

  * Video metadata
  * Image metadata
  * Static metadata
  * Session metadata

  Based on the options selected, the application displays the output on the console.

  ### Known issue

  When the GetTag menu option is selected, the capture options don’t work.
</Accordion>

***

### Camera stream configuration

**Description:** The [gst-add-streams-as-bundle-example](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-add-streams-as-bundle-example) application allows you to activate and configure camera streams as a bundle instead of configuring each stream one at a time thus reducing the delay in configuring new streams.

The following figure shows the pipeline, which takes the configuration parameters such as width and height, processes to either display on a screen, or encodes and saves to a file. For more information about the plugins used in the pipeline, see [Pipeline flow.](#add-stream-as-bundle)

<img alt="add stream as bundle pipeline: gst-add-streams-as-bundle-example pipeline" />

**Figure : gst-add-streams-as-bundle-example pipeline**

**Application:** [`gst-add-streams-as-bundle-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-add-streams-as-bundle-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites.](#prerequisites)
    </Step>

    <Step title="In the terminal of the target device,run the use case:">
      ```bash theme={null}
      gst-add-streams-as-bundle-example <OPTION>
      ```
    </Step>

    <Step title="Run for various use cases">
      Preview the stream:

      ```bash theme={null}
      gst-add-streams-as-bundle-example -o Display
      ```

      Encode the stream and save it to a file:

      ```bash theme={null}
      gst-add-streams-as-bundle-example -o File
      ```

      The encoded stream is saved at `$HOME/Downloads/qimsdk_samples/media/video_%d.mp4`.
    </Step>

    <Step title="To stop the running use case">
      Use CTRL + C.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-add-streams-as-bundle-example --help
      ```
    </Step>

    <Step title="In the terminal of the host computer, run the following command to pull the files from the target device:">
      ```bash theme={null}
      scp root@<IP address of target device>:$HOME/Downloads/qimsdk_samples/media/<filename to pull>
      ```
    </Step>
  </Steps>

  ### Expected output

  One stream is sent to display and the other stream is saved to a file.

  <img alt="Add stream as bundle : Expected output for gst-add-streams-as-bundle-example application" />

  **Figure : Expected output for gst-add-streams-as-bundle-example application**

  ### Pipeline flow

  The following table lists the plugins used in the camera stream configuration pipeline:

  | Plugin | Description |
  | - | - |
  | `qticamsrc` | Captures the video stream and feeds it into the capsfilter. |
  | `Capsfilter` | Branches the source into two separate pipelines. In one pipeline, the output is sent to waylandsink for display. In the other pipeline, the video undergoes the following: 1. Encoded using v4l2h264enc. 2. Parsed with h264parse. 3. Multiplexed into an MP4 format with mp4mux. 4. Written to a file using filesink. |

  ### Known issue

  The application doesn't work due to video encoder issue.
</Accordion>

***

### Camera Burst Capture

**Description:** The [**gst-camera-burst-capture-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-burst-capture-example)  application allows you to capture snapshots in the Burst mode. It captures five burst snapshots delayed by a 10-second timer and displays the snapshots on a screen.

<Note>
  **Note:** This sample application isn’t supported on Dragonwing IQ-8275 and Dragonwing IQ-9075.
</Note>

The following figure shows the pipeline, which captures the snapshots and processes them in using one of the following three branches:

* They’re displayed on the screen.
* They’re encoded and saved to a file.
* They’re saved to a buffer and based on the capture format (BAYER/RAW/JPEG), saved to the device storage.

  For more information about the plugins used in the pipeline, see [Pipeline flow.](#burst-capture-pipeline-flow)

  <img alt="camera burst pipeline" />

  **Figure: gst-camera-burst-capture-example pipeline**

**Application:** [`gst-camera-burst-capture-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-camera-burst-capture-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="In the terminal of the target device, run the respective command for any of the following use cases:">
      **Display stream and JPEG capture:**

      ```bash theme={null}
      gst-camera-burst-capture-example -w 1280 -h 720 -p 1 -c 0
      ```

      **Encode stream and JPEG capture:**

      ```bash theme={null}
      gst-camera-burst-capture-example -w 1280 -h 720 -p 0 -c 0
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-camera-burst-capture-example --help
      ```
    </Step>

    <Step title="In the terminal of the host computer, run the following command to pull the files from the target device:">
      ```bash theme={null}
      scp root@<IP address of target device>:$HOME/Downloads/qimsdk_samples/media/<filename to pull>
      ```
    </Step>
  </Steps>

  ### Expected output

  * If you choose to display the stream, the snapshots are displayed on the screen.
  * If you choose to encode the stream, the encoded stream is saved at `$HOME/Downloads/qimsdk_samples/media/video_%d.mp4`.

  ### Pipeline flow

  The following table lists the plugins used in the camera burst capture pipeline:

  <table>
    <thead>
      <tr>
        <th>Pipeline</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>Display preview</td>

        <td>
          <ol>
            <li>`qticamsrc` captures the camera live stream and uses capsfilter to create a branch for preview.</li>
            <li>The data is sent to the Wayland display sink (`waylandsink`) for preview.</li>
          </ol>
        </td>
      </tr>

      <tr>
        <td>Encode and save to a file</td>

        <td>
          <ol>
            <li>`qticamsrc` captures the camera live stream and uses capsfilter to create a branch for encoding.</li>
            <li>The snapshots are encoded.</li>
            <li>H264parse is used to parse the captured data.</li>
            <li>Mp4mux is used to multiplex the data.</li>
          </ol>
        </td>
      </tr>

      <tr>
        <td>BAYER/RAW/JPEG dump</td>

        <td>
          <ol>
            <li>`qticamsrc` captures the camera live stream and uses capsfilter to create a branch for preview.</li>
            <li>Appsink is used to capture callback and save the snapshots in the storage dump.</li>
          </ol>
        </td>
      </tr>
    </tbody>
  </table>

  ### Known issue

  The test cases for 144p and 240p resolutions are failing.
</Accordion>

***

### RTMP Streaming

**Description:** The [**gst-rtmp-stream-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-rtmp-stream-example) application uses the input from ISP or RTSP camera source to generate RTMP streams on a Waylandsink. You can use this application as a reference to develop and configure new applications.
This application is primarily targeted for security systems and media broadcasting, where a video from RTSP source or camera source streams across a network. You can either view the stream or archive it for future review.

The following figure shows the pipeline involved in RTMP streaming with different sources. For more information about the elements used in this pipeline, see [Pipeline flow.](#rtmp-streaming-pipeline-flow)

<img alt="RTMP streaming pipeline" />

**Figure: gst-rtmp-stream-example pipeline**

**Application:** [`gst-rtmp-stream-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-rtmp-stream-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="In the terminal of the target device, download and run the RTMP server:">
      ```bash theme={null}
      docker pull tiangolo/nginx-rtmp
      docker run -d -p 1935:1935 --name nginx-rtmp tiangolo/nginx-rtmp
      ```
    </Step>

    <Step title="In the terminal of the target device, select any of the following use cases and run the respective command:">
      **RTSP camera AVC to RTMP use case:**

      ```bash theme={null}
      gst-rtmp-stream-example  -u 0 -d rtmp://<192.168.1.171/>live/01 -r <rtsp://admin:qualcomm1@192.168.1.252:554/Streaming/Channels/101 ! >
      ```

      **ISP camera to RTMP use case:**

      ```bash theme={null}
      gst-rtmi-camera-stream-example  -u 1 -d rtmp://192.168.1.171/live/01
      ```
    </Step>

    <Step title="In the terminal of the target device, run the following command to connect VLC to the stream:">
      ```bash theme={null}
      rtmp://192.168.1.171/live/01
      ```

      In this command, replace `rtmp://192.168.1.171/live/01` with the IP address of your device.
    </Step>

    <Step title="In the terminal of the target device, run the following commands to stop the RTMP server:">
      ```bash theme={null}
      docker stop nginx-rtmp
      docker rm nginx-rtmp
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-rtmp-stream-example --help

      ```
    </Step>
  </Steps>

  ### Expected output

  The output is displayed on a Waylandsink.

  ### Pipeline flow

  The following table lists the plugins used in the RTMP stream pipeline:

  <table>
    <thead>
      <tr>
        <th>Plugin</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>qticamsrc</td>

        <td>
          <ol>
            <li>`qticamsrc` Captures the camera feed and sends it into the capsfilter.</li>
          </ol>
        </td>
      </tr>

      <tr>
        <td>Capsfilter</td>

        <td>
          <ol>
            <li>Passes the feed to `waylandsink` for display.</li>
            <li>If display option isn’t provided, passes the feed to appsink.</li>
          </ol>
        </td>
      </tr>

      <tr>
        <td>v4l2h264enc</td>

        <td>
          <ol>
            <li>Encodes the video for AVC (H.264) format.</li>
          </ol>
        </td>
      </tr>

      <tr>
        <td>h264parse</td>

        <td>
          <ol>
            <li>Parses the H.264 video.</li>
          </ol>
        </td>
      </tr>

      <tr>
        <td>flvmux</td>

        <td>
          <ol>
            <li>Wraps the video into an FLV container</li>
          </ol>
        </td>
      </tr>

      <tr>
        <td>rtmp2sink</td>

        <td>
          <ol>
            <li>Transmits to an RTMP server for live streaming.</li>
          </ol>
        </td>
      </tr>
    </tbody>
  </table>
</Accordion>

***

### Smart Codec

**Description:** The [**gst-smartcodec-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-smartcodec-example) application reduces bandwidth and storage resources based on scene change estimate thresholds.
The smart codec plugin, `qtismartvencbin`, uses smart fps, smart I-frame, smart bit rate, and smart ROI to do the following:

* Decrease the bit rate without compromising the quality.
* Enable efficient use of network and storage resources.

The following figure shows the pipeline flow for a two-stream use case. For more information, see [Pipeline flow.](#smartcodec-pipeline-flow)

<img alt="Smart codec pipeline" />

**Figure: gst-smartcodec-example pipeline for two streams**

**Application:** [`gst-smartcodec-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-smartcodec-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="In the terminal of the target device, run the use case with a camera source:">
      ```bash theme={null}
      gst-smartcodec-example -w 1920 -h 1080 -o $HOME/Downloads/qimsdk_samples/media/video.mp4
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-smartcodec-example --help
      ```
    </Step>
  </Steps>

  ### Expected output

  The application generates the output as an encoded MP4 file.
  The following table lists the file sizes of a 1080p video with and without smart codec.
  **Table:Smart codec–file sizes**

  | Smart codec | File size |
  | - | - |
  | Enabled | 3718 KB. |
  | Disabled | 17706 KB. |

  ### Pipeline flow

  The following table lists the plugins used in the smart codec pipeline:

  <table>
    <thead>
      <tr>
        <th>Plugin</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>qticamsrc</td>

        <td>
          <ul>
            <li>Captures the video stream.</li>
            <li>Feeds the stream to capsfilter.</li>
          </ul>
        </td>
      </tr>

      <tr>
        <td>Capsfilter</td>

        <td>
          <ul>
            <li>Branches the source into two separate pipelines. In one pipeline, qtismartvencbin processes the output.</li>

            <li>
              <li>In the other pipeline, the video undergoes the following:</li>

              * Encoded using v4l2h264enc.
              * Parsed with h264parse.
              * Multiplexed into an MP4 format with mp4mux.
                * Written to a file using filesink.
            </li>
          </ul>
        </td>
      </tr>
    </tbody>
  </table>
</Accordion>

***

### GstD single camera streaming

**Description:** The [**GstD-camera-single-stream-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/GstD-camera-single-stream-example) shows how to use a menu‑driven GstD application to create and control a single‑camera pipeline with configurable resolution. The application supports live preview, MP4 recording, and raw YUV dumping, and manages the GStreamer pipeline lifecycle cleanly for capture and archival use cases.

The following figure shows the pipeline flow for various single camera stream use case. For more information, see [Pipeline flow.](#gstd-pipeline-flow)

<img alt="GstD pipeline" />

**Figure: GstD-camera-single-stream-example pipeline**

**Application:** [`GstD-camera-single-stream-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/GstD-camera-single-stream-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="In the terminal of the target device, select any of the following use cases and run the respective command:">
      **Live camera use case:** View the live camera on Waylandsink:

      ```bash theme={null}
      GstD-camera-single-stream-example -o 0 -W 1280 -H 720
      ```

      **MP4 use case:** Dump the MP4 to the `/opt` folder:

      ```bash theme={null}
      GstD-camera-single-stream-example -o 1
      ```

      **YUV use case:** Dump the YUV data to the `/opt` folder:

      ```bash theme={null}
      GstD-camera-single-stream-example -o 2 -W 640 -H 480
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      GstD-camera-single-stream-example --help
      ```
    </Step>

    <Step title="In the terminal of the host computer, run the following command to pull the files from the target device:">
      ```bash theme={null}
      scp root@<IP address of target device>:/opt/<filename to pull> <destination directory>
      ```
    </Step>
  </Steps>

  ### Expected output

  The following table lists the expected output based on the selected use case.

  <img alt="GstD camera Output" />

  **Figure: Expected output for gst-camera-single-stream-example**

  **Table: Expected output**

  | Output Type | Description |
  | - | - |
  | Display | Preview the camera stream on display |
  | MP4 | Saves encoded video to `/opt` |
  | YUV | Dumps raw YUV data to `/opt` |
  | RTSP | View the stream on a remote display |

  ### Pipeline flow

  The following table lists the pipelines and their descriptions:

  | Pipeline | Description |
  | - | - |
  | Live camera preview on display | 1. `qtiqmmfsrc` captures the camera live stream.<br />2. Capsfilter is applied to enforce constraints on the raw video data.<br />3. The data is sent to the Wayland display sink (`waylandsink`) for live preview. |
  | Video encoding | 1. `qtiqmmfsrc` captures the camera live stream.<br />2. Capsfilter is applied to enforce constraints on the raw video data.<br />3. The stream is sent to (`waylandsink`) for preview.<br />4. `v4l2h264enc` encodes the video using the H.264 format.<br />5. `h264parse` parses the encoded stream.<br />6. `mp4mux` multiplexes the stream into an MP4 container.<br />7. `filesink` writes the video to a file. |
  | Dump the camera YUV to a filesink | 1. `qtiqmmfsrc` captures the camera live stream.<br />2. Capsfilter is applied to enforce constraints on the raw video data.<br />3. `filesink` dumps the YUV data into a file. |
</Accordion>

***

### FastCV Padding Strip Example

**Description:** The [**gst-fastcv-padding-strip-example**](https://github.com/qualcomm/gst-plugins-imsdk/tree/main/gst-sample-apps/gst-fastcv-padding-strip-example) application shows how to capture live NV12 camera frames, retrieve stride and offset information from `GstVideoMeta`, remove hardware padding, and generate tightly packed NV12 frame dumps for validation. The application supports FastCV-accelerated processing using `fcvScaleu8_v2` and a CPU-based fallback implementation, and provides configurable frame resolution, output location, and dump count options.

The following figure shows the processing flow for the FastCV Padding Strip use case. For more information, see [pipeline flow.](#fastcv-processing-flow)

<img alt="FastCV pipeline" />

**Figure: FastCV Padding Strip Example pipeline**

**Application:** [`gst-fastcv-padding-strip-example`](https://github.com/qualcomm/gst-plugins-imsdk/tree/main/gst-sample-apps/gst-fastcv-padding-strip-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="In the terminal of the target device, select any of the following use cases and run the respective command:">
      **Default use case:** Dump 5 tightly packed NV12 frames.

      ```bash theme={null}
      gst-fastcv-padding-strip-example
      ```

      **Custom resolution use case:** Dump frames at 1920x1080 resolution.

      ```bash theme={null}
      gst-fastcv-padding-strip-example -W 1920 -H 1080
      ```

      **Custom output path use case:** Save output frames to a specified directory.

      ```bash theme={null}
      gst-fastcv-padding-strip-example -o /opt/output_frames
      ```

      **Single-frame dump use case:** Dump only one frame.

      ```bash theme={null}
      gst-fastcv-padding-strip-example -c 1
      ```
    </Step>

    <Step title="To stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-fastcv-padding-strip-example --help
      ```
    </Step>

    <Step title="In the terminal of the host computer, run the following command to pull the generated files from the target device:">
      ```bash theme={null}
      scp root@<IP address of target device>:<output_path>/<filename>.yuv
      ```
    </Step>
  </Steps>

  ### Expected output

  The following table lists the expected output generated by the application.

  **Table: Expected output**

  | Output Type | Description |
  | - | - |
  | YUV Frame Dump | Saves tightly packed NV12 frame data without hardware padding |

  ### Pipeline flow

  The following table lists the processing stages and their descriptions.

  | Pipeline | Description |
  | - | - |
  | Camera frame capture | 1. `qtiqmmfsrc` captures live NV12 camera frames.<br />2. `qtivtransform` processes the frames using `engine=gles`.<br />3. `appsink` receives frames for application processing. |
  | Metadata extraction | 1. The application reads stride and offset information from `GstVideoMeta`.<br />2. The metadata is used to identify hardware-aligned image layout and padding. |
  | Padding removal | 1. Hardware row padding is removed from the captured frame.<br />2. `fcvScaleu8_v2` performs the copy/remap operation when FastCV support is enabled.<br />3. A CPU-based row copy implementation is used when FastCV is unavailable or fails. |
  | Frame dump | 1. The processed tightly packed NV12 frame is written to a `.yuv` file.<br />2. The generated files can be validated using external viewers such as `ffplay`. |
</Accordion>

***

## Dynamic configuration

A common scenario in Edge AI applications is changing the image resolution at runtime for minimal frame loss or gap in the display or adding and removing several streams for camera reconfiguration. The dynamic configuration sample applications support these runtime changes.

### Activate-deactivate streams at runtime

The [gst-activate-deactivate-streams-runtime-example](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-activate-deactivate-streams-runtime) application allows you to change the resolution dynamically without the need to reconfigure the camera sensor.

This application can be used in surveillance systems, videoconferencing, and broadcasting where several live streams are processed in parallel and you can control the active streams dynamically.

The following figure shows the pipeline, where the application creates two streams of different resolutions, and activates and deactivates them at runtime. For information about the plugins used in this pipeline, see [Pipeline flow.](#activate-deactivate)

<img alt="activate deactivate at runtime : gst-activate-deactivate-streams-runtime-example pipeline" />

**Figure : gst-activate-deactivate-streams-runtime-example pipeline**

**Application:** [`gst-activate-deactivate-streams-runtime-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-activate-deactivate-streams-runtime)

<Accordion title="Try me">
  ### Run the application

  The following table lists the application use case modes. The NULL state is the initial state of a GStreamer element.

  <Note>
    **Note:**: Currently, the application is compatible only with the Basic mode. Full mode will be supported in a future release.
  </Note>

  | Use case mode | Description |
  | - | - |
  | `Basic` | Activates or deactivates several streams without entering the NULL state. |
  | `Full` | Activates or deactivates several streams by entering the NULL state. |

  For more information, see GStreamer states.

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites.](#prerequisites)
    </Step>

    <Step title="In the terminal of the target device, run the respective command for any of the following use cases. Before running the application, ensure to configure the use cases and output type parameters.">
      Basic use cases

      Stream on Waylandsink:

      ```bash theme={null}
      gst-activate-deactivate-streams-runtime-example -u Basic -o Display
      ```

      Encode to MP4 file:

      ```bash theme={null}
      gst-activate-deactivate-streams-runtime-example -u Basic -o File
      ```

      Full use cases

      Stream on Waylandsink:

      ```bash theme={null}
      gst-activate-deactivate-streams-runtime-example -u Full -o Display
      ```

      Encode to MP4 file:

      ```bash theme={null}
      gst-activate-deactivate-streams-runtime-example -u Full -o File
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command in the SSH shell to display the available help options:">
      ```bash theme={null}
      gst-activate-deactivate-streams-runtime-example --help
      ```
    </Step>

    <Step title="In the terminal of the host computer, run the following command to pull the files from the target device:">
      ```bash theme={null}
      scp root@<IP address of target device>:$HOME/Downloads/qimsdk_samples/media/<destination directory>
      ```
    </Step>
  </Steps>

  ### Expected output

  During the execution, the first stream is activated followed by the second stream. After a while, the streams deactivate.

  * If the output option is set to `Display`, the output is displayed as a preview.
  * If the output option is set to `File`, the encoded video files are generated.

  <img alt="activate deactivate output : Expected output for gst-activate-deactivate-streams-runtime-example application" />

  **Figure : Expected output for gst-activate-deactivate-streams-runtime-example application**

  ### Pipeline flow

  The following table lists the plugins used in the gst-activate-deactivate-streams-runtime-example pipeline:

  | Plugin | Description |
  | - | - |
  | `qticamsrc` | Captures the camera feed and sends it into the capsfilter. |
  | `Capsfilter` | Passes the feed to waylandsink for display. Passes the feed to filesink to save it to a file. |
</Accordion>

***

### Add-remove streams at runtime

The [gst-add-remove-streams-runtime](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-add-remove-streams-runtime) application allows you to add or remove several camera streams dynamically during runtime.

The following figure shows the pipeline, which demonstrates the ability of the application to add or remove the streams at runtime with camera reconfiguration. It creates three streams and adds or removes them in different order.

For information about the plugins used in the pipeline, see [Pipeline flow.](#add-remove-stream)

<img alt="Figure : gst-add-remove-streams-runtime pipeline" />

**Figure : gst-add-remove-streams-runtime pipeline**

**Application:** [`gst-add-remove-streams-runtime`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-add-remove-streams-runtime)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites.](#prerequisites)
    </Step>

    <Step title="In the terminal of the target device, run the following command to verify if many streams can run simultaneously:">
      ```bash theme={null}
      gst-add-remove-streams-runtime <OPTION>
      ```
    </Step>

    <Step title="Use the following commands to run the application for various use cases:">
      Default mode preview on the display:

      ```bash theme={null}
      gst-add-remove-streams-runtime
      ```

      Preview the stream:

      ```bash theme={null}
      gst-add-remove-streams-runtime -o Display
      ```

      Encode the stream:

      ```bash theme={null}
      gst-add-remove-streams-runtime -o File
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-add-remove-streams-runtime --help
      ```
    </Step>

    <Step title="In the terminal of the host computer, run the following command to pull the files from the target device:">
      ```bash theme={null}
      scp root@<IP address of target device>:$HOME/Downloads/qimsdk_samples/media/<destination directory>
      ```
    </Step>
  </Steps>

  ### Expected output

  * If you choose to preview, the stream is displayed on the screen.
  * If you choose to encode the stream, the encoded stream is saved at `$HOME/Downloads/qimsdk_samples/media/video_%d.mp4`.

  <img alt="Figure : Expected output for gst-add-remove-streams-runtime application–Preview" />

  **Figure : Expected output for gst-add-remove-streams-runtime application–Preview**

  ### Pipeline flow

  The following table lists the plugins used in the gst-add-remove-streams-runtime pipeline:

  <table>
    <thead>
      <tr>
        <th>`Plugin`</th>
        <th>`Description`</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>qticamsrc</td>
        <td>Captures the video stream and feeds it into the `capsfilter`.</td>
      </tr>

      <tr>
        <td>Capsfilter</td>

        <td>
          Branches the source into two separate pipelines. In one pipeline, the output is sent to `waylandsink` for display.
        </td>
      </tr>

      <tr>
        <td>
          <ul>
            <li>In the other pipeline, the video undergoes the following:</li>
          </ul>
        </td>
      </tr>

      <tr>
        <td>
          <ol>
            <li>Encoded using `v4l2h264enc`.</li>
            <li>Parsed with `h264parse`.</li>
            <li>Multiplexed into an MP4 format with `mp4mux`.</li>
          </ol>
        </td>
      </tr>
    </tbody>
  </table>
</Accordion>

***

## Video and Audio Applications

### Video playback

**Description:** The [**gst-video-playback-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-video-playback-example) application allows you to decode and playback a video file. The standard playback options such as play,
pause, resume, and fast forward are supported.

**Application:** [`gst-video-playback-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-video-playback-example)

The following figure shows the pipeline workflow of the video playback GStreamer application. The file is read,
demultiplexed, parsed, decoded, and displayed on a screen.For more information about the plugins, see [pipeline flow.](#video-playback-pipeline-flow)

<img alt="Video playback pipeline" />

**Figure: gst-video-playback-example pipeline**

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step>
      Input the **AVC.mp4 (H.264)** and **HEVC.mp4 (H.265)** video files on the target device.
    </Step>

    <Steps>
      In the terminal of the target device, play back the video file:

      ```bash theme={null}
      gst-video-playback-example -e filesrc location=$HOME/Downloads/qimsdk_samples/media/video_avc.mp4 ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! waylandsink enable-last-sample=false fullscreen=true
      ```

      The following table provides a description of the command-line parameters:

      | Command-line parameter | Description |
      | - | - |
      | `<input-file>` | Specifies the MP4 input file to playback. |
      | `<video-parser>` | `h264parse` for H.264, `h265parse` for H.265. |
      | `<video-decoder>` | `v4l2h264dec` for H.264, `v4l2h265dec` for H.265. |

      For example, run the following commands for different video playback use cases:

      **Playback AVC (H.264) video:**

      ```bash theme={null}
      gst-video-playback-example -e filesrc location=$HOME/Downloads/qimsdk_samples/media/video_avc.mp4 ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! waylandsink enable-last-sample=false fullscreen=true
      ```

      **Playback HEVC (H.265) video:**

      ```bash theme={null}
      gst-video-playback-example -e filesrc location=$HOME/Downloads/qimsdk_samples/media/video_hevc.mp4 ! qtdemux ! queue ! h265parse ! v4l2h265dec capture-io-mode=4 output-io-mode=4 ! waylandsink enable-last-sample=false fullscreen=true
      ```
    </Steps>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step>
      In the terminal of the target device, run the following command to display the available help options:

      ```bash theme={null}
      gst-video-playback-example --help
      ```
    </Step>
  </Steps>

  ### Expected output

  The video is captured, decoded, and played on the local display.

  <img alt="video playback output" />

  **Figure : Expected output for gst-video-playback-example application**

  ### Pipeline flow

  The following table lists the plugins used to run the video playback pipeline:

  <table>
    <thead>
      <tr>
        <th>Plugin</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>filesrc</td>

        <td>
          <ol>
            Reads the AVC or HEVC MP4 file.
          </ol>
        </td>
      </tr>

      <tr>
        <td>qtdemux</td>

        <td>
          <ol>
            Demultiplexes the AVC or HEVC MP4 file.
          </ol>
        </td>
      </tr>

      <tr>
        <td>h264parse</td>

        <td>
          <ol>
            Parses the video stream for H.264 format.
          </ol>
        </td>
      </tr>

      <tr>
        <td>h265parse</td>

        <td>
          <ol>
            Parses the video stream for H.265 format.
          </ol>
        </td>
      </tr>

      <tr>
        <td>v4l2h264dec</td>

        <td>
          <ol>
            Decodes the video stream for H.264 format.
          </ol>
        </td>
      </tr>

      <tr>
        <td>v4l2h265dec</td>

        <td>
          <ol>
            Decodes the video stream for H.265 format.
          </ol>
        </td>
      </tr>

      <tr>
        <td>waylandsink</td>

        <td>
          <ol>
            Displays the video in full-screen mode on a Wayland display.
          </ol>
        </td>
      </tr>
    </tbody>
  </table>

  **Known Issue**

  The rewind playback option isn’t supported.
</Accordion>

***

### Video Transcode

**Description:** The [**gst-video-transcode-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-video-transcode-example)  application supports video transcoding of AVC-to-HEVC or HEVC-to-AVC format. A few use cases can be video editing, security camera, and augmented reality.

The following figure shows the pipeline, which reads the incoming stream, decodes and encodes it, and either displays the transcoded output on a screen, or saves it to a file. For information about the plugins used in this pipeline, see [Pipeline flow](#video-transcode-pipeline)

<img alt="video transcode pipeline" />

**Figure: gst-video-transcode-example pipeline**

**Application:** [`gst-video-transcode-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-video-transcode-example)

<Accordion title="Try me">
  ### Run the application

  The following table lists the actions that can be performed for the common use cases of gst-videotranscode- example application:

  | Use case | Actions |
  | - | - |
  | Video editing | Rotate, flip, or resize videos. |
  | Security camera | Invert or rotate the video image to compensate for situations where the camera is improperly mounted. |
  | Augmented reality | Scale or rotate the video feed to synchronize with the virtual components. |

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step>
      Input `AVC.mp4` and `HEVC.mp4` video files on the target device.
    </Step>

    <Step title="In the terminal of the target device, transcode the video file:">
      ```bash theme={null}
      gst-video-transcode-example -i $HOME/Downloads/qimsdk_samples/media/<input-file>.mp4 -c <format> -o $HOME/Downloads/qimsdk_samples/media/<output-file>.mp4
      ```

      The following table provides a description of the command-line parameters:

      | Command-line parameter | Description |
      | - | - |
      | `-1` | Specifies the absolute path to the MP4 input file. |
      | `-c` | Specifies the video codec of the input file. |
      | `-o` | Specifies the absolute path to where the transcoded MP4 file is saved. |
    </Step>

    <Step title="Run the use cases for different video formats:">
      * Transcode from AVC to HEVC:

      ```bash theme={null}
      gst-video-transcode-example -i $HOME/Downloads/qimsdk_samples/media/video_avc.mp4 -c 1 -o $HOME/Downloads/qimsdk_samples/media/hevc_transcode_out.mp4
      ```

      * Transcode from HEVC to AVC:

      ```bash theme={null}
      gst-video-transcode-example -i $HOME/Downloads/qimsdk_samples/media/video_hevc.mp4 -c 2 -o $HOME/Downloads/qimsdk_samples/media/avc_transcode_out.mp4
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-video-transcode-example --help
      ```
    </Step>
  </Steps>

  ### Expected output

  The video stream is transcoded from H.264 to H.265 and back again to the H.264 format.

  ### Pipeline flow

  The following table lists the plugins used in the video transcoding pipeline:

  <table>
    <thead>
      <tr>
        <th>Plugin</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>filesrc</td>

        <td>
          <ol>
            Reads the video streams.
          </ol>
        </td>
      </tr>

      <tr>
        <td>qtdemux</td>

        <td>
          <ol>
            Demultiplexes the streams into separate audio and video streams.
          </ol>
        </td>
      </tr>

      <tr>
        <td>h264parse</td>

        <td>
          <ol>
            Parses the H.264 video stream.
          </ol>
        </td>
      </tr>

      <tr>
        <td>h265parse</td>

        <td>
          <ol>
            Parses the H.265 video stream.
          </ol>
        </td>
      </tr>

      <tr>
        <td>v4l2h264dec</td>

        <td>
          <ol>
            Decodes the H.264 video stream using Video4Linux.
          </ol>
        </td>
      </tr>

      <tr>
        <td>v4l2h265dec</td>

        <td>
          <ol>
            Decodes the H.265 video stream using Video4Linux.
          </ol>
        </td>
      </tr>

      <tr>
        <td>waylandsink</td>

        <td>
          <ol>
            Displays the H.264 video stream on Wayland display.
          </ol>
        </td>
      </tr>

      <tr>
        <td>filesink</td>

        <td>
          <ol>
            Writes the H.265 video to a YUV file for playback.
          </ol>
        </td>
      </tr>
    </tbody>
  </table>

  The following table provides information on how the decoded video streams are handled to generate the transcoded output:

  <table>
    <thead>
      <tr>
        <th>Plugin</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>H.264</td>

        <td>
          <ol>
            <li>v4l2h265enc encodes the decoded H.264 video into H.265 format.</li>
            <li>h265parse parses the H.265 video stream.</li>
            <li>mp4mux multiplexes the stream into an MP4 container.</li>
            <li>filesink writes the stream data to a file.</li>
            The AVC (H.264) video is transcoded to HEVC (H.265) format.
          </ol>
        </td>
      </tr>

      <tr>
        <td>H.265</td>

        <td>
          <ol>
            <li>v4l2h264enc encodes the decoded H.265 video into H.264 format.</li>
            <li>h264parse parses the H.264 video stream.</li>
            <li>mp4mux multiplexes the stream into an MP4 container.</li>
            <li>filesink writes the stream data to a file.</li>
            The HEVC (H.265) video is transcoded to AVC (H.264) format.
          </ol>
        </td>
      </tr>
    </tbody>
  </table>
</Accordion>

***

### Video Transform

**Description:** The [**gst-transform-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-transform-example) application supports rotation, flipping, and scaling operations.
You can use this application for a security camera. Adjust the orientation of the displayed image using the flip and mirror operations on the CCTV camera. For more information about the transformation properties, see qtivtransform.

The following figure shows the pipeline, which takes the captured video stream, transforms it, and either displays it or saves it to a file. For information about the plugins used in the pipeline, see [Pipeline flow.](#vtransform-pipeline)

<img alt="Single camera pipeline" />

**Figure: gst-transform-example pipeline**

**Application:** [`gst-transform-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-transform-example)

<Accordion title="Try me">
  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="In the terminal of the target device, run the following use cases to rotate, downscale, or flip the video:">
      * **Use camera source and save output to a file.**

      Perform operations such as rotate, flip, upscale/downscale simultaneously:

      ```bash theme={null}
        gst-transform-example -r 90 -f 2 --output_width 1920 --output_height 1080  -o $HOME/Downloads/qimsdk_samples/media/video_transform.mp4 -i <inputfile>.mp4
      ```

      Rotate:

      ```bash theme={null}
        gst-transform-example -r 90 --input_width 3840 --input_height 2160 -o <output_file>.mp4
      ```

      Downscale/upscale:

      ```bash theme={null}
        gst-transform-example --input_width 3840 --input_height 2160 --output_width 1920 --output_height 1080 -o <output_file>.mp4
      ```

      Flip:

      ```bash theme={null}
        gst-transform-example -f 2 --input_width 3840 --input_height 2160 -o <output_file>.mp4
      ```

      * **Use file source and save output to a file.**

      Perform operations such as rotate, flip, upscale/downscale simultaneously:

      ```bash theme={null}
      gst-transform-example -r 90 -f 2 –i $HOME/Downloads/qimsdk_samples/media/video_avc.mp4 --output_width 1920 --output_height 1080 -o <output_file>.mp4
      ```

      Rotate:

      ```bash theme={null}
      gst-transform-example -r 90 -i $HOME/Downloads/qimsdk_samples/media/video_avc.mp4 -o <output_file>.mp4
      ```

      Downscale/upscale:

      ```bash theme={null}
      gst-transform-example -i $HOME/Downloads/qimsdk_samples/media/video_avc.mp4 --output_width 1920 --output_height 1080 -o <output_file>.mp4
      ```

      Flip:

      ```bash theme={null}
      gst-transform-example -f 2 -i $HOME/Downloads/qimsdk_samples/media/video_avc.mp4 -o <output_file>.mp4
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-transform-example --help
      ```
    </Step>

    <Step title="In the terminal of the host computer, run the following command to pull the files from the target device:">
      ```bash theme={null}
      scp root@<IP address of target device>:$HOME/Downloads/qimsdk_samples/media/<filename to pull>
      ```
    </Step>
  </Steps>

  The following table provides a description of the command-line parameters:

  | Parameter | Description |
  | - | - |
  | `-h`, `--help` | Help |
  | `--output_width` | Downscale/upscale width. <br /><br />- Default value: **1920** <br />- Supports scaling: **1080 → 720** or **720 → 1080** |
  | `--output_height` | Downscale/upscale height. Default value: **1080** |
  | `-o` | Absolute path to generate the file |
  | `-r` | Image rotation. Supported values: `0`, `90`, `180`, `270` |
  | `--input_width` | Camera width. Default value: **1920** |
  | `--input_height` | Camera height. Default value: **1080** |
  | `-f` | Flip the image: <br /><br />- `0`: Don’t flip (Default) <br />- `1`: Flip horizontally <br />- `2`: Flip vertically <br />- `3`: Flip horizontally and vertically |
  | `-I`, `--input_file` | Absolute path of the input file. <br />Example: `-I $HOME/Downloads/qimsdk_samples/media/inputfileh264.mp4` |
  | `-o`, `--output_file` | Absolute path of the output file. <br />Example: `-o $HOME/Downloads/qimsdk_samples/media/video_transform.mp4` |

  ### Expected output

  This pipeline supports simultaneous live preview and recording of the transformed video stream.
  It provides two outputs:

  * A video recording that’s stored as an MP4 file in the specified path.
  * A preview on a display device.
    <img alt="vtransform Output" />
    **Figure: Expected output for gst-transform-example application–Preview**

  ### Pipeline flow

  The following table lists the plugins used to run the video transforming pipeline:

  ## Plugins Overview

  <table>
    <thead>
      <tr>
        <th>Plugin</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>qticamsrc and filesrc</td>

        <td>
          <ol>
            Captures the video data.
          </ol>
        </td>
      </tr>

      <tr>
        <td>capsfilter</td>

        <td>
          <ol>
            Enforces constraints on the raw video data.
          </ol>
        </td>
      </tr>

      <tr>
        <td>qtivtransform</td>

        <td>
          <ol>
            <li>Transforms the video data.</li>
            <li>After transformation, capsfilter is applied again.</li>
          </ol>
        </td>
      </tr>

      <tr>
        <td>tee</td>

        <td>
          <ol>
            <li>Splits the transformed video data into two output paths:</li>

            * Live preview.
            * Save to file.
          </ol>
        </td>
      </tr>

      <tr>
        <td>waylandsink</td>

        <td>
          <ol>
            Displays the video in live preview.
          </ol>
        </td>
      </tr>

      <tr>
        <td>h264parse</td>

        <td>
          <ol>
            Parses the encoded H.264 video stream.
          </ol>
        </td>
      </tr>

      <tr>
        <td>mp4mux</td>

        <td>
          <ol>
            Multiplexes the stream into an MP4 container.
          </ol>
        </td>
      </tr>

      <tr>
        <td>filesink</td>

        <td>
          <ol>
            Writes the video data to a file.
          </ol>
        </td>
      </tr>
    </tbody>
  </table>
</Accordion>

***

### Concurrent Playback & Composition(video wall)

The [**gst-concurrent-videoplay-composition**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-concurrent-videoplay-composition) application supports concurrent video playback for MP4 AVC (H.264) videos and performs composition on a video wall display.

In the concurrent video playback and composition pipeline, four decode and composition pipelines run concurrently. For more information about the plugins used in this pipeline, see [Pipeline flow.](#video-wall)

<img alt="Single camera pipeline" />

**Figure: gst-concurrent-videoplay-composition pipeline with 4 streams**

**Application:** [`gst-concurrent-videoplay-composition`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-concurrent-videoplay-composition)

<Accordion title="Try me">
  ### Run the application

  The following table lists the use cases that are supported through the gst-concurrent-videoplay-composition application:

  <table>
    <thead>
      <tr>
        <th>Use case</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>Video conferencing</td>

        <td>
          <ul>
            <li>Simultaneously decodes the video streams of several participants.</li>
            <li>Useful in a multi-party video conference scenario where the application manages several video streams.</li>
          </ul>
        </td>
      </tr>

      <tr>
        <td>Surveillance systems</td>

        <td>
          <ul>
            <li>Concurrently decodes video feeds from two separate cameras.</li>
          </ul>
        </td>
      </tr>

      <tr>
        <td>Digital signage</td>

        <td>
          <ul>
            <li>Allows simultaneous changes or updates to centrally managed information that is electronically displayed across several locations.</li>
          </ul>
        </td>
      </tr>
    </tbody>
  </table>

  To run the use cases, do the following:

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites.](#prerequisites)
    </Step>

    <Step title="Prepare input video files">
      Add at least one `MP4 AVC (H.264)` video file on the target device.
    </Step>

    <Step title="Run the use case">
      In the terminal of the target device, run:

      ```bash theme={null}
      gst-concurrent-videoplay-composition -c <number-of-streams> -i <input-file-1>.mp4 -i <input-file-2>.mp4
      ```
    </Step>
  </Steps>

  ### Command-line parameters

  | Parameter | Description |
  | - | - |
  | `-c` | Number of streams to decode for composition. Supported values: `2`, `4`, `8`, `16`. |
  | `-i` | Absolute path to an input video file. |

  ### Examples

  * **Concurrent playback of two sessions**

    ```bash theme={null}

    gst-concurrent-videoplay-composition -c 2 -i $HOME/Downloads/qimsdk_samples/media/video_avc.mp4 -i $HOME/Downloads/qimsdk_samples/media/video_avc.mp4
    ```
  * **Concurrent playback of four sessions**

    ```bash theme={null}

    gst-concurrent-videoplay-composition -c 4 -i $HOME/Downloads/qimsdk_samples/media/video_avc.mp4 -i $HOME/Downloads/qimsdk_samples/media/video_avc.mp4 -i $HOME/Downloads/qimsdk_samples/media/video_avc.mp4 -i $HOME/Downloads/qimsdk_samples/media/video_avc.mp4
    ```

    <Step title="4. Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="5. In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-concurrent-videoplay-composition --help
      ```
    </Step>

  ### Expected output

  The individual composed streams are tiled together to display as a unified stream.

  <img alt="Video wall Output" />

  **Figure: Expected output for gst-concurrent-videoplay-composition application**

  ### Pipeline flow

  The following table lists the plugins used to run the video wall pipeline:

  <table>
    <thead>
      <tr>
        <th>Plugin</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>filesrc</td>

        <td>
          <ol>
            Reads the video data.
          </ol>
        </td>
      </tr>

      <tr>
        <td>qtdemux</td>

        <td>
          <ol>
            Demultiplexes the video data.
          </ol>
        </td>
      </tr>

      <tr>
        <td>h264parse</td>

        <td>
          <ol>
            Parses the H.264 video streams.
          </ol>
        </td>
      </tr>

      <tr>
        <td>v4l2h264dec</td>

        <td>
          <ol>
            Decodes the H.264 video streams.
          </ol>
        </td>
      </tr>

      <tr>
        <td>qtivcomposer</td>

        <td>
          <ol>
            Composes the decoded streams and sends them over the display.
          </ol>
        </td>
      </tr>

      <tr>
        <td>waylandsink</td>

        <td>
          <ol>
            Displays the composed video streams on Wayland display.
          </ol>
        </td>
      </tr>
    </tbody>
  </table>

  ### Known issue

  Video playback closes or freezes. About an hour later, it continues running with 16 H.264 720p concurrent video playback sessions.
</Accordion>

***

### Weston Composition

The [**gst-weston-composition-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-weston-composition-example) application allows you to arrange live camera input or offline files on the screen using the picture-in-picture (PiP) or side by side display formats. A few use cases of this application are video conferencing and online educational courses.
This application involves several pipelines and elements. For more information, see [Pipeline flow.](#weston-composition-pipeline-flow)

<img alt="Weston composition pipeline" />

**Figure: gst-weston-composition-example pipeline**

**Application:** [`gst-weston-composition-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-weston-composition-example)

<Accordion title="Try me">
  ### Run the application

  The following table lists the use cases that supported through the gst-weston-composition-example application:

  | Use case | Description |
  | - | - |
  | **Video conferencing** | • Simultaneously displays several video streams from many participants.<br />• Videos are displayed in either PiP or side-by-side format.<br />• Useful in a multi-party video conference scenario where the application manages several video streams. |
  | **Online education** | Enhances the learning experience by enabling the PiP or side-by-side display of the instructional video. |

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step>
      Input a `<h264_file>.mp4` video file on the target device.
    </Step>

    <Step title="In the terminal of the target device, run any of the following use cases:">
      * Compose PiP:

      ```bash theme={null}
      gst-weston-composition-example -t 0 -i $HOME/Downloads/qimsdk_samples/media/video_avc.mp4
      ```

      * Compose side by side:

      ```bash theme={null}
      gst-weston-composition-example -t 1 -i $HOME/Downloads/qimsdk_samples/media/video_avc.mp4
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-weston-composition-example --help
      ```
    </Step>
  </Steps>

  The following table provides a description of the command-line parameters:

  ## Command-line Parameters

  | Command-line parameter | Description |
  | - | - |
  | `-t` | Specifies the type of composition:<br /><br />• `0` = Picture in picture<br />• `1` = Side by side |
  | `-i` | Specifies the absolute path to the input video file. |

  ### Expected output

  The video is rendered as PiP and side by side display on the screen.

  <img alt="Weston compostion Output" />

  **Figure: Expected output for gst-weston-composition-example application**

  ### Pipeline flow

  The following table lists the plugins used to run the video composition pipeline:

  | Plugin | Description |
  | - | - |
  | `qticamsrc` | Captures the video from a source. |
  | `capsfilter` | Filters the video file. |
  | `filesrc` | Reads the video data. |
  | `qtdemux` | Demultiplexes the video data. |
  | `h264parse` | Parses the H.264 video. |
  | `v4l2h264dec` | Decodes H.264 video from a file. |
  | `qtivcomposer` | Composes the decoded video. |
</Accordion>

***

### Multi format decode

**Description:** The [**gst-videocodec-concurrent-playback**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-videocodec-concurrent-playback)  application supports concurrent video playback for AVC (H.264) and HEVC (H.265) videos. It demonstrates the concurrent decoding capabilities of the Qualcomm Linux video engine, which can process different video codecs simultaneously.
As shown in the following figure, the sample application uses H.264 and H.265 videos, both encapsulated in MP4 containers. The pipeline involves several elements for concurrent video playback. For more information about these elements, see [Pipeline flow.](#multi-format-decode-pipeline-flow)

<img alt="Multi format decode pipeline" />

**Figure:  gst-videocodec-concurrent-playback pipeline**

**Application:** [`gst-videocodec-concurrent-playback`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-videocodec-concurrent-playback)

<Accordion title="Try me">
  ### Run the application

  The following table lists the use cases that are supported through the `gst-videocodec-concurrent-playback` application:

  | Use case | Description |
  | - | - |
  | **Video conferencing** | • Simultaneously decodes the video streams of several participants.<br />• Useful in a multi-party video conference scenario where the application manages several video streams. |
  | **Surveillance systems** | Concurrently decodes video feeds from two separate cameras, each employing a different encoding method. |

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step>
      **Input MP4 AVC (`H.264`) and HEVC (`H.265`) video files on the target device.**
    </Step>

    <Step title="In the terminal of the target device, run the use case:">
      ```bash theme={null}
      gst-videocodec-concurrent-playback -i $HOME/Downloads/qimsdk_samples/media/video_avc.mp4 -i $HOME/Downloads/qimsdk_samples/media/video_hevc.mp4 -o $HOME/Downloads/qimsdk_samples/media/h265_dump.yuv
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-videocodec-concurrent-playback --help
      ```
    </Step>
  </Steps>

  <Note>
    **Note:** Due to the large size of YUV data, it’s recommended to run this use case for a short duration (less than 5 seconds) to avoid exhausting the storage space.
  </Note>

  ### Command-line parameters

  | Command-line parameter | Description |
  | - | - |
  | `-i` | Specifies the absolute path to the H.264 and H.265 input video files. Ensure that the H.264 input file is specified before H.265. |
  | `-o` | Specifies the absolute path to the H.265 output YUV file. |

  ### Expected output

  * The H.264 video is displayed on the Wayland display.
  * The H.265 video is written to the YUV file.

  To play the YUV file:

  1. Pull the YUV dump file from the device.
  2. Use a YUV player running on the host to play the video.
     For example, select `YUV420planarNV12` format, and use `1920` and `1088` as the stride and scan lines.

  <Note>
    **Note:** If a default YUV player is not available on the host computer, use [Raw YUV Player](https://sourceforge.net/projects/raw-yuvplayer/).
  </Note>

  ### Pipeline flow

  The following table lists the plugins used to run the videocodec concurrent playback pipeline:

  | Plugin | Description |
  | - | - |
  | `filesrc` | Reads the video data. |
  | `qtdemux` | Demultiplexes the video data. |
  | `h264parse` | Parses the H.264 video stream. |
  | `h265parse` | Parses the H.265 video stream. |
  | `v4l2h264dec` | Decodes the H.264 video stream using Video4Linux. |
  | `v4l2h265dec` | Decodes the H.265 video stream using Video4Linux. |
  | `waylandsink` | Displays the H.264 video stream on Wayland display. |
  | `filesink` | Writes the H.265 video to a YUV file for playback. |

  ### Known issues

  * Video playback may not work as expected when the stream is generated from a file source.
</Accordion>

***

### JPEG Decode

**Description:** The [**gst-jpg-decode-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-jpg-decode-example) application allows you to decode JPEG images and view the decoded images on a screen.

The following figure shows the pipeline for gst-jpg-decode-example. For information about the plugins used in the pipeline flow, see [Pipeline flow.](#jpeg-decode-pipeline-flow)

<img alt="JPEG decode images pipeline" />

**Figure: gst-jpg-decode-example pipeline**

**Application:** [`gst-jpg-decode-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-jpg-decode-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step>
      Input at least one imagefiles\_%d.jpg on the target device.
    </Step>

    <Step title="In the terminal of the target device, run the use case. Specify the application with parameters such as width and height.">
      ```bash theme={null}
      gst-jpg-decode-example -w 1920 -h 1080 -i $HOME/Downloads/qimsdk_samples/media/imagefiles_%d.jpg
      ```
    </Step>

    <Step title="To Stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display the available help options:">
      ```bash theme={null}
      gst-jpg-decode-example --help
      ```
    </Step>
  </Steps>

  ### Expected output

  The decoded image is displayed on the screen for preview.

  ### Pipeline flow

  The following table lists the plugins used to run the decode JPEG images pipeline:

  <table>
    <thead>
      <tr>
        <th>Plugin</th>
        <th>Description</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>multifilesrc</td>

        <td>
          <ol>
            Reads the video data from sequentially named files.
          </ol>
        </td>
      </tr>

      <tr>
        <td>capsfilter</td>

        <td>
          <ol>
            Enforces constraints on the video data.
          </ol>
        </td>
      </tr>

      <tr>
        <td>jpegdec</td>

        <td>
          <ol>
            Decodes the JPEG video stream.
          </ol>
        </td>
      </tr>

      <tr>
        <td>videoconvert</td>

        <td>
          <ol>
            Converts the video frames from one format to another.
          </ol>
        </td>
      </tr>

      <tr>
        <td>waylandsink</td>

        <td>
          <ol>
            Displays the video stream on the Wayland display.
          </ol>
        </td>
      </tr>
    </tbody>
  </table>
</Accordion>

***

### Audio Playback

**Description:** The [**gst-audio-decode-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-audio-decode-example) application demonstrates the playback of audio files in WAV, MP3, and FLAC formats. Separate pipelines for each audio format ensure efficient processing and compatibility.

This sample application provides a basic template that can be extended and adapted.

The following figure shows the pipeline, which reads and parses the audio file, and then decodes and plays the audio. For information about the plugins used in the pipeline, see [Pipeline flow.](#audio-playback-pipeline-flow)

<img alt="Audio decode pipeline" />

**Figure: gst-audio-decode-example pipeline**

**Application:** [`gst-audio-decode-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-audio-decode-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step>
      **Input at least one of these audio files on the target device: `<filename>.mp3`, `<filename>.wav`, or `<filename>.flac`.**
    </Step>

    <Step title="In the terminal of the target device, run the application:">
      ```bash theme={null}
      gst-audio-decode-example -i $HOME/Downloads/qimsdk_samples/media/<filename> --audio_format=<format>
      ```
    </Step>

    <Step title="To stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display available help options:">
      ```bash theme={null}
      gst-audio-decode-example --help
      ```
    </Step>
  </Steps>

  ### Command-line parameters

  | Command-line parameter | Description |
  | - | - |
  | `-i` | Specifies the absolute path to the input audio file to decode. |
  | `--audio_format` | Specifies the format of the audio file:<br />• `1` = MP3<br />• `2` = WAV<br />• `3` = FLAC |

  For example, run the application as follows:

  * Decode MP3:
    ```bash theme={null}
    gst-audio-decode-example -i $HOME/Downloads/qimsdk_samples/media/audio.mp3 --audio_format=1
    ```

  * Decode WAV:
    ```bash theme={null}
    gst-audio-decode-example -i $HOME/Downloads/qimsdk_samples/media/audio.wav --audio_format=2
    ```

  * Decode FLAC:
    ```bash theme={null}
    gst-audio-decode-example -i $HOME/Downloads/qimsdk_samples/media/audio.flac --audio_format=3
    ```

  ### Expected output

  Based on the selected format (MP3, WAV, or FLAC), the decoded audio file is played back.

  ### Pipeline flow

  The following table lists the plugins used to run the audio decode pipeline:

  | Plugin | Description |
  | - | - |
  | filesrc | Reads the audio files. |
  | mpegaudioparse | Parses MPEG audio files. |
  | wavparse | Parses WAV audio files. |
  | flacparse | Parses FLAC audio files. |
  | mpg123audiodec | Decodes MP3 files. |
  | flacdec | Decodes FLAC files. |
  | pulsesink | Plays the decoded audio. |

  ### Known issues

  * Audio playback may not work if the audio sink isn't selected using wpctl.
</Accordion>

***

### Audio Encode

**Description:** The [**gst-audio-encode-example**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-audio-encode-example) application encodes audio in FLAC or WAV format.
A common use case for this application can be to encode live audio data for streaming or storage.
The following figure shows the pipeline, which captures the audio, converts the stream into FLAC or WAV formats, encodes the stream, and stores it in a file. For more information about the plugins used in the pipeline, see [Pipeline flow.](#audio-recording-pipeline-flow)

<img alt="Single camera pipeline" />

**Figure: gst-audio-encode-example pipeline**

**Application:** [`gst-audio-encode-example`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-audio-encode-example)

<Accordion title="Try me">
  ### Run the application

  For this use case, if you have a live audio source (such as a microphone or an internet radio stream), you can encode this
  audio data into a more compact or standardized format for efficient transmission or storage.

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="In the terminal of the target device, run the application for different audio formats:">
      ```bash theme={null}
      gst-audio-encode-example -o <path-file> --audio_format=<format>
      ```
    </Step>

    <Step title="To stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display available help options:">
      ```bash theme={null}
      gst-audio-encode-example --help
      ```
    </Step>

    <Step title="In the terminal of the host computer,run the following command to pull files from the target device:">
      ```bash theme={null}
      scp root@<IP address of target device>:$HOME/Downloads/qimsdk_samples/media/<destination directory>
      ```
    </Step>
  </Steps>

  ### Command-line parameters

  | Command-line parameter | Description |
  | - | - |
  | `-o` | Specifies the absolute path to the output audio file. |
  | `--audio_format` | Specifies the format of the audio file:<br />• `1` = FLAC<br />• `2` = WAV |

  For example, use the following commands to run the application for FLAC and WAV audio formats::

  * Encode FLAC audio:
    ```bash theme={null}
    gst-audio-encode-example -o $HOME/Downloads/qimsdk_samples/media/audio_record.flac.flac --audio_format=1
    ```
  * Encode WAV audio:
    ```bash theme={null}
    gst-audio-encode-example -o $HOME/Downloads/qimsdk_samples/media/audio_record.flac.wav --audio_format=2
    ```

  ### Expected output

  The captured audio is encoded and written to a file based on the selected format (FLAC or WAV).

  ### Pipeline flow

  In audio encoding, the pipeline differs slightly based on the encoding format.
  The following table lists the plugins used to run the audio recording pipeline:

  | Plugin | Description |
  | - | - |
  | pulsesrc | Captures audio from the system sound device. |
  | capsfilter | Ensures that the audio is in a format suitable for the encoder. |
  | audioconvert | Converts the audio to the required format. |
  | flacenc | Encodes audio into FLAC format. |
  | wavenc | Encodes audio into WAV format. |
  | filesink | Writes the encoded audio to a file. |

  ### Known issues

  * Audio encode may not work if the audio sink is not selected using wpctl.
</Accordion>

***

### Audio-Video Encode

**Description:** The [**gst-audio-video-encode**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-audio-video-encode) application encodes audio and video data, which is stored in a user-specified output file. A few use cases for this application are streaming services, video conferencing, and surveillance systems.

The following figure shows the pipeline, which receives the video and audio feed, encodes, and parses the video stream. It then captures the audio stream and encodes it in MP3 format. Both streams are multiplexed and written to a file.

For information about the plugins used in the pipeline, see [Pipeline flow.](#audio-video-encode-pipeline-flow)

<img alt="Audio video pipeline" />

**Figure: gst-audio-video-encode pipeline**

**Application:** [`gst-audio-video-encode`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-audio-video-encode)

<Accordion title="Try me">
  ### Run the application

  The following table lists the actions that can be performed for common use cases of the `gst-audio-video-encode`
  application:

  | Use case | Actions |
  | - | - |
  | **Streaming services** | Encode audio and video streams for live internet streaming. |
  | **Security camera** | Invert or rotate the video image to compensate for situations where the camera is improperly mounted. |
  | **Augmented reality** | Scale or rotate the video feed to synchronize with virtual components. |

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="In the terminal of the target device, run the use case:">
      ```bash theme={null}
      gst-audio-video-encode -w <width> -h <height> -c <inputcodec> -o $HOME/Downloads/qimsdk_samples/media/<output-file>.mp4
      ```
    </Step>

    <Step title="To stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display available help options:">
      ```bash theme={null}
      gst-audio-video-encode --help
      ```
    </Step>

    <Step title="In the terminal of the host computer,run the following command to pull files from the target device:">
      ```bash theme={null}
      scp root@<IP address of target device>:$HOME/Downloads/qimsdk_samples/media/<destination directory>
      ```
    </Step>
  </Steps>

  ### Command-line parameters

  | Command-line parameter | Description |
  | - | - |
  | `-w` | Specifies the camera width. |
  | `-h` | Specifies the camera height. |
  | `-c` | Specifies the video codec:<br />• `1` = AVC (H.264)<br />• `2` = HEVC (H.265) |
  | `-o` | Provides the absolute path to the output file. |

  For example, run different video encode use cases as follows:

  * AVC:
    ```bash theme={null}
    gst-audio-video-encode -w 1920 -h 1080 -c 1 -o $HOME/Downloads/qimsdk_samples/media/audiovideo.mp4
    ```
  * HEVC:
    ```bash theme={null}
    gst-audio-video-encode -w 1920 -h 1080 -c 2 -o $HOME/Downloads/qimsdk_samples/media/audiovideo.mp4
    ```

  ### Expected output

  The encoded audio and video streams are multiplexed into an MP4 container and written to a file.

  ### Pipeline flow

  The following table lists the plugins used to run the audio and video stream encoding pipeline:

  | Plugin | Description |
  | - | - |
  | `qticamsrc` and `capsfilter` | Captures the media stream. |
  | `v4l2h264enc` | Encodes video for AVC (H.264) format. |
  | `v4l2h265enc` | Encodes video for HEVC (H.265) format. |
  | `h264parse` | Parses the H.264 video stream. |
  | `h265parse` | Parses the H.265 video stream. |
  | `pulsesrc` | Captures the audio stream, filtered using capsfilter. |
  | `audioconvert` | Converts the audio for encoding. |
  | `lamemp3enc` | Encodes the audio in MP3 format. |
  | `mp4mux` | Multiplexes encoded video and audio into an MP4 container. |
  | `filesink` | Writes the stream into a file. |

  ### Known issues

  * Audio and video encode may not work if the sink is not selected using wpctl.
</Accordion>

***

### Audio-Video Playback

The [**gst-audio-video-playback**](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-video-playback-example) application supports simultaneous audio and video decoding and playback of FLAC/MP3 audio and AVC/HEVC video.

The following are the applicable scenarios to use the gst-audio-video-playback application:

* Media player applications
* Streaming services
* Video conferencing
* Surveillance systems
* Educational platforms
* Entertainment platforms that can play back audio and video files in various formats

As shown in the following figure, the pipeline involves several elements for audio and video playback. For more information about these elements, see [Pipeline flow.](#audio-video-playback-pipeline-flow)

<img alt="Audio video playback pipeline" />

**Figure: gst-audio-video-playback pipeline**

**Application:** [`gst-audio-video-playback`](https://github.com/qualcomm/qimsdk/tree/main/gstreamer/gst-sample-apps/gst-video-playback-example)

<Accordion title="Try me">
  ### Run the application

  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step>
      Input these video files on the target device: `<avc_flac_file>.mp4`, `<hevc_flac_file>.mp4`, `<avc_mp3_file>.mp4`, and `<hevc_mp3_file>.mp4`.
    </Step>

    <Step title="In the terminal of the target device, run the use case:">
      ```bash theme={null}
      gst-audio-video-playback -v 1 -a 1 -i $HOME/Downloads/qimsdk_samples/media/<filename>.mp4
      ```
    </Step>

    <Step title="To stop the running use case">
      Press `Ctrl + C`.
    </Step>

    <Step title="In the terminal of the target device, run the following command to display available help options:">
      ```bash theme={null}
      gst-audio-video-playback --help
      ```
    </Step>
  </Steps>

  ### Command-line parameters

  | Command-line parameter | Description |
  | - | - |
  | `-v` | Specifies the video codec:<br /> `1` = AVC (H.264)<br /> `2` = HEVC (H.265) |
  | `-a` | Specifies the audio codec:<br /> `1` = FLAC<br /> `2` = MP3 |
  | `-i` | Specifies the absolute path to the MP4 input file. |

  For example, run the following for different audio/video formats:

  * AVC video with FLAC audio:
    ```bash theme={null}
    gst-audio-video-playback -v 1 -a 1 -i $HOME/Downloads/qimsdk_samples/media/avc_flac.mp4
    ```
  * AVC video with MP3 audio:
    ```bash theme={null}
    gst-audio-video-playback -v 1 -a 2 -i $HOME/Downloads/qimsdk_samples/media/avc_mp3.mp4
    ```
  * HEVC video with FLAC audio:
    ```bash theme={null}
    gst-audio-video-playback -v 2 -a 1 -i $HOME/Downloads/qimsdk_samples/media/hevc_flac.mp4
    ```
  * HEVC video with MP3 audio:
    ```bash theme={null}
    gst-audio-video-playback -v 2 -a 2 -i $HOME/Downloads/qimsdk_samples/media/hevc_mp3.mp4
    ```

  ### Expected output

  * The AVC/HEVC video is displayed on the screen.
  * The audio plays through the device speaker.
    <img alt="audio video output" />
    **Figure: Expected output for gst-audio-video-playback application**

  ### Pipeline flow

  The following table lists the plugins used to run the audio-video playback pipeline:

  | Plugin | Description |
  | - | - |
  | `filesrc` | Reads the video data. |
  | `qtdemux` | Demultiplexes the video data into separate audio and video streams. |
  | `h264parse` | Parses the H.264 video stream. |
  | `h265parse` | Parses the H.265 video stream. |
  | `v4l2h264dec` | Decodes the H.264 video stream. |
  | `v4l2h265dec` | Decodes the H.265 video stream. |
  | `waylandsink` | Displays the video stream on Wayland display. |
  | `flacparse` or `mpegaudioparse` | Parses the audio stream. |
  | `flacdec` or `mpg123audiodec` | Decodes the audio stream. |
  | `pulsesink` | Allows playback of the audio stream. |

  ### Known issues

  * Audio and video playback may not work if the sink is not selected using wpctl.
</Accordion>


# AI
Source: https://imsdkdocs.qualcomm.com/sample-application/python-ai-sample-applications

Ready-to-run Python AI applications demonstrating QIM SDK inference capabilities using GObject Introspection (GI) bindings

This section covers Python-based AI sample applications that demonstrate vision and multi-model inference on Qualcomm platforms.

The following tables list all available Python AI applications and their platform support. Select the appropriate configuration tab for your setup.

<Tabs>
  <Tab title="Config #1">
    <Note>
      None of the Python AI applications are supported in `Config #1` of the `QLI 2.1 GA` release. Select the Config #2 tab.
    </Note>
  </Tab>

  <Tab title="Config #2">
    | Application | Source code | Description | Input sources | Runtime | QCS6490 / IQ-8275 / IQ-9075 | IQ-X7181/IQ-X5121 | IQ-615 / IQ-2390 |
    | - | - | - | - | - | :-: | :-: | :-: |
    | Object detection and display | [`gst-ai-object-detection.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-ai-object-detection.py) | Object detection with bounding box overlay on the display. | Camera, file | dsp | ✓ | ✓ | × |
    | Decode and object detection using RTSP stream | [`gst-rtspsrc-detection-display.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-rtspsrc-detection-display.py) | Object detection on a decoded RTSP stream. | RTSP | dsp | ✓ | ✓ | × |
    | Object detection and classification | [`gst-camera-two-stream-detection-and-classification-side-by-side.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-two-stream-detection-and-classification-side-by-side.py) | Detection and classification displayed side by side. | Camera | dsp | ✓ | ✓ | × |
    | Object detection, classification, and segmentation | [`gst-filesrc-2detection-classification-segmentation-side-by-side.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-filesrc-2detection-classification-segmentation-side-by-side.py) | Detection, classification, and segmentation displayed side by side. | File | dsp | ✓ | ✓ | × |
    | Camera encode, object detection, and display | [`gst-camera-two-stream-encode-file-detection-display.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-two-stream-encode-file-detection-display.py) | Encodes the camera stream to a file while running object detection. | Camera | dsp | ✓ | ✓ | × |
    | Parallel inference using Python | [`gst-parallel-inference.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-parallel-inference.py) | Four-channel parallel inference: classification, detection, pose, and segmentation. | Camera, file, RTSP | dsp | ✓ | ✓ | × |
    | Daisy chain detection and pose detection using Python | [`gst-daisychain-detection-pose.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-daisychain-detection-pose.py) | Cascaded object detection and pose detection. | Camera, file, RTSP | dsp | ✓ | ✓ | × |
  </Tab>
</Tabs>

***

## Prerequisites

<Accordion title="Try me">
  <Note>
    Some of the steps in the pre-requisties will be removed from future releases once the necessary fixes are mainlined.
  </Note>

  <Steps>
    <Step title="Set up Wi-Fi">
      Connect to the Wireless Access Point (Wi-Fi Router):

      ```bash theme={null}
      nmcli dev wifi connect <WiFi-SSID> password <WiFi-password>
      ```

      Check the connection and device status:

      ```bash theme={null}
      nmcli -p device
      ```

      **Login to the target device**

      Locate the IP address of the device according to the type of network connection,
      using the device shell on the Linux host:

      ```bash theme={null}
      ip address show
      ```

      Use the IP address from the Linux host to establish an SSH connection to the device:

      ```bash theme={null}
      ssh root@<ip-address>
      ```

      Example:

      ```bash theme={null}
      ssh root@192.168.0.222
      ```

      Connect to the SSH shell using the following password:

      ```bash theme={null}
      oelinux123
      ```
    </Step>

    <Step title="Download Models and Artifacts">
      On the target device, obtain the `download_artifacts.sh` script, set executable
      permissions, and run it to download the model, media, and label files:

      ```bash theme={null}
      cd /tmp/
      curl -L -O https://raw.githubusercontent.com/qualcomm/sample-apps-for-qualcomm-linux/refs/heads/main/qualcomm-linux/scripts/download_artifacts_2.x.sh    
      chmod +x download_artifacts_2.x.sh
      ./download_artifacts_2.x.sh
      ```

      <Note>
        On `Qualcomm Ubuntu`, run the above command with `sudo`.
      </Note>
    </Step>

    <Step title="Enable qticamsrc">
      In the terminal of the target device, run the following command to enable the
      `qticamsrc` on Config #2:

      ```bash theme={null}
      echo -n "camx" > /var/data
      efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -w -f /var/data
      efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -p
      sync
      reboot
      ```
    </Step>

    <Step title="Enable Audio and GPU Delegate">
      In the terminal of the target device, run the following command to enable audio:

      ```bash theme={null}
      systemctl stop pipewire wireplumber pipewire.socket pipewire-manager.socket
      chmod 777 /dev/dma_heap/system
      adsprpcd audiopd &
      systemctl start pipewire wireplumber
      wpctl status
      ```

      To set the default devices for sink and source, get the device numbers from
      `wpctl status` and run the following command:

      ```bash theme={null}
      wpctl set-default <device ID>
      ```

      In the terminal of the target device, run the following command to enable the
      GPU delegate and backend:

      ```bash theme={null}
      mount -o rw,remount /usr
      ```

      ```bash theme={null}
      export OCL_ICD_FILENAMES=/usr/lib/libOpenCL_adreno.so.1
      ```
    </Step>
  </Steps>
</Accordion>

## AI Vision Applications

### Object Detection and Display

The [**gst-ai-object-detection.py**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-ai-object-detection.py) script receives an RTSP stream as a source, decodes it, uses YOLOv8 LiteRT model to identify the object in a scene from the camera stream and overlay the bounding boxes over the detected objects. The results are shown on the display.

For information about the plugins used in this pipeline, see [**Pipeline flow**](#object-detection-pipeline-flow).

<Note>
  This application isn't supported in `Config #1` of `Qualcomm Linux` because CPU runtime is not supported.
</Note>

**Application:** [`gst-ai-object-detection.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-ai-object-detection.py)

<img alt="Pipeline Diagram" />

<Accordion title="Try me">
  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | `yolov8_det_w8a8.tflite`, `yolox_quantized.tflite` | `yolov8.json`, `yolox.json` |
  | Qualcomm Neural Processing SDK | `yolonas.dlc` | `yolonas.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Ensure that you complete the [`Prerequisites`](#prerequisites). This downloads all required artifacts to the target device.
    </Step>

    <Step title="Run the application">
      Run any of the following use cases:

      * Display with the primary and secondary cameras respectively:

      ```bash theme={null}
      gst-ai-object-detection.py -c 0 -f 2 -m ${HOME}/Downloads/qimsdk_samples/models/yolox_quantized.tflite -l ${HOME}/Downloads/qimsdk_samples/labels/yolox.json -ml "yolov8"
      ```

      ```bash theme={null}
      gst-ai-object-detection.py -c 1 -f 2 -m ${HOME}/Downloads/qimsdk_samples/models/yolox_quantized.tflite -l ${HOME}/Downloads/qimsdk_samples/labels/yolox.json -ml "yolov8"
      ```

      * Display with input from a video file:

      ```bash theme={null}
      gst-ai-object-detection.py -s ${HOME}/Downloads/qimsdk_samples/media/video.mp4 -f 2 -m ${HOME}/Downloads/qimsdk_samples/models/yolox_quantized.tflite -l ${HOME}/Downloads/qimsdk_samples/labels/yolox.json -ml "yolov8"
      ```

      * YOLOv8 with LiteRT:

      ```bash theme={null}
      gst-ai-object-detection.py -f 2 -m ${HOME}/Downloads/qimsdk_samples/models/yolov8_det_w8a8.tflite -l ${HOME}/Downloads/qimsdk_samples/labels/yolov8.json -ml "yolov8"
      ```

      <Note>
        Both `yolox_quantized.tflite` and `yolov8_det_w8a8.tflite` model files use the `Yolov8` module.
      </Note>

      To display all available options:

      ```bash theme={null}
      gst-ai-object-detection.py -h
      ```

      To stop the use case, press **CTRL + C**.
    </Step>
  </Steps>

  #### Expected Output

  Detected objects with bounding boxes and labels are overlaid on the video and displayed on the local display.

  <img alt="Output Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the object detection pipeline:

  | Process | Description |
  | - | - |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | 1. Collects the video stream (source) and creates two copies of the source: <br /><br /> • One stream is sent to the `qtimetamux` plugin to retain the video stream. <br /><br /> • The other stream is sent to an ML inferencing pipeline. |
  | `filesrc` | Reads the video data. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | 1. Receives the video stream on its sinkpad. <br /><br /> 2. Performs preprocessing: color conversion, scaling, normalization. <br /><br /> 3. Converts the video stream to a tensor stream on its source pad. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | Loads the object detection model, runs inference, and produces a tensor stream with detection results. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | Converts inference tensors into bounding boxes and labels. Loads the YOLOv8 submodule and sends results to `qtimetamux`. |
  | [`qtimetamux`](../plugin-reference/qtimetamux) | Synchronizes video and bounding box metadata streams. Attaches bounding boxes as `GstVideoRegionOfInterest` metadata. |
  | [`qtivoverlay`](../plugin-reference/qtivoverlay) | Overlays bounding boxes on the video frame using CL. |
  | [`waylandsink`](../plugin-reference/waylandsink) | Renders the video stream on a local display via Weston. |
  | `filesink` | Writes the video to a file. |
</Accordion>

***

### Decode and Object Detection Using RTSP Stream

The [**gst-rtspsrc-detection-display.py**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-rtspsrc-detection-display.py) script receives an RTSP stream as a source, decodes it, uses YOLOv8 LiteRT model to identify the object in a scene from the camera stream and overlay the bounding boxes over the detected objects. The results are shown on the display.

For information about the plugins used in this pipeline, see [**Pipeline flow**](#rtsp-detection-pipeline-flow).

<Note>
  This application isn't supported in `Config #1` of `Qualcomm Linux` because CPU runtime is not supported.
</Note>

**Application:** [`gst-rtspsrc-detection-display.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-rtspsrc-detection-display.py)

<img alt="Pipeline Diagram" />

<Accordion title="Try me">
  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | `yolox_quantized.tflite` | `yolox.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Ensure that you complete the [`Prerequisites`](#prerequisites). This downloads all required artifacts to the target device.
    </Step>

    <Step title="Run the application">
      Run the object detection script on the target device:

      Console 1 — start the RTSP server:

      ```bash theme={null}
      gst-launch-1.0 -e qticamsrc camera=0 ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! h264parse config-interval=1 ! qtirtspbin
      ```

      Console 2 — run the detection script:

      ```bash theme={null}
      gst-rtspsrc-detection-display.py
      ```

      To display all available options:

      ```bash theme={null}
      gst-rtspsrc-detection-display.py -h
      ```

      <Note>
        The default paths for model and label files are provided in the help section.
      </Note>

      To stop the use case, press **CTRL + C**.
    </Step>
  </Steps>

  #### Expected Output

  <img alt="Output Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the rtsp detection pipeline:

  | Process | Description |
  | - | - |
  | `rtspsrc` | Receives the RTSP stream from `rtsp://<ip>:<port>/live`. |
  | `rtph264depay` | Extracts the video data from the RTSP stream. |
  | `h264parse` | Parses the H.264 video. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | Decodes the video. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | Preprocesses frames (color conversion, scaling, normalization) and converts to tensor stream. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | Runs object detection inference. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | Converts inference tensors into bounding boxes and labels. |
  | [`qtimetamux`](../plugin-reference/qtimetamux) | Synchronizes video and metadata streams. |
  | [`qtivoverlay`](../plugin-reference/qtivoverlay) | Overlays bounding boxes on the video frame. |
  | [`waylandsink`](../plugin-reference/waylandsink) | Renders the video stream on a local display. |
</Accordion>

***

### Object Detection and Classification

The [**gst-camera-two-stream-detection-and-classification-side-by-side.py**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-two-stream-detection-and-classification-side-by-side.py) application uses a YOLOX LiteRT model to detect and `inception_v3` to classify objects in the scene displayed by the AI overlay composer.

For information about the plugins used in this pipeline, see [**Pipeline flow**](#detection-and-classification-pipeline-flow).

<Note>
  This application isn't supported in `Config #1` of `Qualcomm Linux` because CPU runtime is not supported.
</Note>

**Application:** [`gst-camera-two-stream-detection-and-classification-side-by-side.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-two-stream-detection-and-classification-side-by-side.py)

<img alt="Pipeline Diagram" />

<Accordion title="Try me">
  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | detection: `yolox_quantized.tflite`<br />classification: `inception_v3_quantized.tflite` | detection: `yolox.json`<br />classification: `classification.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Ensure that you complete the [`Prerequisites`](#prerequisites). This downloads all required artifacts to the target device.
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      gst-camera-two-stream-detection-and-classification-side-by-side.py
      ```

      To display all available options:

      ```bash theme={null}
      gst-camera-two-stream-detection-and-classification-side-by-side.py -h
      ```

      <Note>
        The default paths for model and label files are provided in the help section.
      </Note>

      To stop the use case, press **CTRL + C**.
    </Step>
  </Steps>

  #### Expected Output

  Detection and classification display are shown side-by-side.

  <img alt="Pipeline Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the detection and classification pipeline:

  | Process | Description |
  | - | - |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | Collects two video streams — one for detection, one for classification — each split via `tee`. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | Preprocesses frames for each branch (color conversion, scaling, normalization). |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | Runs detection (YOLOX) and classification (InceptionV3) inference sequentially. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (detection) | Converts detection tensors to bounding boxes and labels. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (classification) | Converts classification tensors to label overlay frames. |
  | [`qtimetamux`](../plugin-reference/qtimetamux) | Synchronizes video and metadata for the detection branch. |
  | [`qtivoverlay`](../plugin-reference/qtivoverlay) | Overlays bounding boxes on the detection stream. |
  | [`qtivcomposer`](../plugin-reference/qtivcomposer) | Composes the detection and classification streams side by side. |
  | [`waylandsink`](../plugin-reference/waylandsink) | Renders the composed output on a local display. |

  #### Known Issues

  <AccordionGroup>
    <Accordion title="Config #2">
      Output labels are blurred.
    </Accordion>
  </AccordionGroup>
</Accordion>

***

### Object Detection, Classification, and Segmentation

The [**gst-filesrc-2detection-classification-segmentation-side-by-side.py**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-filesrc-2detection-classification-segmentation-side-by-side.py) script identifies objects, classifies scenes, and produces semantic segmentation from a video file. The output is displayed side by side.

For information about the plugins used in this pipeline, see [**Pipeline flow**](#detection-classification-segmentation-pipeline-flow).

<Note>
  This application isn't supported in `Config #1` of `Qualcomm Linux` because CPU runtime is not supported.
</Note>

**Application:** [`gst-filesrc-2detection-classification-segmentation-side-by-side.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-filesrc-2detection-classification-segmentation-side-by-side.py)

<img alt="Pipeline Diagram" />

<Accordion title="Try me">
  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | detection: `yolox_quantized.tflite`<br />classification: `inception_v3_quantized.tflite`<br />segmentation: `deeplabv3_plus_mobilenet_quantized.tflite` | detection: `yolox.json`<br />classification: `classification.json`<br />segmentation: `deeplabv3_resnet50.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Ensure that you complete the [`Prerequisites`](#prerequisites). This downloads all required artifacts to the target device.
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      gst-filesrc-2detection-classification-segmentation-side-by-side.py
      ```

      <Note>
        The default input video for classification, segmentation, and detection is at \$/Downloads/qimsdk\_samples/media/video.mp4\`.
      </Note>

      To display all available options:

      ```bash theme={null}
      gst-filesrc-2detection-classification-segmentation-side-by-side.py -h
      ```

      <Note>
        The default paths for model and label files are provided in the help section.
      </Note>

      To stop the use case, press **CTRL + C**.
    </Step>
  </Steps>

  #### Expected Output

  The four streams are previewed side by side on a local display.

  #### Pipeline Flow

  The following table lists the plugins used in the detection, classification and segmentation pipeline:

  | Process | Description |
  | - | - |
  | `filesrc` | Reads the video data from a file. |
  | `qtdemux` | Demultiplexes the video data. |
  | `h264parse` | Parses H.264 video. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | Decodes the H.264 video. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | Preprocesses frames for each inference branch. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | Runs detection, classification, and segmentation inference. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (detection) | Produces bounding boxes and labels. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (classification) | Produces classification label frames. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (segmentation) | Produces segmentation mask frames via `deeplab-argmax`. |
  | [`qtimetamux`](../plugin-reference/qtimetamux) | Synchronizes video and metadata for the detection branch. |
  | [`qtivoverlay`](../plugin-reference/qtivoverlay) | Overlays bounding boxes on the detection stream. |
  | [`qtivcomposer`](../plugin-reference/qtivcomposer) | Composes all four streams side by side. |
  | [`waylandsink`](../plugin-reference/waylandsink) | Renders the composed output on a local display. |
</Accordion>

***

### Camera Encode, Object Detection, and Display

The [**gst-camera-two-stream-encode-file-detection-display.py**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-two-stream-encode-file-detection-display.py) application encodes the camera stream to a file while simultaneously running YOLOX object detection and displaying results.

For information about the plugins used in this pipeline, see [**Pipeline flow**](#camera-encode-detection-pipeline-flow).

<Note>
  This application isn't supported in `Config #1` of `Qualcomm Linux` because CPU runtime is not supported.
</Note>

**Application:** [`gst-camera-two-stream-encode-file-detection-display.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-two-stream-encode-file-detection-display.py)

<img alt="Pipeline Diagram" />

<Accordion title="Try me">
  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | `yolox_quantized.tflite` | `yolox.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Ensure that you complete the [`Prerequisites`](#prerequisites). This downloads all required artifacts to the target device.
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      gst-camera-two-stream-encode-file-detection-display.py
      ```

      To display all available options:

      ```bash theme={null}
      gst-camera-two-stream-encode-file-detection-display.py -h
      ```

      <Note>
        The default paths for model and label files are provided in the help section.
      </Note>

      To stop the use case, press **CTRL + C**.
    </Step>
  </Steps>

  #### Expected Output

  The camera encode output is saved at `${HOME}/Downloads/qimsdk_samples/media/test.mp4`.

  The object detection output is rendered on the display.

  <img alt="Pipeline Diagram" />

  #### Pipeline Flow

  The following table lists the plugins used in the camera encode detection pipeline:

  | Process | Description |
  | - | - |
  | [`qticamsrc`](../plugin-reference/qticamsrc) | Collects two streams — one for detection, one for encoding. |
  | [`v4l2h264enc`](../plugin-reference/v4l2h264enc) | Encodes H.264 video. |
  | `h264parse` | Parses H.264 video. |
  | `mp4mux` | Multiplexes video into MP4. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | Preprocesses frames for detection inference. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | Runs YOLOX object detection inference. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) | Produces bounding boxes and labels. |
  | [`qtimetamux`](../plugin-reference/qtimetamux) | Synchronizes video and metadata. |
  | [`qtivoverlay`](../plugin-reference/qtivoverlay) | Overlays bounding boxes on the video frame. |
  | `filesink` | Saves the encoded video to a file. |
  | [`waylandsink`](../plugin-reference/waylandsink) | Renders the detection output on a local display. |
</Accordion>

***

### Parallel Inference Using Python

The [**gst-parallel-inference.py**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-parallel-inference.py) application receives a video input from a camera, file, or RTSP stream and runs four-channel parallel AI inference (classification, object detection, pose detection, and segmentation).

For information about the plugins used in this pipeline, see [**Pipeline flow**](#parallel-inference-pipeline-flow).

<Note>
  This application isn't supported in `Config #1` of `Qualcomm Linux` because CPU runtime is not supported.
</Note>

**Application:** [`gst-parallel-inference.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-parallel-inference.py)

<img alt="Pipeline Diagram" />

<Accordion title="Try me">
  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | detection: `yolox_quantized.tflite`<br />classification: `inception_v3_quantized.tflite`<br />segmentation: `deeplabv3_plus_mobilenet_quantized.tflite`<br />pose: `hrnet_pose_quantized.tflite` | detection: `yolox.json`<br />classification: `classification.json`<br />segmentation: `deeplabv3_resnet50.json`<br />pose: `hrnet_pose.json`, `hrnet_settings.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Ensure that you complete the [`Prerequisites`](#prerequisites). This downloads all required artifacts to the target device.
    </Step>

    <Step title="Run the application">
      * Input from the camera:

      ```bash theme={null}
      gst-parallel-inference.py --camera
      ```

      * Input from a file:

      ```bash theme={null}
      gst-parallel-inference.py --file "${HOME}/Downloads/qimsdk_samples/media/video.mp4"
      ```

      * Input from an RTSP stream:

      ```bash theme={null}
      gst-parallel-inference.py --rtsp "rtsp://<ip>:<port>/<stream>"
      ```

      To display all available options:

      ```bash theme={null}
      gst-parallel-inference.py -h
      ```

      <Note>
        The default paths for model and label files are provided in the help section.
      </Note>

      To stop the use case, press **CTRL + C**.
    </Step>
  </Steps>

  #### Expected Output

  The four inference results are displayed side by side on the screen.

  <img alt="Expected Output" />

  #### Pipeline Flow

  The following table lists the plugins used in the parallel inference pipeline:

  | Process | Description |
  | - | - |
  | [`qticamsrc`](../plugin-reference/qticamsrc) / `filesrc` / `rtspsrc` | Captures stream from camera, file, or RTSP. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | Decodes H.264 video. |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | Preprocesses frames for each of the four inference branches. |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | Runs detection, classification, segmentation, and pose inference in parallel. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (detection) | Produces bounding boxes and labels. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (classification) | Produces classification label frames. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (segmentation) | Produces segmentation mask frames. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (pose) | Produces pose key point frames via HRNet module. |
  | [`qtimetamux`](../plugin-reference/qtimetamux) | Synchronizes video and metadata for the detection branch. |
  | [`qtivoverlay`](../plugin-reference/qtivoverlay) | Overlays bounding boxes on the detection stream. |
  | [`qtivcomposer`](../plugin-reference/qtivcomposer) | Composes all four output streams side by side. |
  | [`waylandsink`](../plugin-reference/waylandsink) | Renders the composed output on a local display. |
</Accordion>

***

### Daisy Chain Detection and Pose Detection Using Python

The [**gst-ai-daisychain-detection-pose.py**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-daisychain-detection-pose.py) application performs cascaded object detection and pose detection with input from a camera, file, or RTSP stream.

For information about the plugins used in this pipeline, see [**Pipeline flow**](#daisychain-pose-pipeline-flow).

<Note>
  This application isn't supported in `Config #1` of `Qualcomm Linux` because CPU runtime is not supported.
</Note>

**Application:** [`gst-ai-daisychain-detection-pose.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-daisychain-detection-pose.py)

<img alt="Pipeline Diagram" />

<Accordion title="Try me">
  #### Sample Model and Label Files

  | Runtime | Model file | Label file |
  | - | - | - |
  | LiteRT | detection: `yolox_quantized.tflite`<br />pose: `hrnet_pose_quantized.tflite` | detection: `yolox.json`<br />pose: `hrnet_pose.json`, `hrnet_settings.json` |

  #### Run the application on the target device

  <Steps>
    <Step title="Download artifacts">
      Ensure that you complete the [`Prerequisites`](#prerequisites). This downloads all required artifacts to the target device.
    </Step>

    <Step title="Run the application">
      * Input from the camera:

      ```bash theme={null}
      gst-daisychain-detection-pose.py --camera
      ```

      * Input from a file:

      ```bash theme={null}
      gst-daisychain-detection-pose.py --file ${HOME}/Downloads/qimsdk_samples/media/video.mp4
      ```

      * Input from an RTSP stream:

      ```bash theme={null}
      gst-daisychain-detection-pose.py --rtsp "rtsp://<ip>:<port>/<stream>"
      ```

      To display all available options:

      ```bash theme={null}
      gst-daisychain-detection-pose.py -h
      ```

      <Note>
        The default paths for model and label files are provided in the help section.
      </Note>

      To stop the use case, press **CTRL + C**.
    </Step>
  </Steps>

  #### Expected Output

  <img alt="Expected Output" />

  #### Pipeline Flow

  The following table lists the plugins used in the daisychain detection and pose pipeline:

  | Process | Description |
  | - | - |
  | [`qticamsrc`](../plugin-reference/qticamsrc) / `filesrc` / `rtspsrc` | Captures stream from camera, file, or RTSP. |
  | [`v4l2h264dec`](../plugin-reference/v4l2h264dec) | Decodes H.264 video. |
  | [`qtivsplit`](../plugin-reference/qtivsplit) | Crops full frame into smaller frames based on detected bounding boxes (maximum 4). |
  | [`qtimlvconverter`](../plugin-reference/qtimlvconverter) | Preprocesses frames for detection (stage 1) and pose estimation (stage 2). |
  | [`qtimltflite`](../plugin-reference/qtimltflite) | Runs detection and pose inference sequentially. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (detection) | Produces bounding boxes. Loads YOLOv8 submodule. |
  | [`qtimlpostprocess`](../plugin-reference/qtimlpostprocess) (pose) | Produces pose key point frames. Loads HRNet module. |
  | [`qtimetamux`](../plugin-reference/qtimetamux) | Synchronizes video and metadata. |
  | [`qtivcomposer`](../plugin-reference/qtivcomposer) | Composes the final output stream. |
  | [`waylandsink`](../plugin-reference/waylandsink) | Renders the output on a local display. |
</Accordion>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Model not available after running the download script?">
    If any model isn’t available after downloading the script file, you can download the model manually from [IoT — Qualcomm AI Hub](https://aihub.qualcomm.com)
    and push it to the target device:

    ```bash theme={null}
    scp <model filename> root@<ip addr of the target device>:${HOME}/Downloads/qimsdk_samples/models
    ```

    For example:

    ```bash theme={null}
    scp mobilenet_v2_quantized.tflite root@<ip addr of the target device>:${HOME}/Downloads/qimsdk_samples/models
    ```
  </Accordion>

  <Accordion title="Running sample applications from the UART shell">
    Remount the file system with read/write permissions:

    For Qualcomm Linux:

    ```bash theme={null}
    mount -o remount,rw /usr
    ```
  </Accordion>

  <Accordion title="Cannot locate the qticamsrc plugin?">
    If you cannot locate the qticamsrc plugin, ensure that the camera server is
    running and clear the GStreamer cache using the following commands:

    ```bash theme={null}
    ps -ef | grep cam-server 
    ```

    ```bash theme={null}
    rm ~/.cache/gstreamer-1.0/registry.aarch64.bin 
    ```
  </Accordion>

  <Accordion title="GStreamer logging and debugging">
    To enable basic GStreamer logging, run the following before launching the
    application:

    ```bash theme={null}
    export GST_DEBUG=2 
    ```

    To increase verbosity for specific
    plugins, use a comma-separated list with log levels (1–7):

    ```bash theme={null}
    export GST_DEBUG=3,qticamsrc:5,qtimlvconverter:5,qtimltflite:5 
    ```

    To redirect logs to a file for offline analysis:

    ```bash theme={null}
    export GST_DEBUG=3 
    export GST_DEBUG_FILE=/tmp/gst_classification_debug.log 
    ```
  </Accordion>
</AccordionGroup>


# Multimedia
Source: https://imsdkdocs.qualcomm.com/sample-application/python-multimedia-sample-applications

Ready-to-run Python multimedia applications demonstrating QIM SDK camera, video, and streaming capabilities

This section covers Python-based multimedia sample applications that demonstrate camera streaming, video playback, and transformation on Qualcomm platforms.

The following tables list all available Python multimedia applications and their platform support. Select the appropriate configuration tab for your setup.

<Tabs>
  <Tab title="Config #1">
    | Application | Source code | Description | QCS6490 | IQ-8275 | IQ-9075 | IQ-615 | IQx7181 | IQx5121 | IQ-2390 |
    | - | - | - | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
    | Camera encoding | [`gst-camera-encode.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-encode.py) | Records and encodes a single camera stream. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | Camera Streaming using OpenCV | [`gst-camera-opencv-resize.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-opencv-resize.py) | Captures and resizes a camera stream using OpenCV. | ✓ | ✓ | ✓ | ✓ | TBD | TBD | TBD |
    | Concurrent video playback (video wall) using Python | [`gst-concurrent-videoplay-composition.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-concurrent-videoplay-composition.py) | Concurrent MP4 (H.264) playback composed on a video wall. | ✓ | ✓ | ✓ | ✓ | × | ✓ | ✓ |
    | Video transformation using OpenCV | [`gst-opencv-transform.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-opencv-transform.py) | Captures and transforms a video frame using OpenCV. | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
    | Decode JPEG images using Python | [`gst-jpg-image-decode.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-jpg-image-decode.py) | Decodes JPEG images and displays them on the screen. | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
    | Transform and encode a camera stream | [`gst-camera-rotate-downscale-file.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-rotate-downscale-file.py) | Rotates, downscales, and encodes a camera stream to a file. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
  </Tab>

  <Tab title="Config #2">
    | Application | Source code | Description | QCS6490 | IQ-8275 | IQ-9075 | IQ-615 | IQx7181 | IQx5121 | IQ-2390 |
    | - | - | - | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
    | Camera encoding | [`gst-camera-encode.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-encode.py) | Records and encodes a single camera stream. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | Camera Streaming using OpenCV | [`gst-camera-opencv-resize.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-opencv-resize.py) | Captures and resizes a camera stream using OpenCV. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
    | Concurrent video playback (video wall) using Python | [`gst-concurrent-videoplay-composition.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-concurrent-videoplay-composition.py) | Concurrent MP4 (H.264) playback composed on a video wall. | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
    | Video transformation using OpenCV | [`gst-opencv-transform.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-opencv-transform.py) | Captures and transforms a video frame using OpenCV. | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
    | Multi-camera streaming using Python | [`gst-multi-camera-stream-example.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-multi-camera-stream-example.py) | Streams from two camera sensors simultaneously. | ✓ | ✓ | ✓ | × | ✓ | TBD | TBD |
    | Decode JPEG images using Python | [`gst-jpg-image-decode.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-jpg-image-decode.py) | Decodes JPEG images and displays them on the screen. | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
    | Transform and encode a camera stream | [`gst-camera-rotate-downscale-file.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-rotate-downscale-file.py) | Rotates, downscales, and encodes a camera stream to a file. | ✓ | ✓ | ✓ | ✓ | ✓ | TBD | TBD |
  </Tab>
</Tabs>

***

## Config 1: Supported Camera Slots by Platform

<Tabs>
  <Tab title="QCS6490">
    <Note>
      Connect the IMX577 MIPI camera sensor to the CAM3 slot of the RB3 device.
    </Note>
  </Tab>

  <Tab title="IQ-9075-EVK">
    <Note>
      Connect the IMX577 MIPI camera sensor to the CAM1 slot of the IQ-9075-EVK device.
    </Note>
  </Tab>

  <Tab title="IQ-8275-EVK">
    <Note>
      Connect the IMX577 MIPI camera sensor to the CAM1 slot of the IQ-8275-EVK device.
    </Note>
  </Tab>

  <Tab title="IQ615">
    <Note>
      Connect the IMX577 MIPI camera sensor to the CAM1 slot of the IQ615 device.
    </Note>
  </Tab>

  <Tab title="IQx7181">
    <Note>
      Connect the IMX577 MIPI camera sensor to the CAM1 (CSI1 22-pin) slot of the IQx7181 device.
    </Note>
  </Tab>
</Tabs>

***

## Prerequisites

See the [Prerequisites](/sample-application/python-ai-sample-applications#prerequisites) section in the Python AI page for device setup, Wi-Fi, artifact download, and enabling qticamsrc and libcamera.

***

## Camera Applications

### Camera Encoding

The [**gst-camera-encode.py**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-encode.py) application allows you to record and encode a single camera stream.

For information about the plugins used in this pipeline, see [Pipeline flow](#camera-encoding-py-pipeline).

<img alt="camera encode pipeline" />

**Application:** [`gst-camera-encode.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-encode.py)

<Accordion title="Try me">
  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      gst-camera-encode.py
      ```

      To display all available options:

      ```bash theme={null}
      gst-camera-encode.py -h
      ```
    </Step>

    <Step title="Pull output from device">
      ```bash theme={null}
      scp root@<IP address of target device>:$HOME/Downloads/qimsdk_samples/media/recording.mp4 .
      ```
    </Step>
  </Steps>

  ### Expected output

  The output is saved in a file at `$HOME/Downloads/qimsdk_samples/media/recording.mp4`.

  ### Pipeline flow

  | Plugin | Description |
  | - | - |
  | `qticamsrc` | Captures the camera live stream. Capsfilter enforces constraints on raw video data. |
  | `v4l2h264enc` | Encodes the video using H.264 format. |
  | `h264parse` | Parses the encoded video. |
  | `mp4mux` | Multiplexes the video into an MP4 container. |
  | `filesink` | Writes the video to a file. |
</Accordion>

***

### Camera Streaming Using OpenCV

The [**gst-camera-opencv-resize.py**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-opencv-resize.py) application uses OpenCV APIs to capture a camera frame from a live camera stream and apply color conversion or resize. The output is displayed on the screen.

For information about the plugins used in this pipeline, see [Pipeline flow](#camera-resize-py-pipeline).

<img alt="camera resize pipeline" />

**Application:** [`gst-camera-opencv-resize.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-opencv-resize.py)

<Accordion title="Try me">
  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      gst-camera-opencv-resize.py --inwidth 1280 --inheight 720 --outwidth 640 --outheight 480
      ```

      To display all available options:

      ```bash theme={null}
      gst-camera-opencv-resize.py -h
      ```
    </Step>
  </Steps>

  ### Expected output

  The stream is displayed on the screen.

  <img alt="Camera Resize Output" />

  ### Troubleshoot GTK backend failure in root mode

  When running a GTK-based application as root, GTK may fail to initialize the backend. Export the active display session before running:

  ```bash theme={null}
  ```

  ### Pipeline flow

  | Plugin | Description |
  | - | - |
  | `qticamsrc` | Captures the video stream and feeds it into the capsfilter. |
  | `capsfilter` | Enforces constraints on the raw video data. |
  | `queue` | Buffers video data. |
  | `qtivtransform` | Transforms the video data. |
  | `waylandsink` | Displays the video stream on the Wayland display. |
</Accordion>

***

### Multi-Camera Streaming Using Python

The [**gst-multi-camera-stream-example.py**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-multi-camera-stream-example.py) application streams from two camera sensors simultaneously, composing the feeds side by side or encoding to files.

For information about the plugins used in this pipeline, see [Pipeline flow](#multi-camera-streaming-py-pipeline).

<img alt="multi camera streaming pipeline" />

**Application:** [`gst-multi-camera-stream-example.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-multi-camera-stream-example.py)

<Note>
  This application isn't supported in the Config #1 of the QLI 2.0 RC3 release.
</Note>

<Accordion title="Try me">
  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="Run the application">
      * Preview on display:

      ```bash theme={null}
      gst-multi-camera-stream-example.py -D 1 --width=1920 --height=1080
      ```

      * Encode to file:

      ```bash theme={null}
      gst-multi-camera-stream-example.py -D 0 --width=1920 --height=1080 --framerate=30/1
      ```

      To display all available options:

      ```bash theme={null}
      gst-multi-camera-stream-example.py --help
      ```
    </Step>
  </Steps>

  ### Expected output

  For the video composition pipeline, the output is displayed as a preview.

  <img alt="Multi Camera Streaming Output" />

  ### Pipeline flow

  | Pipeline | Description |
  | - | - |
  | Preview on display | `qticamsrc` (cam0 + cam1) → capsfilter → `qtivcomposer` → `waylandsink` |
  | Encoder dump | `qticamsrc` (cam0 + cam1) → capsfilter → `v4l2h264enc` → `h264parse` → `mp4mux` → `filesink` |
</Accordion>

***

### Decode JPEG Images Using Python

The [**gst-jpg-image-decode.py**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-jpg-image-decode.py) application decodes JPEG images and displays them on a screen.

For information about the plugins used in this pipeline, see [Pipeline flow](#jpeg-decode-py-pipeline).

<img alt="jpeg decode pipeline" />

**Application:** [`gst-jpg-image-decode.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-jpg-image-decode.py)

<Accordion title="Try me">
  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      gst-jpg-image-decode.py -i $HOME/Downloads/qimsdk_samples/media/imagefiles_%d.jpg
      ```

      To display all available options:

      ```bash theme={null}
      gst-jpg-image-decode.py -h
      ```
    </Step>
  </Steps>

  ### Expected output

  The decoded images are displayed on the screen.

  <img alt="JPEG Decode Output" />

  ### Pipeline flow

  | Plugin | Description |
  | - | - |
  | `multifilesrc` | Reads video data from sequentially named files. |
  | `capsfilter` | Enforces constraints on the video data. |
  | `jpegdec` | Decodes the JPEG video stream. |
  | `videoconvert` | Converts video frames from one format to another. |
  | `waylandsink` | Displays the video stream on the Wayland display. |
</Accordion>

***

### Transform and Encode a Camera Stream

The [**gst-camera-rotate-downscale-file.py**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-rotate-downscale-file.py) application rotates, downscales, and encodes a camera stream, saving the result to a file.

For information about the plugins used in this pipeline, see [Pipeline flow](#transform-encode-py-pipeline).

<img alt="camera transform encode pipeline" />

**Application:** [`gst-camera-rotate-downscale-file.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-camera-rotate-downscale-file.py)

<Accordion title="Try me">
  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      gst-camera-rotate-downscale-file.py
      ```

      To display all available options:

      ```bash theme={null}
      gst-camera-rotate-downscale-file.py -h
      ```
    </Step>
  </Steps>

  ### Expected output

  The output is saved to a file at `$HOME/Downloads/qimsdk_samples/media/test.mp4`.

  ### Pipeline flow

  | Plugin | Description |
  | - | - |
  | `qticamsrc` | Captures the video stream. |
  | `qtivtransform` | Rotates and downscales the video stream. |
  | `v4l2h264enc` | Encodes H.264 video. |
  | `h264parse` | Parses the encoded H.264 video stream. |
  | `mp4mux` | Multiplexes the stream into an MP4 container. |
  | `filesink` | Writes the video data to a file. |
</Accordion>

***

### Dynamic Camera Overlay Using Python

The [**gst-camera-overlay.py**](https://github.com/qualcomm/gst-plugins-imsdk/blob/main/gst-python-examples/gst-camera-overlay.py) application demonstrates how to render dynamic text overlays on a live camera preview stream. The application injects overlay data into video buffers as ROI metadata and uses `qtivoverlay` to render text on the display. It supports runtime updates to overlay text, position, and color, and automatically selects the available camera.

For information about the plugins used in this pipeline, see [Pipeline flow](#gst-camera-overlay-pipeline).

<img alt="camera overlay pipeline" />

**Application:** [`gst-camera-overlay.py`](https://github.com/qualcomm/gst-plugins-imsdk/blob/main/gst-python-examples/gst-camera-overlay.py)

<Accordion title="Try me">
  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      gst-camera-overlay.py
      ```

      To display all available options:

      ```bash theme={null}
      gst-camera-overlay.py -h
      ```
    </Step>
  </Steps>

  ### Expected output

  Displays a live camera preview through waylandsink.

  ### Pipeline flow

  | Pipeline | Description |
  | - | - |
  | Preview on display | `qtiqmmfsrc` (cam0 + cam1) → queue → `qtivoverlay` → `waylandsink` |
</Accordion>

***

## Video and Audio Applications

### Concurrent Video Playback (Video Wall) Using Python

The [**gst-concurrent-videoplay-composition.py**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-concurrent-videoplay-composition.py) application supports concurrent video playback for MP4 AVC (H.264) videos and performs composition on a video wall display.

For information about the plugins used in this pipeline, see [Pipeline flow](#concurrent-playback-py-pipeline).

<img alt="concurrent playback pipeline" />

**Application:** [`gst-concurrent-videoplay-composition.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-concurrent-videoplay-composition.py)

<Accordion title="Try me">
  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      gst-concurrent-videoplay-composition.py --infile $HOME/Downloads/qimsdk_samples/media/video_avc.mp4
      ```

      To change the number of sessions, update `-c` with `2`, `4`, `8`, or `16`.

      To display all available options:

      ```bash theme={null}
      gst-concurrent-videoplay-composition.py -h
      ```
    </Step>
  </Steps>

  ### Expected output

  The individual composed streams are tiled together to display as a unified stream.

  <img alt="Concurrent Playback Output" />

  ### Pipeline flow

  | Plugin | Description |
  | - | - |
  | `filesrc` | Reads the video data. |
  | `qtdemux` | Demultiplexes the video data. |
  | `h264parse` | Parses H.264 video streams. |
  | `v4l2h264dec` | Decodes H.264 video streams. |
  | `qtivcomposer` | Composes decoded streams and sends to display. |
  | `waylandsink` | Displays the video stream on the Wayland display. |
</Accordion>

***

### Video Transformation Using OpenCV

The [**gst-opencv-transform.py**](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-opencv-transform.py) application captures and transforms a video frame then displays the video on the screen using the OpenCV `VideoCapture()` function.

For information about the plugins used in this pipeline, see [Pipeline flow](#camera-transform-py-pipeline).

<img alt="camera transform pipeline" />

**Application:** [`gst-opencv-transform.py`](https://github.com/qualcomm/qimsdk/blob/main/gstreamer/gst-python-examples/gst-opencv-transform.py)

<Accordion title="Try me">
  <Steps>
    <Step title="Complete prerequisites">
      Ensure that you complete the [Prerequisites](#prerequisites).
    </Step>

    <Step title="Run the application">
      ```bash theme={null}
      gst-opencv-transform.py --infile $HOME/Downloads/qimsdk_samples/media/video_avc.mp4
      ```

      To display all available options:

      ```bash theme={null}
      gst-opencv-transform.py -h
      ```
    </Step>
  </Steps>

  ### Expected output

  The stream is displayed on the screen.

  <img alt="Camera Transform Output" />

  ### Pipeline flow

  | Plugin | Description |
  | - | - |
  | `filesrc` | Reads the video data. |
  | `qtdemux` | Demultiplexes the video data. |
  | `h264parse` | Parses H.264 video streams. |
  | `v4l2h264dec` | Decodes H.264 video streams. |
  | `qtivtransform` | Transforms the video data. |
  | `videoconvert` | Converts video frames from one format to another. |
  | `capsfilter` | Enforces constraints on the video data. |
  | `waylandsink` | Displays the video stream on the Wayland display. |
</Accordion>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Model not available after running the download script?">
    Download the model manually from [IoT — Qualcomm AI Hub](https://aihub.qualcomm.com) and push it to the device:

    ```bash theme={null}
    scp <model filename> root@<IP addr of the target device>:/etc/models
    ```
  </Accordion>

  <Accordion title="Running sample applications from the UART shell">
    Remount the file system with read/write permissions:

    For Qualcomm Linux:

    ```bash theme={null}
    mount -o remount,rw /usr
    ```

    For Ubuntu Server:

    ```bash theme={null}
    mount -o remount,rw /
    ```
  </Accordion>

  <Accordion title="Cannot locate the qticamsrc plugin?">
    ```bash theme={null}
    ps -ef | grep cam-server
    rm ~/.cache/gstreamer-1.0/registry.aarch64.bin
    ```
  </Accordion>

  <Accordion title="GStreamer logging and debugging">
    ```bash theme={null}
    export GST_DEBUG=2
    export GST_DEBUG=3,qticamsrc:5,qtimlvconverter:5,qtimltflite:5
    export GST_DEBUG_FILE=/tmp/gst_debug.log
    ```
  </Accordion>
</AccordionGroup>


# AI
Source: https://imsdkdocs.qualcomm.com/sample-pipelines/aipipelines



This section covers QIM SDK AI pipelines that use LiteRT for inference.

## Vision AI Pipelines

### Object Detection

#### Single‑Stream Object Detection Pipeline.

Detects objects in each frame using a [YOLOX](https://aihub.qualcomm.com/iot/models/yolox) LiteRT model and overlays bounding boxes and labels.

**Pipeline Diagram**

<Frame>
  <img alt="Pipeline Diagram" />
</Frame>

***

<Accordion title="Try me">
  <Steps>
    <Step title="Download Required Files:">
      | File | Download | Save as |
      | - | - | - |
      | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
      | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
      | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

      <Note>
        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
        `unzip filename.zip`
      </Note>
    </Step>

    <Step title="Copy files to device">
      Create the required directories and transfer the downloaded files to your device.

      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
        # Run from your host machine — replace <user> and <device-ip>

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
        scp yolox_w8a8.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Connect to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Run from your host machine — replace <user> and <device-ip>
        ssh <user>@<device-ip>
        ```
      </CodeGroup>
    </Step>

    <Step title="Set environment variables">
      Run below command on your device

      ```bash theme={null}
      export MODEL_NAME=yolox_w8a8.tflite
      export LABELS_NAME=yolov8.json
      export SRC_VIDEO_NAME=ai_demo_sample.mp4
      ```
    </Step>

    <Step title="Run the pipeline">
      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
      t. ! queue ! qtimlvconverter ! queue ! \
      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME bbox-stabilization=true ! text/x-raw ! queue ! obj_mux.
      ```
    </Step>

    <Step title="Expected Output">
      The pipeline overlays bounding boxes and class labels on each video frame. Results are rendered on the display or saved to the output file.

      <img alt="Expected Output" />
    </Step>
  </Steps>

  **Object Detection Pipelines with Various Input and Output Configurations**

  <Note>
    Make sure you have completed **Download Required Files (Step 1)** and **Set Environment Variables (Step 2)** before running the pipelines below.
  </Note>

  **Render object detection result on display**

  <AccordionGroup>
    <Accordion title="Input:- filesrc">
      **Pipeline Diagram**

      <Frame>
        <img alt="Pipeline Diagram" />
      </Frame>

      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
      t. ! queue ! qtimlvconverter ! queue ! \
      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" bbox-stabilization=true ! text/x-raw ! queue ! obj_mux.
      ```
    </Accordion>

    <Accordion title="Input:- USB Camera(v4l2src)">
      **Pipeline Diagram**

      <Frame>
        <img alt="Pipeline Diagram" />
      </Frame>

      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      v4l2src device=/dev/video0 ! video/x-raw,format=YUY2 ! qtivtransform ! video/x-raw,format=NV12 ! queue ! \
      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
      t. ! queue ! qtimlvconverter ! queue ! \
      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" bbox-stabilization=true ! text/x-raw ! queue ! obj_mux.
      ```
    </Accordion>

    <Accordion title="Input:- RTSP(rtspsrc)">
      **Pipeline Diagram**

      <Frame>
        <img alt="Pipeline Diagram" />
      </Frame>

      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      rtspsrc location=rtsp://<ip>:<port>/stream ! rtph264depay ! h264parse ! \
      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
      t. ! queue ! qtimlvconverter ! queue ! \
      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" bbox-stabilization=true ! text/x-raw ! queue ! obj_mux.
      ```
    </Accordion>

    <Accordion title="Input:- ISP Camera (qticamsrc)">
      **Pipeline Diagram**

      <Frame>
        <img alt="Pipeline Diagram" />
      </Frame>

      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! \
      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
      t. ! queue ! qtimlvconverter ! queue ! \
      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" bbox-stabilization=true ! text/x-raw ! queue ! obj_mux.
      ```
    </Accordion>
  </AccordionGroup>

  **Encode object detection result into file**

  <AccordionGroup>
    <Accordion title="Input:- filesrc">
      **Pipeline Diagram**

      <Frame>
        <img alt="Pipeline Diagram" />
      </Frame>

      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! h264parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/obj_detect_out.mp4 \
      t. ! queue ! qtimlvconverter ! queue ! \
      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" bbox-stabilization=true ! text/x-raw ! queue ! obj_mux.
      ```
    </Accordion>

    <Accordion title="Input:- USB Camera(v4l2src)">
      **Pipeline Diagram**

      <Frame>
        <img alt="Pipeline Diagram" />
      </Frame>

      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      v4l2src device=/dev/video0 ! video/x-raw,format=YUY2 ! qtivtransform ! video/x-raw,format=NV12 ! queue ! \
      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! h264parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/obj_detect_out.mp4 \
      t. ! queue ! qtimlvconverter ! queue ! \
      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw bbox-stabilization=true ! queue ! obj_mux.
      ```
    </Accordion>

    <Accordion title="Input:- RTSP (rtspsrc)">
      **Pipeline Diagram**

      <Frame>
        <img alt="Pipeline Diagram" />
      </Frame>

      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      rtspsrc location=rtsp://<ip>:<port>/stream ! rtph264depay ! h264parse ! \
      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! h264parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/obj_detect_out.mp4 \
      t. ! queue ! qtimlvconverter ! queue ! \
      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" bbox-stabilization=true ! text/x-raw ! queue ! obj_mux.
      ```
    </Accordion>

    <Accordion title="Input:- ISP Camera (qticamsrc)">
      **Pipeline Diagram**

      <Frame>
        <img alt="Pipeline Diagram" />
      </Frame>

      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! \
      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! h264parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/output/obj_detect_out.mp4 \
      t. ! queue ! qtimlvconverter ! queue ! \
      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" bbox-stabilization=true ! text/x-raw ! queue ! obj_mux.
      ```
    </Accordion>
  </AccordionGroup>

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Reads an H.264 encoded video file as the pipeline source. |
    | [v4l2h264dec](../plugin-reference/v4l2h264dec) | Hardware-decodes the H.264 stream to raw NV12 frames using V4L2. |
    | tee | Duplicates the decoded video stream for parallel video passthrough and ML inference branches. |
    | [qtimlvconverter](../plugin-reference/qtimlvconverter) | Preprocesses video frames (color conversion, scaling, normalization) and converts to tensor stream. |
    | [qtimltflite](../plugin-reference/qtimltflite) | Loads the TFLite model, applies the chosen delegate, and runs inference to produce result tensors. |
    | [qtimlpostprocess](../plugin-reference/qtimlpostprocess) | Post-processes detection tensors, applies confidence threshold, and forwards bounding-box metadata. |
    | [qtimetamux](../plugin-reference/qtimetamux) | Merges video and metadata/text streams, attaching inference results as GST buffer metadata. |
    | [qtivoverlay](../plugin-reference/qtivoverlay) | Overlays inference results (labels, bounding boxes, keypoints) onto the video frame using CL. |
    | [v4l2h264enc](../plugin-reference/v4l2h264enc) | Hardware-encodes the video stream to H.264 using V4L2. |
    | filesink | Writes the encoded video stream to an output file. |
  </Accordion>
</Accordion>

#### Two‑Stream Object Detection Pipeline

Object detection on Stream 1 with side‑by‑side composition on Stream 2

**Pipeline Diagram**

<Frame>
  <img alt="Pipeline Diagram" />
</Frame>

***

<Accordion title="Try me">
  <Steps>
    <Step title="Download Required Files:">
      | File | Download | Save as |
      | - | - | - |
      | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
      | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |

      <Note>
        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
        `unzip filename.zip`
      </Note>
    </Step>

    <Step title="Copy files to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
        scp yolox_w8a8.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json          <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        ```
      </CodeGroup>
    </Step>

    <Step title="Connect to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Run from your host machine — replace <user> and <device-ip>
        ssh <user>@<device-ip>
        ```
      </CodeGroup>
    </Step>

    <Step title="Set environment variables">
      Run below command on your device

      ```bash theme={null}
      export MODEL_NAME=yolox_w8a8.tflite
      export LABELS_NAME=yolov8.json
      ```
    </Step>

    <Step title="Run the pipeline">
      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      qtivcomposer name=comp \
        sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" \
        sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! \
      queue ! waylandsink fullscreen=true sync=true \
      qtimetamux name=obj_mux ! queue ! qtivoverlay ! queue ! comp.sink_1 \
      qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! \
      tee name=t_src \
      t_src. ! queue ! comp.sink_0 \
      t_src. ! queue ! qtimlvconverter ! queue ! \
      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" bbox-stabilization=true ! text/x-raw ! queue ! obj_mux. \
      t_src. ! queue ! obj_mux.
      ```
    </Step>

    <Step title="Expected Output">
      The pipeline overlays bounding boxes and class labels on each video frame. Results are rendered on the display or saved to the output file.

      <img alt="Expected Output" />
    </Step>
  </Steps>

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | [qticamsrc](../plugin-reference/qticamsrc) | Captures live video from the ISP camera as the pipeline source. |
    | tee | Splits the camera stream into three branches: raw passthrough, ML inference, and metadata mux. |
    | [qtimlvconverter](../plugin-reference/qtimlvconverter) | Preprocesses video frames (color conversion, scaling, normalization) and converts to tensor stream. |
    | [qtimltflite](../plugin-reference/qtimltflite) | Loads the TFLite model, applies the chosen delegate, and runs inference to produce result tensors. |
    | [qtimlpostprocess](../plugin-reference/qtimlpostprocess) | Post-processes detection tensors, applies confidence threshold, and forwards bounding-box metadata. |
    | [qtimetamux](../plugin-reference/qtimetamux) | Merges video and metadata/text streams, attaching inference results as GST buffer metadata. |
    | [qtivoverlay](../plugin-reference/qtivoverlay) | Overlays inference results (labels, bounding boxes, keypoints) onto the video frame using CL. |
    | [qtivcomposer](../plugin-reference/qtivcomposer) | Composites the raw camera stream (sink\_0) and the overlay stream (sink\_1) side-by-side. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders the final composited video stream to a local display via Weston. |
  </Accordion>
</Accordion>

#### Three-Stream Object Detection Pipeline

Object detection on Stream 1, side‑by‑side composition on Stream 2, and video encoding to file on Stream 3

**Pipeline Diagram**

<Frame>
  <img alt="Pipeline Diagram" />
</Frame>

***

<Accordion title="Try me">
  <Steps>
    <Step title="Download Required Files:">
      | File | Download | Save as |
      | - | - | - |
      | YOLOX W8A8 model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
      | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |

      <Note>
        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
        `unzip filename.zip`
      </Note>
    </Step>

    <Step title="Copy files to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
        scp yolox_w8a8.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json          <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        ```
      </CodeGroup>
    </Step>

    <Step title="Connect to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Run from your host machine — replace <user> and <device-ip>
        ssh <user>@<device-ip>
        ```
      </CodeGroup>
    </Step>

    <Step title="Set environment variables">
      Run below command on your device

      ```bash theme={null}
      export MODEL_NAME=yolox_w8a8.tflite
      export LABELS_NAME=yolov8.json
      ```
    </Step>

    <Step title="Run the pipeline">
      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      qtivcomposer name=comp \
        sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" \
        sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! \
      queue ! waylandsink fullscreen=true sync=true \
      qtimetamux name=obj_mux ! queue ! tee name=ai_tee \
      ai_tee. ! queue ! qtivoverlay ! queue ! comp.sink_1 \
      ai_tee. ! queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! h264parse ! mp4mux ! \
      filesink location=$HOME/Downloads/qimsdk_samples/media/output/obj_detect_out.mp4 sync=false \
      qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! \
      tee name=t_src \
      t_src. ! queue ! comp.sink_0 \
      t_src. ! queue ! qtimlvconverter ! queue ! \
      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" bbox-stabilization=true ! text/x-raw ! queue ! obj_mux. \
      t_src. ! queue ! obj_mux.
      ```
    </Step>

    <Step title="Expected Output">
      The pipeline overlays bounding boxes and class labels on each video frame. Results are rendered on the display or saved to the output file.

      <img alt="Expected Output" />
    </Step>
  </Steps>

  ***

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | [qticamsrc](../plugin-reference/qticamsrc) | Captures live video from the ISP camera as the pipeline source. |
    | tee | Splits the stream into branches for display composition, ML inference, and file encoding. |
    | [qtimlvconverter](../plugin-reference/qtimlvconverter) | Preprocesses video frames (color conversion, scaling, normalization) and converts to tensor stream. |
    | [qtimltflite](../plugin-reference/qtimltflite) | Loads the TFLite model, applies the chosen delegate, and runs inference to produce result tensors. |
    | [qtimlpostprocess](../plugin-reference/qtimlpostprocess) | Post-processes detection tensors, applies confidence threshold, and forwards bounding-box metadata. |
    | [qtimetamux](../plugin-reference/qtimetamux) | Merges video and metadata/text streams, attaching inference results as GST buffer metadata. |
    | [qtivcomposer](../plugin-reference/qtivcomposer) | Composites the raw camera stream (sink\_0) and the overlay stream (sink\_1) side-by-side. |
    | [qtivoverlay](../plugin-reference/qtivoverlay) | Overlays inference results (labels, bounding boxes, keypoints) onto the video frame using CL. |
    | [v4l2h264enc](../plugin-reference/v4l2h264enc) | Hardware-encodes the video stream to H.264 using V4L2. |
    | filesink | Writes the encoded video stream to an output file. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders the final composited video stream to a local display via Weston. |
  </Accordion>
</Accordion>

***

### Face Detection

Detects faces using a quantized [Face Detection Lite](https://aihub.qualcomm.com/iot/models/face_det_lite) model accelerated via QNN (HTP backend).

**Pipeline Diagram**

<Frame>
  <img alt="Pipeline Diagram" />
</Frame>

***

<Accordion title="Try me">
  <Steps>
    <Step title="Download Required Files:">
      | File | Download | Save as |
      | - | - | - |
      | Face Detection Lite model | [Qualcomm AI Hub — Face Detection Lite](https://aihub.qualcomm.com/iot/models/face_det_lite) | `face_det_lite_w8a8.tflite` |
      | Detection labels | <a href="../labels/face_det_lite.json">face\_det\_lite labels</a> | `face_det_lite.json` |
      | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

      <Note>
        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
        `unzip filename.zip`
      </Note>
    </Step>

    <Step title="Copy files to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
        scp face_det_lite_w8a8.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp face_det_lite.json    <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4                              <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Connect to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Run from your host machine — replace <user> and <device-ip>
        ssh <user>@<device-ip>
        ```
      </CodeGroup>
    </Step>

    <Step title="Set environment variables">
      Run below command on your device

      ```bash theme={null}
      export MODEL_NAME=face_det_lite_w8a8.tflite
      export LABELS_NAME=face_det_lite.json
      export SRC_VIDEO_NAME=ai_demo_sample.mp4
      ```
    </Step>

    <Step title="Run the pipeline">
      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
      tee name=t ! qtimetamux name=face_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
      t. ! queue ! qtimlvconverter ! queue ! \
      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
      qtimlpostprocess module=qfd labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! text/x-raw ! queue ! face_mux.
      ```
    </Step>

    <Step title="Expected Output">
      The pipeline detects faces and overlays bounding boxes on each frame. Results are rendered on the display or saved to the output file.

      <img alt="Expected Output" />
    </Step>
  </Steps>

  ***

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Reads an H.264 encoded video file as the pipeline source. |
    | [v4l2h264dec](../plugin-reference/v4l2h264dec) | Hardware-decodes the H.264 stream to raw NV12 frames using V4L2. |
    | tee | Splits the decoded stream for video passthrough and ML inference branches. |
    | [qtimlvconverter](../plugin-reference/qtimlvconverter) | Preprocesses video frames (color conversion, scaling, normalization) and converts to tensor stream. |
    | [qtimltflite](../plugin-reference/qtimltflite) | Loads the TFLite model, applies the chosen delegate, and runs inference to produce result tensors. |
    | [qtimlpostprocess](../plugin-reference/qtimlpostprocess) | Post-processes face detection tensors and forwards bounding-box/landmark metadata. |
    | [qtimetamux](../plugin-reference/qtimetamux) | Merges video and metadata/text streams, attaching inference results as GST buffer metadata. |
    | [qtivoverlay](../plugin-reference/qtivoverlay) | Overlays inference results (labels, bounding boxes, keypoints) onto the video frame using CL. |
    | [v4l2h264enc](../plugin-reference/v4l2h264enc) | Hardware-encodes the video stream to H.264 using V4L2. |
    | filesink | Writes the encoded video stream to an output file. |
  </Accordion>
</Accordion>

***

### Image Classification

Classifies each video frame into predefined scene categories using the [MobileNet-v2](https://aihub.qualcomm.com/iot/models/mobilenet_v2) LiteRT model and overlays the top classification results on the video stream.

**Pipeline Diagram**

<Frame>
  <img alt="Pipeline Diagram" />
</Frame>

***

<Accordion title="Try me">
  <Steps>
    <Step title="Download Required Files:">
      | File | Download | Save as |
      | - | - | - |
      | MobileNet-v2 model | [Qualcomm AI Hub — MobileNet-v2](https://aihub.qualcomm.com/iot/models/mobilenet_v2) | `mobilenet_v2_w8a8.tflite` |
      | Classification labels | <a href="../labels/mobilenet.json">mobilenet.json</a> | `mobilenet.json` |
      | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

      <Note>
        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
        `unzip filename.zip`
      </Note>
    </Step>

    <Step title="Copy files to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
        scp mobilenet_v2_w8a8.tflite       <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp mobilenet.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4      <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Connect to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Run from your host machine — replace <user> and <device-ip>
        ssh <user>@<device-ip>
        ```
      </CodeGroup>
    </Step>

    <Step title="Set environment variables">
      Run below command on your device

      ```bash theme={null}
      export MODEL_NAME=mobilenet_v2_w8a8.tflite
      export LABELS_NAME=mobilenet.json
      export SRC_VIDEO_NAME=ai_demo_sample.mp4
      ```
    </Step>

    <Step title="Run the pipeline">
      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
      t. ! queue ! qtimlvconverter ! queue ! \
      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
      qtimlpostprocess module=mobilenet results=1 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
      ```
    </Step>

    <Step title="Expected Output">
      The pipeline classifies each frame and overlays the top label in the top-left corner of the frame. Results are rendered on the display or saved to the output file.

      <img alt="Image of a elephant classification" />
    </Step>
  </Steps>

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Reads an H.264 encoded video file as the pipeline source. |
    | [v4l2h264dec](../plugin-reference/v4l2h264dec) | Hardware-decodes the H.264 stream to raw NV12 frames using V4L2. |
    | tee | Splits the decoded stream for video passthrough and ML inference branches. |
    | [qtimlvconverter](../plugin-reference/qtimlvconverter) | Preprocesses video frames (color conversion, scaling, normalization) and converts to tensor stream. |
    | [qtimltflite](../plugin-reference/qtimltflite) | Loads the TFLite model, applies the chosen delegate, and runs inference to produce result tensors. |
    | [qtimlpostprocess](../plugin-reference/qtimlpostprocess) | Post-processes classification tensors, applies confidence threshold, and produces top-N label text. |
    | [qtimetamux](../plugin-reference/qtimetamux) | Merges video and metadata/text streams, attaching inference results as GST buffer metadata. |
    | [qtivoverlay](../plugin-reference/qtivoverlay) | Overlays inference results (labels, bounding boxes, keypoints) onto the video frame using CL. |
    | [v4l2h264enc](../plugin-reference/v4l2h264enc) | Hardware-encodes the video stream to H.264 using V4L2. |
    | filesink | Writes the encoded video stream to an output file. |
  </Accordion>
</Accordion>

### Segmentation

Performs pixel-wise semantic segmentation using [DeepLabV3+](https://aihub.qualcomm.com/iot/models/deeplabv3_plus_mobilenet) and blends the segmentation mask with the original video.

**Pipeline Diagram**

<Frame>
  <img alt="Pipeline Diagram" />
</Frame>

***

<Accordion title="Try me">
  <Steps>
    <Step title="Download Required Files:">
      | File | Download | Save as |
      | - | - | - |
      | DeepLabV3+ model | [Qualcomm AI Hub — DeepLabV3+](https://aihub.qualcomm.com/iot/models/deeplabv3_plus_mobilenet) | `deeplabv3_plus_mobilenet_w8a8.tflite` |
      | Segmentation labels | <a href="../labels/dv3-argmax.json">dv3-argmax.json</a> | `dv3-argmax.json` |
      | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

      <Note>
        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
        `unzip filename.zip`
      </Note>
    </Step>

    <Step title="Copy files to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
        scp deeplabv3_plus_mobilenet_w8a8.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp dv3-argmax.json                        <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4             <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Connect to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Run from your host machine — replace <user> and <device-ip>
        ssh <user>@<device-ip>
        ```
      </CodeGroup>
    </Step>

    <Step title="Set environment variables">
      Run below command on your device

      ```bash theme={null}
      export MODEL_NAME=deeplabv3_plus_mobilenet_w8a8.tflite
      export LABELS_NAME=dv3-argmax.json
      export SRC_VIDEO_NAME=ai_demo_sample.mp4
      ```
    </Step>

    <Step title="Run the pipeline">
      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
      tee name=t \
      t. ! queue ! qtivcomposer name=seg_mix sink_1::alpha=0.5 ! queue ! waylandsink fullscreen=true sync=true \
      t. ! queue ! qtimlvconverter ! queue ! \
      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
      qtimlpostprocess module=deeplab-argmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw ! queue ! seg_mix.
      ```
    </Step>

    <Step title="Expected Output">
      The pipeline blends the segmentation mask with the original video frame. Results are rendered on the display or saved to the output file.

      <img alt="Expected Output" />
    </Step>
  </Steps>

  ***

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Reads an H.264 encoded video file as the pipeline source. |
    | [v4l2h264dec](../plugin-reference/v4l2h264dec) | Hardware-decodes the H.264 stream to raw NV12 frames using V4L2. |
    | [qtivtransform](../plugin-reference/qtivtransform) | Performs GPU-accelerated color/format conversion on the video frame. |
    | tee | Splits the stream for video passthrough and ML inference branches. |
    | [qtimlvconverter](../plugin-reference/qtimlvconverter) | Preprocesses video frames (color conversion, scaling, normalization) and converts to tensor stream. |
    | [qtimltflite](../plugin-reference/qtimltflite) | Loads the TFLite model, applies the chosen delegate, and runs inference to produce result tensors. |
    | [qtimlpostprocess](../plugin-reference/qtimlpostprocess) | Applies argmax post-processing to segmentation tensors and outputs an RGBA mask frame. |
    | [qtivcomposer](../plugin-reference/qtivcomposer) | Blends the original video frame with the segmentation mask (alpha composite). |
    | [v4l2h264enc](../plugin-reference/v4l2h264enc) | Hardware-encodes the video stream to H.264 using V4L2. |
    | filesink | Writes the encoded video stream to an output file. |
  </Accordion>
</Accordion>

***

### Pose Estimation

This pipeline performs real-time Human Pose Estimation using the [HRNet Pose](https://aihub.qualcomm.com/iot/models/hrnet_pose) model. It analyzes video frames to identify individuals and precisely maps their anatomical keypoints (such as shoulders, elbows, knees, and ankles). It then generates a skeletal overlay on the video stream, allowing for the tracking of body posture and movement dynamics.

**Pipeline Diagram**

<Frame>
  <img alt="Pipeline Diagram" />
</Frame>

***

<Accordion title="Try me">
  <Steps>
    <Step title="Download Required Files:">
      | File | Download | Save as |
      | - | - | - |
      | Person/foot detection model | [Qualcomm AI Hub — HRNet Pose](https://aihub.qualcomm.com/iot/models/foot_track_net) | `person_foot_detection_w8a8.tflite` |
      | Person detection labels | <a href="../labels/foot_track_net.json">foot\_track\_net.json</a> | `foot_track_net.json` |
      | Foot track net settings | <a href="../labels/foot_track_net_settings.json">foot\_track\_net\_settings.json</a> | `foot_track_net_settings.json` |
      | HRNet pose model | [Qualcomm AI Hub — HRNet Pose](https://aihub.qualcomm.com/iot/models/hrnet_pose) | `hrnetpose_w8a8.tflite` |
      | Pose labels | <a href="../labels/hrnet.json">hrnet.json</a> | `hrnet.json` |
      | HRNet settings | <a href="../labels/hrnet_settings.json">hrnet\_settings.json</a> | `hrnet_settings.json` |
      | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `pose_sample.mp4` |

      <Note>
        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
        `unzip filename.zip`
      </Note>
    </Step>

    <Step title="Copy files to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
        scp person_foot_detection_w8a8.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp foot_track_net.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp foot_track_net_settings.json       <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp hrnetpose_w8a8.tflite              <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp hrnet.json                         <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp hrnet_settings.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp pose_sample.mp4                    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Connect to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Run from your host machine — replace <user> and <device-ip>
        ssh <user>@<device-ip>
        ```
      </CodeGroup>
    </Step>

    <Step title="Set environment variables">
      Run below command on your device

      ```bash theme={null}
      export MODEL_NAME_1=person_foot_detection_w8a8.tflite
      export LABELS_NAME_1=foot_track_net.json
      export SETTINGS_1=foot_track_net_settings.json
      export MODEL_NAME_2=hrnetpose_w8a8.tflite
      export LABELS_NAME_2=hrnet.json
      export SETTINGS_2=hrnet_settings.json
      export SRC_VIDEO_NAME=pose_sample.mp4
      ```
    </Step>

    <Step title="Run the pipeline">
      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
        qtimlvconverter name=stage_01_preproc \
        qtimltflite name=stage_01_inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
        external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" \
        model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_1 \
        qtimlpostprocess name=stage_01_postproc results=10 module=qpd labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_1 \
        settings=$HOME/Downloads/qimsdk_samples/labels/$SETTINGS_1 \
        qtimlvconverter name=stage_02_preproc mode=roi-batch-cumulative image-disposition=centre \
        qtimltflite name=stage_02_inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
        external-delegate-options="QNNExternalDelegate,backend_type=htp,htp_performance_mode=(string)2,log_level=(string)1;" \
        model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_2 \
        qtimlpostprocess name=stage_02_postproc results=2 module=hrnet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_2 \
        settings=$HOME/Downloads/qimsdk_samples/labels/$SETTINGS_2 \
        filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
        v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
        tee name=t_split_1 \
        t_split_1. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! stage_01_inference. stage_01_inference. ! queue ! \
        stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! qtimetamux name=metamux_1 \
        t_split_1. ! queue ! metamux_1. metamux_1. ! queue ! tee name=t_split_2 \
        t_split_2. ! queue ! stage_02_preproc. stage_02_preproc. ! queue ! stage_02_inference. stage_02_inference. ! queue ! \
        stage_02_postproc. stage_02_postproc. ! text/x-raw ! queue ! qtimetamux name=metamux_2 \
        metamux_2. ! queue ! qtivoverlay ! queue ! waylandsink fullscreen=true sync=true \
        t_split_2. ! queue ! metamux_2.
      ```
    </Step>

    <Step title="Expected Output">
      The pipeline detects persons and overlays skeleton keypoints on each frame. Results are rendered on the display or saved to the output file.

      <img alt="Expected Output" />
    </Step>
  </Steps>

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Reads an H.264 encoded video file as the pipeline source. |
    | [v4l2h264dec](../plugin-reference/v4l2h264dec) | Hardware-decodes the H.264 stream to raw NV12 frames using V4L2. |
    | [qtivtransform](../plugin-reference/qtivtransform) | Performs GPU-accelerated color/format conversion on the video frame. |
    | tee | Splits the stream for video passthrough and person-detection inference. |
    | [qtimlvconverter](../plugin-reference/qtimlvconverter) | Preprocesses frames for Stage 1 (person detection) and Stage 2 (pose estimation) respectively. |
    | [qtimltflite](../plugin-reference/qtimltflite) | Runs Stage 1 (foot/person detection) and Stage 2 (HRNet pose estimation) inference sequentially. |
    | [qtimlpostprocess](../plugin-reference/qtimlpostprocess) | Post-processes detection and pose tensors, producing keypoint metadata for overlay. |
    | [qtimetamux](../plugin-reference/qtimetamux) | Merges video and metadata/text streams, attaching inference results as GST buffer metadata. |
    | [qtivoverlay](../plugin-reference/qtivoverlay) | Overlays inference results (labels, bounding boxes, keypoints) onto the video frame using CL. |
    | [v4l2h264enc](../plugin-reference/v4l2h264enc) | Hardware-encodes the video stream to H.264 using V4L2. |
    | filesink | Writes the encoded video stream to an output file. |
  </Accordion>
</Accordion>

### AI Wall

This use-case demonstrates the capability to run **4 parallel AI inference sessions** simultaneously using [MobileNet-v2](https://aihub.qualcomm.com/iot/models/mobilenet_v2), [Face Detection Lite](https://aihub.qualcomm.com/iot/models/face_det_lite), [DeepLabV3+](https://aihub.qualcomm.com/iot/models/deeplabv3_plus_mobilenet), and [YOLOX](https://aihub.qualcomm.com/iot/models/yolox). The results are composed into a single 2x2 grid display. This use case highlights the multi-stream processing and compositing capabilities of the platform.

**Pipeline Diagram**

<Frame>
  <img alt="Pipeline Diagram" />
</Frame>

***

<Accordion title="Try me">
  <Steps>
    <Step title="Download Required Files:">
      | File | Download | Save as |
      | - | - | - |
      | Classification model | [Qualcomm AI Hub — MobileNet-v2](https://aihub.qualcomm.com/iot/models/mobilenet_v2) | `mobilenet_v2_w8a8.tflite` |
      | Classification labels | <a href="../labels/mobilenet.json">mobilenet.json</a> | `mobilenet.json` |
      | Face detection model | [Qualcomm AI Hub — Face Detection Lite](https://aihub.qualcomm.com/iot/models/face_det_lite) | `face_det_lite_w8a8.tflite` |
      | Face detection labels | <a href="../labels/face_det_lite.json">face\_det\_lite labels</a> | `face_det_lite.json` |
      | Segmentation model | [Qualcomm AI Hub — DeepLabV3+](https://aihub.qualcomm.com/iot/models/deeplabv3_plus_mobilenet) | `deeplabv3_plus_mobilenet_w8a8.tflite` |
      | Segmentation labels | <a href="../labels/dv3-argmax.json">dv3-argmax.json</a> | `dv3-argmax.json` |
      | Object detection model | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
      | Object detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
      | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

      <Note>
        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
        `unzip filename.zip`
      </Note>
    </Step>

    <Step title="Copy files to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
        scp mobilenet_v2_w8a8.tflite            <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp mobilenet.json                       <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp face_det_lite_w8a8.tflite            <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp face_det_lite.json                   <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp deeplabv3_plus_mobilenet_w8a8.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp dv3-argmax.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp yolox_w8a8.tflite                    <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                          <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4            <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Connect to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Run from your host machine — replace <user> and <device-ip>
        ssh <user>@<device-ip>
        ```
      </CodeGroup>
    </Step>

    <Step title="Set environment variables">
      Run below command on your device

      ```bash theme={null}
      export MODEL_NAME_1=mobilenet_v2_w8a8.tflite
      export LABELS_NAME_1=mobilenet.json
      export MODEL_NAME_2=face_det_lite_w8a8.tflite
      export LABELS_NAME_2=face_det_lite.json
      export MODEL_NAME_3=deeplabv3_plus_mobilenet_w8a8.tflite
      export LABELS_NAME_3=dv3-argmax.json
      export MODEL_NAME_4=yolox_w8a8.tflite
      export LABELS_NAME_4=yolov8.json
      export SRC_VIDEO_NAME=ai_demo_sample.mp4
      ```
    </Step>

    <Step title="Run the pipeline">
      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      qtimlvconverter name=class_pre \
      qtimltflite name=class_infer model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_1 delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" \
      qtimlpostprocess name=class_post results=5 module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_1 settings="{\"confidence\": 51.0}" \
      qtimetamux name=class_mux \
      qtivoverlay name=class_overlay \
      qtimlvconverter name=face_pre \
      qtimltflite name=face_infer model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_2 delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" \
      qtimlpostprocess name=face_post module=qfd results=6 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_2 \
      qtimetamux name=face_mux \
      qtivoverlay name=face_overlay \
      qtimlvconverter name=seg_pre \
      qtimltflite name=seg_infer model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_3 delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" \
      qtimlpostprocess name=seg_post module=deeplab-argmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_3 \
      qtivcomposer name=seg_mix sink_1::alpha=0.5 \
      qtimlvconverter name=obj_pre \
      qtimltflite name=obj_infer model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_4 delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" \
      qtimlpostprocess name=obj_post module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_4 settings="{\"confidence\": 51.0}" \
      qtimetamux name=obj_mux \
      qtivcomposer name=comp \
        sink_0::position="<0, 0>" sink_0::dimensions="<960, 540>" \
        sink_1::position="<960, 0>" sink_1::dimensions="<960, 540>" \
        sink_2::position="<0, 540>" sink_2::dimensions="<960, 540>" \
        sink_3::position="<960, 540>" sink_3::dimensions="<960, 540>" ! \
      queue ! waylandsink fullscreen=true sync=true \
      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
      tee name=class_tee \
      class_tee. ! queue ! class_mux. \
      class_tee. ! queue ! class_pre. class_pre. ! queue ! class_infer. class_infer. ! queue ! class_post. class_post. ! text/x-raw ! queue ! class_mux. \
      class_mux. ! queue ! class_overlay. class_overlay. ! queue ! comp.sink_0 \
      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
      tee name=face_tee \
      face_tee. ! queue ! face_mux. \
      face_tee. ! queue ! face_pre. face_pre. ! queue ! face_infer. face_infer. ! queue ! face_post. face_post. ! text/x-raw ! queue ! face_mux. \
      face_mux. ! queue ! face_overlay. face_overlay. ! queue ! comp.sink_1 \
      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
      tee name=seg_tee \
      seg_tee. ! queue ! seg_mix. \
      seg_tee. ! queue ! seg_pre. seg_pre. ! queue ! seg_infer. seg_infer. ! queue ! seg_post. seg_post. ! video/x-raw ! queue ! seg_mix. \
      seg_mix. ! video/x-raw,format=NV12 ! queue ! comp.sink_2 \
      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
      tee name=obj_tee \
      obj_tee. ! queue ! obj_mux. \
      obj_tee. ! queue ! obj_pre. obj_pre. ! queue ! obj_infer. obj_infer. ! queue ! obj_post. obj_post. ! text/x-raw ! queue ! obj_mux. \
      obj_mux. ! queue ! qtivoverlay ! queue ! comp.sink_3
      ```
    </Step>

    <Step title="Expected Output">
      The pipeline processes multiple streams simultaneously and renders all detection results in a composed multi-stream view on the display.

      <img alt="Expected Output" />
    </Step>
  </Steps>

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Four independent file sources feed the four parallel AI branches. |
    | [v4l2h264dec](../plugin-reference/v4l2h264dec) | Hardware-decodes each H.264 stream to raw NV12 frames using V4L2. |
    | tee | Splits each branch stream for video passthrough and ML inference. |
    | [qtimlvconverter](../plugin-reference/qtimlvconverter) | Preprocesses each branch's video frames into tensors for inference. |
    | [qtimltflite](../plugin-reference/qtimltflite) | Runs branch-specific inference: classification, face detection, segmentation, and object detection. |
    | [qtimlpostprocess](../plugin-reference/qtimlpostprocess) | Post-processes each branch's tensors (labels, bounding boxes, masks) for overlay or compositing. |
    | [qtimetamux](../plugin-reference/qtimetamux) | Merges video and metadata/text streams, attaching inference results as GST buffer metadata. |
    | [qtivcomposer](../plugin-reference/qtivcomposer) | Composites all four inference-overlaid streams into a 2×2 grid display. |
    | [qtivoverlay](../plugin-reference/qtivoverlay) | Overlays inference results (labels, bounding boxes, keypoints) onto the video frame using CL. |
    | [v4l2h264enc](../plugin-reference/v4l2h264enc) | Hardware-encodes the video stream to H.264 using V4L2. |
    | filesink | Writes the encoded video stream to an output file. |
  </Accordion>
</Accordion>

***

### Super Resolution

Real-time AI video upscaling using [quicksrnetlarge](https://aihub.qualcomm.com/iot/models/quicksrnetlarge) that reconstructs high-definition details from low-resolution inputs, visualized via a side-by-side comparison.

**Pipeline Diagram**

<Frame>
  <img alt="Pipeline Diagram" />
</Frame>

***

<Accordion title="Try me">
  <Steps>
    <Step title="Download Required Files:">
      | File | Download | Save as |
      | - | - | - |
      | QuickSRNet Large model | [Qualcomm AI Hub — QuickSRNet Large](https://aihub.qualcomm.com/iot/models/quicksrnetlarge) | `quicksrnetlarge_w8a8.tflite` |
      | Sample video | Use an input video of resolution 128x128 or similar low-resolution source | `ai_demo_sample.mp4` |

      <Note>
        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
        `unzip filename.zip`
      </Note>
    </Step>

    <Step title="Copy files to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
        scp quicksrnetlarge_w8a8.tflite    <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp ai_demo_sample.mp4      <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Connect to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Run from your host machine — replace <user> and <device-ip>
        ssh <user>@<device-ip>
        ```
      </CodeGroup>
    </Step>

    <Step title="Set environment variables">
      Run below command on your device

      ```bash theme={null}
      export MODEL_NAME=quicksrnetlarge_w8a8.tflite
      export SRC_VIDEO_NAME=ai_demo_sample.mp4
      ```
    </Step>

    <Step title="Run the pipeline">
      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
      tee name=t \
      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
      t. ! qtimlvconverter ! queue ! \
      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
      ```
    </Step>

    <Step title="Expected Output">
      The pipeline outputs an upscaled high-resolution video. Results are rendered on the display or saved to the output file.

      <img alt="Expected Output" />
    </Step>
  </Steps>

  ***

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Reads an H.264 encoded video file as the pipeline source. |
    | [v4l2h264dec](../plugin-reference/v4l2h264dec) | Hardware-decodes the H.264 stream to raw NV12 frames using V4L2. |
    | tee | Splits the decoded stream — one branch for the original view, one for SR inference. |
    | [qtimlvconverter](../plugin-reference/qtimlvconverter) | Preprocesses video frames (color conversion, scaling, normalization) and converts to tensor stream. |
    | [qtimltflite](../plugin-reference/qtimltflite) | Loads the TFLite model, applies the chosen delegate, and runs inference to produce result tensors. |
    | [qtimlpostprocess](../plugin-reference/qtimlpostprocess) | Post-processes tensors to reconstruct high-definition output. |
    | [qtivcomposer](../plugin-reference/qtivcomposer) | Composites the original and upscaled streams side-by-side for comparison. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders the final composited video stream to a local display via Weston. |
  </Accordion>
</Accordion>

***

### Daisy Chain

***

#### Detection-Classification Daisy Chain

This section details the Detection-Classification Daisy Chain pipeline. This pipeline demonstrates a cascaded inference approach where the output of the [YOLOX](https://aihub.qualcomm.com/iot/models/yolox) detection model is used to crop regions of interest (ROIs) which are then fed into the [MobileNet-v2](https://aihub.qualcomm.com/iot/models/mobilenet_v2) classification model.

**Pipeline Diagram**

<Frame>
  <img alt="Pipeline Diagram" />
</Frame>

***

<Accordion title="Try me">
  <Steps>
    <Step title="Download Required Files:">
      | File | Download | Save as |
      | - | - | - |
      | Detection model (YOLOX) | [Qualcomm AI Hub — YOLOX](https://aihub.qualcomm.com/iot/models/yolox) | `yolox_w8a8.tflite` |
      | Detection labels | <a href="../labels/yolov8.json">yolov8.json</a> | `yolov8.json` |
      | Classification model (MobileNet-v2) | [Qualcomm AI Hub — MobileNet-v2](https://aihub.qualcomm.com/iot/models/mobilenet_v2) | `mobilenet_v2_w8a8.tflite` |
      | Classification labels | <a href="../labels/mobilenet.json">mobilenet.json</a> | `mobilenet.json` |
      | Sample video | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | `ai_demo_sample.mp4` |

      <Note>
        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
        `unzip filename.zip`
      </Note>
    </Step>

    <Step title="Copy files to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
        scp yolox_w8a8.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp mobilenet_v2_w8a8.tflite    <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp mobilenet.json               <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Connect to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Run from your host machine — replace <user> and <device-ip>
        ssh <user>@<device-ip>
        ```
      </CodeGroup>
    </Step>

    <Step title="Set environment variables">
      Run below command on your device

      ```bash theme={null}
      export MODEL_NAME_1=yolox_w8a8.tflite
      export LABELS_NAME_1=yolov8.json
      export MODEL_NAME_2=mobilenet_v2_w8a8.tflite
      export LABELS_NAME_2=mobilenet.json
      export SRC_VIDEO_NAME=ai_demo_sample.mp4
      ```
    </Step>

    <Step title="Run the pipeline">
      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
        qtimlvconverter name=stage_01_preproc \
        qtimltflite name=stage_01_inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
        external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" \
        model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_1 \
        qtimlpostprocess results=1 name=stage_01_postproc module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_1 \
        settings="{\"confidence\": 51.0}" \
        qtimetamux name=metamux_1 \
        qtivoverlay name=main_overlay \
        qtimlvconverter name=stage_02_preproc \
        qtimltflite name=stage_02_inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
        external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" \
        model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_2 \
        qtimlpostprocess results=1 name=stage_02_postproc module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_2 \
        settings="{\"confidence\": 51.0}" \
        qtimetamux name=metamux_2 \
        qtivoverlay name=cls_overlay \
        filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
        v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
        tee name=t_split_1 \
        t_split_1. ! queue ! metamux_1. \
        t_split_1. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! stage_01_inference. stage_01_inference. ! queue ! \
        stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! metamux_1. \
        metamux_1. ! queue ! tee name=t_split_2 \
        t_split_2. ! queue ! metamux_2. \
        t_split_2. ! queue ! stage_02_preproc. stage_02_preproc. ! queue ! stage_02_inference. stage_02_inference. ! queue ! \
        stage_02_postproc. stage_02_postproc. ! text/x-raw ! queue ! metamux_2. \
        metamux_2. ! queue ! cls_overlay. cls_overlay. ! queue ! waylandsink sync=true fullscreen=true
      ```
    </Step>

    <Step title="Expected Output">
      The pipeline detects one object with the highest confidence, classifies the object detected and overlays the bounding box and top label in the top-left corner of the frame. Results are rendered on the display or saved to the output file.

      <img alt="Image of a camel classification" />
    </Step>
  </Steps>

  ***

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Reads an H.264 encoded video file as the pipeline source. |
    | [v4l2h264dec](../plugin-reference/v4l2h264dec) | Hardware-decodes the H.264 stream to raw NV12 frames using V4L2. |
    | tee | Splits the stream for Stage 1 video passthrough and YOLOX detection inference. |
    | [qtimlvconverter](../plugin-reference/qtimlvconverter) | Preprocesses frames for Stage 1 (YOLOX detection) and Stage 2 (MobileNet classification). |
    | [qtimltflite](../plugin-reference/qtimltflite) | Runs YOLOX (Stage 1) and MobileNet (Stage 2) inference sequentially. |
    | [qtimlpostprocess](../plugin-reference/qtimlpostprocess) | Post-processes detection and classification tensors, forwarding structured metadata. |
    | [qtimetamux](../plugin-reference/qtimetamux) | Merges video and metadata/text streams, attaching inference results as GST buffer metadata. |
    | [qtivoverlay](../plugin-reference/qtivoverlay) | Overlays inference results (labels, bounding boxes, keypoints) onto the video frame using CL. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders the final composited video stream to a local display via Weston. |
  </Accordion>
</Accordion>

#### Gesture Recognition

A four-stage cascading pipeline that performs palm detection, hand landmark estimation, gesture embedding, and gesture classification on a live camera stream using ROI-based metadata propagation.

**Pipeline Diagram**

<Frame>
  <img alt="Pipeline Diagram" />
</Frame>

***

<Accordion title="Try me">
  <Steps>
    <Step title="Download Required Files:">
      Download the gesture recognizer models from Google MediaPipe:

      ```bash theme={null}
      # Download the gesture recognizer task bundle
      wget https://storage.googleapis.com/mediapipe-models/gesture_recognizer/gesture_recognizer/float16/latest/gesture_recognizer.task

      # Extract the top-level task
      unzip gesture_recognizer.task

      # Extract hand landmarker models
      unzip hand_landmarker.task
      # → hand_detector.tflite, hand_landmarks_detector.tflite

      # Extract gesture recognizer models
      unzip hand_gesture_recognizer.task
      # → gesture_embedder.tflite, canned_gesture_classifier.tflite
      ```

      <Note>These are FLOAT precision models.</Note>

      | File | Download | Save as |
      | - | - | - |
      | Palm detection model | See download steps above | `hand_detector.tflite` |
      | Palm detection labels | <a href="../labels/palmd_labels.json">palmd\_labels.json</a> | `palmd_labels.json` |
      | Palm detection settings | <a href="../labels/palmd_settings.json">palmd\_settings.json</a> | `palmd_settings.json` |
      | Hand landmark model | See download steps above | `hand_landmarks_detector.tflite` |
      | Hand landmark labels | <a href="../labels/hlandmark_labels.json">hlandmark\_labels.json</a> | `hlandmark_labels.json` |
      | Hand landmark settings | <a href="../labels/hlandmark_settings.json">hlandmark\_settings.json</a> | `hlandmark_settings.json` |
      | Gesture embedder model | See download steps above | `gesture_embedder.tflite` |
      | Gesture classifier model | See download steps above | `canned_gesture_classifier.tflite` |
      | Gesture labels | <a href="../labels/gesture_labels.json">gesture\_labels.json</a> | `gesture_labels.json` |
    </Step>

    <Step title="Copy files to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Run from your host machine — replace <user> and <device-ip>
        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels"
        scp hand_detector.tflite              <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp palmd_labels.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp palmd_settings.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp hand_landmarks_detector.tflite     <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp hlandmark_labels.json              <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp hlandmark_settings.json            <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp gesture_embedder.tflite            <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp canned_gesture_classifier.tflite   <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp gesture_labels.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        ```
      </CodeGroup>
    </Step>

    <Step title="Connect to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Run from your host machine — replace <user> and <device-ip>
        ssh <user>@<device-ip>
        ```
      </CodeGroup>
    </Step>

    <Step title="Set environment variables">
      Run below command on your device

      ```bash theme={null}
      mkdir -p $HOME/Downloads/qimsdk_samples/models $HOME/Downloads/qimsdk_samples/labels
      export MODEL_NAME_1=hand_detector.tflite
      export LABELS_NAME_1=palmd_labels.json
      export LABELS_NAME_2=palmd_settings.json
      export MODEL_NAME_2=hand_landmarks_detector.tflite
      export LABELS_NAME_3=hlandmark_labels.json
      export LABELS_NAME_4=hlandmark_settings.json
      export MODEL_NAME_3=gesture_embedder.tflite
      export MODEL_NAME_4=canned_gesture_classifier.tflite
      export LABELS_NAME_5=gesture_labels.json
      ```
    </Step>

    <Step title="Run the pipeline">
      ```bash theme={null}
      gst-launch-1.0 -e --gst-debug=2 \
        qtimlvconverter name=stage_01_preproc \
        qtimltflite name=stage_01_inference model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_1 delegate=gpu \
        qtimlpostprocess name=stage_01_postproc results=1 module=palmd \
        labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_1 settings=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_2 \
        qtimlvconverter name=stage_02_preproc mode=roi-batch-non-cumulative \
        qtimltflite name=stage_02_inference model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_2 delegate=gpu \
        qtimlpostprocess name=stage_02_1_postproc results=6 module=hlandmark \
        labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_3 settings=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_4 \
        qtimlpostprocess name=stage_02_2_postproc results=6 module=tensor \
        qtimltflite name=stage_03_1_inference model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_3 delegate=gpu \
        qtimltflite name=stage_03_2_inference model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_4 delegate=gpu \
        qtimlpostprocess name=stage_03_postproc results=8 module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_5 \
        qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! \
        tee name=t_split_1 \
        t_split_1. ! queue ! qtimetamux name=metamux_1 ! queue ! qtimetatransform module=roi-palmd ! \
        queue ! tee name=t_split_2 \
        t_split_1. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! stage_01_inference. \
        stage_01_inference. ! queue ! stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! metamux_1. \
        t_split_2. ! queue ! qtimetamux name=metamux_2 ! queue ! qtivoverlay ! waylandsink fullscreen=true sync=true \
        t_split_2. ! queue ! stage_02_preproc. stage_02_preproc. ! queue ! stage_02_inference. \
        stage_02_inference. ! queue ! tee name=t_split_3 \
        t_split_3. ! queue ! stage_02_1_postproc. stage_02_1_postproc. ! text/x-raw ! metamux_2. \
        t_split_3. ! queue ! stage_02_2_postproc. stage_02_2_postproc. ! queue ! \
        stage_03_1_inference. stage_03_1_inference. ! stage_03_2_inference. \
        stage_03_2_inference. ! stage_03_postproc. stage_03_postproc. ! text/x-raw ! metamux_2.
      ```
    </Step>

    <Step title="Expected Output">
      The pipeline detects hands, estimates keypoints, and recognizes gestures. Results are overlaid on each frame and rendered on the display.

      <img alt="Expected Output" />
    </Step>
  </Steps>

  ***

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | [qticamsrc](../plugin-reference/qticamsrc) | Captures live video from the ISP camera as the pipeline source. |
    | tee | Splits the stream for palm detection and downstream ROI-based stages. |
    | [qtimlvconverter](../plugin-reference/qtimlvconverter) | Preprocesses full frames (Stage 1) and ROI-cropped patches (Stage 2) for inference. |
    | [qtimltflite](../plugin-reference/qtimltflite) | Runs palm detection, hand landmark, gesture embedder, and gesture classifier inference sequentially. |
    | [qtimlpostprocess](../plugin-reference/qtimlpostprocess) | Post-processes each stage's tensors (palm ROIs, landmarks, gesture labels). |
    | qtimetatransform | Transforms ROI palm-detection metadata into cropped regions for the landmark stage. |
    | [qtimetamux](../plugin-reference/qtimetamux) | Merges video and metadata/text streams, attaching inference results as GST buffer metadata. |
    | [qtivoverlay](../plugin-reference/qtivoverlay) | Overlays inference results (labels, bounding boxes, keypoints) onto the video frame using CL. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders the final annotated video stream to a local display via Weston. |
  </Accordion>
</Accordion>

***

## Audio AI Pipelines

### Audio Classification (FLAC File Decode)

Classifies audio events from a video file containing a FLAC audio track using [YAMNet](https://aihub.qualcomm.com/iot/models/yamnet). The audio is decoded and processed in parallel with video playback, with classification results overlaid on the display.

**Pipeline Diagram**

<Frame>
  <img alt="Pipeline Diagram" />
</Frame>

***

<Accordion title="Try me">
  <Steps>
    <Step title="Download Required Files:">
      | File | Download | Save as |
      | - | - | - |
      | YAMNet model | [Qualcomm AI Hub — YAMNet](https://aihub.qualcomm.com/iot/models/yamnet) | `yamnet_float.tflite` |
      | Audio classification labels | <a href="../labels/yamnet.json">yamnet.json</a> | `yamnet.json` |
      | Sample video with FLAC audio | Use a H264 encoded mp4 container file containing FLAC audio track | `H264_720p_30fps_FLAC.mp4` |

      <Note>
        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
        `unzip filename.zip`
      </Note>
    </Step>

    <Step title="Copy files to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
        scp yamnet_float.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
        scp yamnet.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
        scp H264_720p_30fps_FLAC.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
        ```
      </CodeGroup>
    </Step>

    <Step title="Connect to device">
      <CodeGroup>
        ```bash SCP (SSH) theme={null}
        # Run from your host machine — replace <user> and <device-ip>
        ssh <user>@<device-ip>
        ```
      </CodeGroup>
    </Step>

    <Step title="Set environment variables">
      Run below command on your device

      ```bash theme={null}
      export MODEL_NAME=yamnet_float.tflite
      export LABELS_NAME=yamnet.json
      export SRC_VIDEO_NAME=H264_720p_30fps_FLAC.mp4
      ```
    </Step>

    <Step title="Run the pipeline">
      ```bash theme={null}
      gst-launch-1.0 -e filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux name=demux demux. ! queue ! h264parse ! \
      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw, format=NV12 ! qtivcomposer name=mixer sink_1::position="<50, 50>" sink_1::dimensions="<368, 64>" ! \
      queue ! waylandsink fullscreen=true demux. ! queue ! flacparse ! flacdec ! queue ! audioconvert ! audioresample ! \
      audiobuffersplit output-buffer-size=31200 ! queue ! qtimlaconverter  sample-rate=16000 feature=lmfe params="params,nfft=96,nhop=160,nmels=64,chunklen=0.96;" ! \
      queue ! qtimltflite name=infeng model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME ! qtimlpostprocess settings="{\"confidence\": 10.0}" results=3 module=yamnet \
      labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw,width=368,height=64 ! queue ! mixer.
      ```
    </Step>

    <Step title="Expected Output">
      Classification results are printed to the terminal. Each detected audio class with its confidence score is output per audio segment processed.
    </Step>
  </Steps>

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Reads an MP4 container file with H.264 video and FLAC audio as the source. |
    | qtdemux | Demultiplexes the container into separate H.264 video and FLAC audio elementary streams. |
    | h264parse | Parses the H.264 bitstream for downstream decoding. |
    | [v4l2h264dec](../plugin-reference/v4l2h264dec) | Hardware-decodes the H.264 stream to raw NV12 frames using V4L2. |
    | flacparse | Parses the FLAC audio bitstream from the demuxed stream. |
    | flacdec | Decodes the FLAC audio stream to raw PCM. |
    | audioconvert | Converts decoded PCM to the required sample format (S16LE). |
    | audioresample | Resamples the audio to the model's required sample rate. |
    | audiobuffersplit | Splits the audio into fixed-size buffers for frame-by-frame inference. |
    | qtimlaconverter | Converts raw PCM audio into the feature representation expected by the model. |
    | [qtimltflite](../plugin-reference/qtimltflite) | Loads the TFLite model, applies the chosen delegate, and runs inference to produce result tensors. |
    | qtimlaclassification | Post-processes audio inference tensors and produces classification label overlays. |
    | [qtivcomposer](../plugin-reference/qtivcomposer) | Overlays the audio classification result panel onto the video playback stream. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders the final composited video stream to a local display via Weston. |
  </Accordion>
</Accordion>


# Multimedia
Source: https://imsdkdocs.qualcomm.com/sample-pipelines/multimediapipelines



This section covers QIM SDK multimedia pipelines that use LiteRT for inference.

## 1. Camera

***

### 1.1 Single 1080p YUV stream from live source

**Description**
Captures a live feed from a camera device (e.g., USB via V4L2 or MIPI camera via qticamsrc) and displays it on the screen using Wayland.

**Pipeline Diagram**

This Pipeline diagram for USB camera

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  <CodeGroup>
    ```bash ISP camera theme={null}
    gst-launch-1.0 \
      qticamsrc name=camsrc ! \
      video/x-raw,width=1920,height=1080,framerate=30/1 ! \
      waylandsink fullscreen=true
    ```

    ```bash USB camera theme={null}
    gst-launch-1.0 \
      v4l2src device=/dev/video0 ! \
      video/x-raw,format=YUY2 ! \
      qtivtransform ! video/x-raw,format=NV12 ! \
      waylandsink fullscreen=true
    ```

    ```bash RTSP camera theme={null}
    gst-launch-1.0 \
      rtspsrc location={file_path} latency=200 ! rtph264depay ! h264parse ! \
      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! \
      waylandsink fullscreen=true
    ```
  </CodeGroup>

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | [qticamsrc](../plugin-reference/qticamsrc) | ISP camera source plugin for capturing raw video frames from MIPI camera. |
    | [v4l2src](../plugin-reference/v4l2src) | V4L2-based video capture source for USB cameras. |
    | [videoconvert](../plugin-reference/videoconvert) | Converts video between different color formats. |
    | [qtivtransform](../plugin-reference/qtivtransform) | Hardware-accelerated video transformation (format conversion, scaling). |
    | [waylandsink](../plugin-reference/waylandsink) | Renders video frames to a Wayland display surface. |
  </Accordion>
</Accordion>

<Note>
  In case of an **Internal data stream error** for USB camera, ensure that the USB camera device node or for RTSP camera RTSP stream location are correctly configured.
</Note>

***

### 1.2 Three 1080p YUV streams from live source

**Description**
Captures three 1080p streams from a single multi-stream capable camera source (`qticamsrc`) and displays them at different positions on the same screen.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash theme={null}
  gst-launch-1.0 -e -v \
    qtivcomposer name=mixer \
      sink_0::position="<0,0>"  sink_0::dimensions="<480,270>" \
      sink_1::position="<480,0>"  sink_1::dimensions="<480,270>" \
      sink_2::position="<960,0>"  sink_2::dimensions="<480,270>" \
    mixer. ! queue ! waylandsink fullscreen=true \
    qticamsrc name=camsrc camera=0 video_0::type=preview video_1::type=video \
    camsrc. ! queue ! video/x-raw,format=NV12_Q08C,width=1920,height=1080,framerate=30/1 \
             ! qtivtransform ! video/x-raw,width=480,height=270 \
             ! mixer.sink_0 \
    camsrc. ! queue ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 \
             ! qtivtransform ! video/x-raw,width=480,height=270 \
             ! mixer.sink_1 \
    camsrc. ! queue ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 \
             ! qtivtransform ! video/x-raw,width=480,height=270 \
             ! mixer.sink_2
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | [qticamsrc](../plugin-reference/qticamsrc) | ISP camera source plugin supporting multiple simultaneous output streams. |
    | [qtivcomposer](../plugin-reference/qtivcomposer) | Hardware video compositor that blends multiple input streams into one output. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders video frames to a Wayland display surface. |
  </Accordion>
</Accordion>

***

## 2. Camera and Video Encode

### 2.1  One Stream — 1080p AVC Video Record

**Description**
This pipeline captures a single 1080p live camera stream, encodes it to H.264 (AVC) using hardware acceleration, muxes it into an MP4 container, and saves it to the device filesystem.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  gst-launch-1.0 -e \
    qticamsrc name=camsrc camera=0 video_0::type=preview video_1::type=video ! \
    video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
    queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! \
    queue ! h264parse ! mp4mux ! queue ! \
    filesink location=$HOME/Downloads/qimsdk_samples/media/mux.mp4
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | [qticamsrc](../plugin-reference/qticamsrc) | ISP camera source plugin for capturing raw video frames. |
    | [v4l2h264enc](../plugin-reference/v4l2h264enc) | Hardware-accelerated H.264 video encoder. |
    | h264parse | Parses H.264 bitstream and adds framing metadata. |
    | mp4mux | Muxes encoded streams into an MP4 container. |
    | filesink | Writes pipeline data to a file on the filesystem. |
  </Accordion>
</Accordion>

Pull Recorded File to Host PC

```bash theme={null}
# Using SCP
scp -r root@[DEVICE IP-ADDR]:~/mux.mp4 .
```

***

### 2.2  One Stream - 1080p AVC RTSP from live source

**Description**
Captures 1080p video, encodes it to H.264, and streams it over RTSP using `gst-rtsp-server` (via UDP sink).

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash theme={null}
  ## 1. Start RTSP Server (in background or separate terminal)
  gst-rtsp-server -p 8900 -m /live "( udpsrc name=pay0 port=8554 caps=\"application/x-rtp,media=video,clock-rate=90000,encoding-name=H264,payload=96\" )" &

  ## 2. Run Pipeline
  gst-launch-1.0 -e \
    qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
    v4l2h264enc capture-io-mode=4 output-io-mode=5 ! \
    h264parse config-interval=-1 ! rtph264pay pt=96 ! \
    udpsink host=127.0.0.1 port=8554
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | [qticamsrc](../plugin-reference/qticamsrc) | ISP camera source plugin for capturing raw video frames. |
    | [v4l2h264enc](../plugin-reference/v4l2h264enc) | Hardware-accelerated H.264 video encoder. |
    | h264parse | Parses H.264 bitstream and adds framing metadata. |
    | rtph264pay | Payloads H.264 video into RTP packets. |
    | udpsink | Sends RTP packets over UDP for RTSP streaming. |
  </Accordion>
</Accordion>

***

### 2.3 Two streams - 4K AVC and 480p AVC from live source

**Description**
Simultaneously captures and records two streams: one 4K and one 480p, both encoded in H.264 and saved as MP4.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  gst-launch-1.0 -e --gst-debug=2 \
    qticamsrc name=camsrc camera=0 video_0::type=preview video_1::type=video \
    camsrc. ! video/x-raw,format=NV12,width=3840,height=2160,framerate=30/1 ! \
        queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! \
        queue ! h264parse ! mp4mux ! queue ! filesink location=$HOME/Downloads/qimsdk_samples/media/video1.mp4 \
    camsrc. ! video/x-raw,format=NV12,width=640,height=480,framerate=30/1 ! \
        queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! \
        queue ! h264parse ! mp4mux ! queue ! filesink location=$HOME/Downloads/qimsdk_samples/media/video2.mp4
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | [qticamsrc](../plugin-reference/qticamsrc) | ISP camera source supporting multiple simultaneous output streams. |
    | [v4l2h264enc](../plugin-reference/v4l2h264enc) | Hardware-accelerated H.264 video encoder. |
    | h264parse | Parses H.264 bitstream and adds framing metadata. |
    | mp4mux | Muxes encoded streams into an MP4 container. |
    | filesink | Writes pipeline data to a file on the filesystem. |
  </Accordion>
</Accordion>

***

### 2.4 Three 1080p AVC streams from live source

**Description**
Records 1080p AVC (H.264), 1080p HEVC (H.265), and displays a live 1080p preview simultaneously.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  gst-launch-1.0 -e --gst-debug=2 \
    qticamsrc name=camsrc camera=0 video_0::type=preview video_1::type=video \
    camsrc. ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
        queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! \
        queue ! h264parse ! mp4mux ! queue ! filesink location=$HOME/Downloads/qimsdk_samples/media/video2.mp4 \
    camsrc. ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
        queue ! v4l2h265enc capture-io-mode=4 output-io-mode=5 ! \
        queue ! h265parse ! mp4mux ! queue ! filesink location=$HOME/Downloads/qimsdk_samples/media/video2_hevc.mp4 \
    camsrc. ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
        waylandsink sync=true fullscreen=true enable-last-sample=false
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | [qticamsrc](../plugin-reference/qticamsrc) | ISP camera source supporting multiple simultaneous output streams. |
    | [v4l2h264enc](../plugin-reference/v4l2h264enc) | Hardware-accelerated H.264 video encoder. |
    | h264parse | Parses H.264 bitstream and adds framing metadata. |
    | [v4l2h265enc](../plugin-reference/v4l2h265enc) | Hardware-accelerated H.265/HEVC video encoder. |
    | h265parse | Parses H.265 bitstream and adds framing metadata. |
    | mp4mux | Muxes encoded streams into an MP4 container. |
    | filesink | Writes pipeline data to a file on the filesystem. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders video frames to a Wayland display surface. |
  </Accordion>
</Accordion>

***

### 2.5 Three stream - 4K JPEG Snapshot, 1080p AVC MP4, 1080p YUV from live source

**Description**
Captures 4K JPEG snapshots, records 1080p video, and displays a preview simultaneously.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  gst-launch-1.0 -e --gst-debug=2 \
    qticamsrc name=camsrc camera=0 video_0::type=preview video_1::type=video \
    camsrc. ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
        queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 ! \
        queue ! h264parse ! mp4mux ! queue ! filesink location=$HOME/Downloads/qimsdk_samples/media/video2.mp4 \
    camsrc.image_1 ! "image/jpeg,width=3840,height=2160,framerate=30/1" ! \
        multifilesink location=$HOME/Downloads/qimsdk_samples/media/frame%d.jpg sync=true async=false \
    camsrc. ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
        waylandsink fullscreen=true async=true sync=true
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | [qticamsrc](../plugin-reference/qticamsrc) | ISP camera source supporting video, preview, and image output streams. |
    | [v4l2h264enc](../plugin-reference/v4l2h264enc) | Hardware-accelerated H.264 video encoder. |
    | h264parse | Parses H.264 bitstream and adds framing metadata. |
    | mp4mux | Muxes encoded streams into an MP4 container. |
    | filesink | Writes pipeline data to a file on the filesystem. |
    | multifilesink | Writes a sequence of JPEG snapshot files to disk. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders video frames to a Wayland display surface. |
  </Accordion>
</Accordion>

***

## 3. Multi Camera/Multi client use cases

### 3.1 Two streams: Main camera: 4K AVC MP4, 1080p YUV Preview and Secondary Camera: 720p AVC, 720p YUV

**Description**
Uses two camera sensors. Main camera provides 4K recording and 1080p preview. Secondary camera provides 720p recording and 720p preview.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  gst-launch-1.0 -e \
    qticamsrc name=camsrc_0 camera=0 video_0::type=video video_1::type=preview \
    camsrc_0. ! video/x-raw,format=NV12,width=3840,height=2160,framerate=30/1 \
        ! queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 \
        ! queue ! h264parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/main_4k.mp4 \
    camsrc_0. ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 \
        ! queue ! waylandsink sync=true \
    qticamsrc name=camsrc_1 camera=1 video_0::type=video video_1::type=preview \
    camsrc_1. ! video/x-raw,format=NV12,width=1280,height=720,framerate=30/1 \
        ! queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 \
        ! queue ! h264parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/secondary_720p.mp4 \
    camsrc_1. ! video/x-raw,format=NV12,width=1280,height=720,framerate=30/1 \
        ! queue ! filesink location=$HOME/Downloads/qimsdk_samples/media/secondary_720p.yuv
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | [qticamsrc](../plugin-reference/qticamsrc) | ISP camera source plugin; one instance per camera sensor. |
    | [v4l2h264enc](../plugin-reference/v4l2h264enc) | Hardware-accelerated H.264 video encoder. |
    | h264parse | Parses H.264 bitstream and adds framing metadata. |
    | mp4mux | Muxes encoded streams into an MP4 container. |
    | filesink | Writes pipeline data to a file on the filesystem. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders video frames to a Wayland display surface. |
  </Accordion>
</Accordion>

***

### 3.2 Two streams - both 720p - one from each camera with Side-By-Side stitching sent to display

**Description**
Stitches two 720p streams (one from each camera) into a single side-by-side frame using [`qtivcomposer`](../plugin-reference/qtivcomposer). The result is recorded and streamed via RTSP.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  gst-launch-1.0 -e -v \
    qtivcomposer name=mixer \
      sink_0::position="<0,0>" sink_0::dimensions="<1280,720>" \
      sink_1::position="<1280,0>" sink_1::dimensions="<1280,720>" \
    mixer. ! queue ! tee name=t_split \
      t_split. ! queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 \
        ! queue ! h264parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/sidebyside.mp4 \
      t_split. ! queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 \
        ! queue ! h264parse config-interval=-1 ! rtph264pay pt=96 \
        ! udpsink host=127.0.0.1 port=8554 \
    qticamsrc name=camsrc_0 camera=0 \
      ! video/x-raw,format=NV12,width=1280,height=720,framerate=30/1 \
      ! queue ! mixer.sink_0 \
    qticamsrc name=camsrc_1 camera=1 \
      ! video/x-raw,format=NV12,width=1280,height=720,framerate=30/1 \
      ! queue ! mixer.sink_1
  ```
</Accordion>

***

### 3.3 Two streams - both 720p - one from each camera with Picture in Picture composition and sent to display

**Description**
Composes two streams (Picture-in-Picture) using [`qtivcomposer`](../plugin-reference/qtivcomposer). The result is recorded and streamed via RTSP.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  gst-launch-1.0 -e -v \
    qtivcomposer name=mixer \
      sink_0::position="<0,0>" sink_0::dimensions="<1280,720>" \
      sink_1::position="<590,310>" sink_1::dimensions="<640,360>" \
    mixer. ! queue ! tee name=t_split \
      t_split. ! queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 \
        ! queue ! h264parse ! mp4mux ! filesink location=$HOME/Downloads/qimsdk_samples/media/pip.mp4 \
      t_split. ! queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 \
        ! queue ! h264parse config-interval=-1 ! rtph264pay pt=96 \
        ! udpsink host=127.0.0.1 port=8554 \
    qticamsrc name=camsrc_0 camera=0 \
      ! video/x-raw,format=NV12,width=1280,height=720,framerate=30/1 \
      ! queue ! mixer.sink_0 \
    qticamsrc name=camsrc_1 camera=1 \
      ! video/x-raw,format=NV12,width=1280,height=720,framerate=30/1 \
      ! queue ! mixer.sink_1

  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | [qticamsrc](../plugin-reference/qticamsrc) | ISP camera source plugin; one instance per camera sensor. |
    | [qtivcomposer](../plugin-reference/qtivcomposer) | Hardware video compositor composing Picture-in-Picture layout. |
    | [v4l2h264enc](../plugin-reference/v4l2h264enc) | Hardware-accelerated H.264 video encoder. |
    | h264parse | Parses H.264 bitstream and adds framing metadata. |
    | mp4mux | Muxes encoded streams into an MP4 container. |
    | filesink | Writes pipeline data to a file on the filesystem. |
    | rtph264pay | Payloads H.264 video into RTP packets. |
    | udpsink | Sends RTP packets over UDP for RTSP streaming. |
  </Accordion>
</Accordion>

***

## 4. Transform and Transcode use-cases

### 4.1 Rotate, Flip, and Scale

**Description**
Demonstrates hardware-accelerated transformations (Rotation, Flip, Downscale) using [`qtivtransform`](../plugin-reference/qtivtransform).

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  <CodeGroup>
    ```bash Rotate 180° theme={null}
    gst-launch-1.0 \
      qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080 ! \
      qtivtransform rotate=180 ! \
      waylandsink fullscreen=true
    ```

    ```bash Downscale 4K to 1080p + Flip theme={null}
    gst-launch-1.0 \
      qticamsrc ! video/x-raw,format=NV12,width=3840,height=2160 ! \
      qtivtransform flip-horizontal=true ! \
      video/x-raw,width=1920,height=1080 ! \
      waylandsink fullscreen=true
    ```
  </CodeGroup>

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | [qticamsrc](../plugin-reference/qticamsrc) | ISP camera source plugin for capturing raw video frames. |
    | [qtivtransform](../plugin-reference/qtivtransform) | Hardware-accelerated video transformation: rotation, flip, and scaling. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders video frames to a Wayland display surface. |
  </Accordion>
</Accordion>

## 5. Video Playback use cases

***

### 5.1 Single stream video playback

**Description**
The pipeline demonstrates playback of Video stream from a file of media container format like mp4.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  gst-launch-1.0 -e -v \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! \
    qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! waylandsink
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Reads a media file from the local filesystem. |
    | qtdemux | Demultiplexes MP4/QuickTime container into audio and video streams. |
    | h264parse | Parses H.264 bitstream and adds framing metadata. |
    | [v4l2h264dec](../plugin-reference/v4l2h264dec) | Hardware-accelerated H.264 video decoder. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders video frames to a Wayland display surface. |
  </Accordion>
</Accordion>

***

### 5.2 Two 1080p streams video playback simultaneously

**Description**
Plays two 1080p video files simultaneously to the screen.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  # Run in separate consoles or combine with &
  gst-launch-1.0 -e -v \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! \
    qtdemux ! queue ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! \
    video/x-raw,format=NV12 ! queue ! waylandsink &

  gst-launch-1.0 -e -v \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! \
    qtdemux ! queue ! h264parse ! \
    v4l2h264dec capture-io-mode=4 output-io-mode=4 ! \
    video/x-raw,format=NV12 ! queue ! waylandsink
  ``
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Reads a media file from the local filesystem. |
    | qtdemux | Demultiplexes MP4/QuickTime container into audio and video streams. |
    | h264parse | Parses H.264 bitstream and adds framing metadata. |
    | [v4l2h264dec](../plugin-reference/v4l2h264dec) | Hardware-accelerated H.264 video decoder. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders video frames to a Wayland display surface. |
  </Accordion>
</Accordion>

***

### 5.3 8-Stream Grid Playback

**Description**
Decodes 8 1080p video files simultaneously and composes them into a 4x2 grid on the display.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  gst-launch-1.0 -e --gst-debug=3 \
    qtivcomposer name=comp \
      sink_0::position="<0, 0>" sink_0::dimensions="<480, 540>" \
      sink_1::position="<480, 0>" sink_1::dimensions="<480, 540>" \
      sink_2::position="<960, 0>" sink_2::dimensions="<480, 540>" \
      sink_3::position="<1440, 0>" sink_3::dimensions="<480, 540>" \
      sink_4::position="<0, 540>" sink_4::dimensions="<480, 540>" \
      sink_5::position="<480, 540>" sink_5::dimensions="<480, 540>" \
      sink_6::position="<960, 540>" sink_6::dimensions="<480, 540>" \
      sink_7::position="<1440, 540>" sink_7::dimensions="<480, 540>" ! \
    queue ! waylandsink fullscreen=true \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_0 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_1 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_3 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_4 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_5 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_6 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_7 
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Reads a media file from the local filesystem. |
    | qtdemux | Demultiplexes MP4/QuickTime container into audio and video streams. |
    | h264parse | Parses H.264 bitstream and adds framing metadata. |
    | [v4l2h264dec](../plugin-reference/v4l2h264dec) | Hardware-accelerated H.264 video decoder. |
    | [qtivcomposer](../plugin-reference/qtivcomposer) | Hardware video compositor arranging decoded streams in a 4×2 grid. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders composed video frames to a Wayland display surface. |
  </Accordion>
</Accordion>

### 5.4 16-Stream Grid Playback

**Description**
Decodes 16 1080p video files simultaneously and composes them into a 4x4 grid on the display.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  gst-launch-1.0 -e --gst-debug=3 \
    qtivcomposer name=comp \
      sink_0::position="<0, 0>" sink_0::dimensions="<480, 270>" \
      sink_1::position="<480, 0>" sink_1::dimensions="<480, 270>" \
      sink_2::position="<960, 0>" sink_2::dimensions="<480, 270>" \
      sink_3::position="<1440, 0>" sink_3::dimensions="<480, 270>" \
      sink_4::position="<0, 270>" sink_4::dimensions="<480, 270>" \
      sink_5::position="<480, 270>" sink_5::dimensions="<480, 270>" \
      sink_6::position="<960, 270>" sink_6::dimensions="<480, 270>" \
      sink_7::position="<1440, 270>" sink_7::dimensions="<480, 270>" \
      sink_8::position="<0, 540>" sink_8::dimensions="<480, 270>" \
      sink_9::position="<480, 540>" sink_9::dimensions="<480, 270>" \
      sink_10::position="<960, 540>" sink_10::dimensions="<480, 270>" \
      sink_11::position="<1440, 540>" sink_11::dimensions="<480, 270>" \
      sink_12::position="<0, 810>" sink_12::dimensions="<480, 270>" \
      sink_13::position="<480, 810>" sink_13::dimensions="<480, 270>" \
      sink_14::position="<960, 810>" sink_14::dimensions="<480, 270>" \
      sink_15::position="<1440, 810>" sink_15::dimensions="<480, 270>" ! \
    queue ! waylandsink fullscreen=true \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_0 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_1 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_2 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_3 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_4 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_5 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_6 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_7 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_8 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_9 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_10 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_11 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_12 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_13 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_14 \
    filesrc location=$HOME/Downloads/qimsdk_samples/media/video.mp4 ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! comp.sink_15
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Reads a media file from the local filesystem. |
    | qtdemux | Demultiplexes MP4/QuickTime container into audio and video streams. |
    | h264parse | Parses H.264 bitstream and adds framing metadata. |
    | [v4l2h264dec](../plugin-reference/v4l2h264dec) | Hardware-accelerated H.264 video decoder. |
    | [qtivcomposer](../plugin-reference/qtivcomposer) | Hardware video compositor arranging decoded streams in a 4×4 grid. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders composed video frames to a Wayland display surface. |
  </Accordion>
</Accordion>

<Note>
  16 hardware decode is supported only on IQ‑9075.
</Note>

## 6. Audio use cases

***

<Note>
  **Prerequisites for Audio Use Cases:**

  Before running audio pipelines, ensure the correct audio source is configured:

  ```bash theme={null}
  # Check available audio sources and their node numbers
  wpctl status

  # Set the default audio source (replace <node no.> with the actual source node number)
  wpctl set-default <node no.>
  ```
</Note>

### 6.1 PCM Audio Capture

**Description**
Captures audio from the microphone via PulseAudio, converts it, encodes it to WAV, and saves it to a file.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  gst-launch-1.0 -v pulsesrc volume=10 ! audioconvert ! wavenc ! filesink location=$HOME/Downloads/qimsdk_samples/media/<Audio_PCM>.wav
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | pulsesrc | Captures audio from a PulseAudio input device (microphone). |
    | audioconvert | Converts audio between different formats/rates. |
    | wavenc | Encodes raw PCM audio into WAV format. |
    | filesink | Writes pipeline data to a file on the filesystem. |
  </Accordion>
</Accordion>

***

### 6.2 PCM Audio Playback

**Description**
A basic audio playback pipeline that reads a WAV file and routes the PCM stream to the PulseAudio sink.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  gst-launch-1.0 -e filesrc location=$HOME/Downloads/qimsdk_samples/media/<wav_file>.wav ! wavparse ! audioconvert ! pulsesink volume=10
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Reads a WAV file from the local filesystem. |
    | wavparse | Parses WAV container and extracts PCM audio stream. |
    | audioconvert | Converts audio between different formats/rates. |
    | pulsesink | Plays audio through a PulseAudio output device (speaker). |
  </Accordion>
</Accordion>

***

### 6.3 MP3 Audio Playback (Software Decode)

**Description**
Reads an MP3 file and decodes it using `mpg123audiodec` to PulseAudio.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  gst-launch-1.0 -e filesrc location=$HOME/Downloads/qimsdk_samples/media/<mp3_file>.mp3 ! mpegaudioparse ! mpg123audiodec ! pulsesink volume=10
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Reads an MP3 file from the local filesystem. |
    | mpegaudioparse | Parses MPEG audio bitstream framing. |
    | mpg123audiodec | Software MP3 audio decoder. |
    | pulsesink | Plays audio through a PulseAudio output device (speaker). |
  </Accordion>
</Accordion>

***

### 6.4 Record 1080p Video with Audio (AVC/MP3 Capture)

**Description**
Captures 1080p video from the camera and audio from the microphone.

* **Video:** Hardware encoded (H.264).
* **Audio:** Software encoded (MP3) using `lamemp3enc`.
* **Muxing:** Muxed into an MP4 container.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  gst-launch-1.0 -e \
  pulsesrc do-timestamp=true provide-clock=false volume=10 ! audio/x-raw,format=S16LE,channels=1,rate=48000 ! audioconvert ! queue ! lamemp3enc ! queue ! mpegaudioparse ! queue ! mp4mux name=muxer ! queue ! filesink location=$HOME/Downloads/qimsdk_samples/media/1080p_AVC_MP3.mp4 \
  qticamsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1,interlace-mode=progressive,colorimetry=bt601 ! queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 extra-controls="controls,video_bitrate=1000000,video_gop_size=29;" ! queue ! h264parse ! muxer.
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | [qticamsrc](../plugin-reference/qticamsrc) | ISP camera source plugin for capturing raw 1080p video. |
    | [v4l2h264enc](../plugin-reference/v4l2h264enc) | Hardware-accelerated H.264 video encoder. |
    | h264parse | Parses H.264 bitstream and adds framing metadata. |
    | pulsesrc | Captures audio from a PulseAudio input device (microphone). |
    | audioconvert | Converts audio between different formats/rates. |
    | lamemp3enc | Software MP3 audio encoder. |
    | aacparse | Parses AAC/MP3 audio bitstream for muxing. |
    | mp4mux | Muxes video and audio streams into an MP4 container. |
    | filesink | Writes the muxed AV file to the filesystem. |
  </Accordion>
</Accordion>

***

### 6.5 1080p AVC/MP3 Video Playback

**Description**
Reads an MP4 file and demuxes it into separate audio and video streams.

* **Video:** H.264 decoded via hardware and rendered to display.
* **Audio:** MP3 decoded via software (`mpg123audiodec`) and sent to speaker.

**Pipeline Diagram**

<img alt="Pipeline Diagram" />

**Pipeline Command**

<Accordion title="Try me">
  ```bash export command theme={null}
  mkdir -p $HOME/Downloads/qimsdk_samples/media $HOME/Downloads/qimsdk_samples/models
  ```

  ```bash pipeline command theme={null}
  gst-launch-1.0 -e filesrc location=$HOME/Downloads/qimsdk_samples/media/<AV_file_with_H264_video_and_MP3_audio>.mp4 ! qtdemux name=demux demux. ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! waylandsink fullscreen=true demux. ! queue ! mpegaudioparse ! mpg123audiodec ! pulsesink volume=10
  ```

  <Accordion title="Plugins used in Pipeline">
    | Plugin | Description |
    | - | - |
    | filesrc | Reads an MP4 file from the local filesystem. |
    | qtdemux | Demultiplexes MP4 container into video and audio streams. |
    | h264parse | Parses H.264 video bitstream. |
    | [v4l2h264dec](../plugin-reference/v4l2h264dec) | Hardware-accelerated H.264 video decoder. |
    | [waylandsink](../plugin-reference/waylandsink) | Renders decoded video to a Wayland display surface. |
    | mpegaudioparse | Parses MPEG audio bitstream framing. |
    | mpg123audiodec | Software MP3 audio decoder. |
    | pulsesink | Plays decoded audio through PulseAudio speaker output. |
  </Accordion>
</Accordion>

asdfsadfhs


# Audio Classification
Source: https://imsdkdocs.qualcomm.com/supportedmodels/audio-classification

QIM SDK supported models for Audio Classification

***

<Tabs>
  <Tab title="TFLite Runtime">
    <div>
      <table>
        <colgroup>
          <col />

          <col />

          <col />
        </colgroup>

        <thead>
          <tr>
            <th>No.</th>
            <th>Model Name</th>
            <th>Description</th>
          </tr>
        </thead>

        <tbody>
          <tr>
            <td>1</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/yamnet">YAMNET (FLAC File Decode)</a></td>
            <td>Classifies audio events from a video file containing a FLAC audio track, decoded and processed in parallel with video playback.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/yamnet) | yamnet\_float.tflite |
                      | <a href="../labels/yamnet.json">yamnet.json</a> | yamnet.json |
                      | Sample video with FLAC audio (H.264 video + FLAC audio track in MP4 container) | H264\_720p\_30fps\_FLAC.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp yamnet_float.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp yamnet.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp H264_720p_30fps_FLAC.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=yamnet_float.tflite
                      export LABELS_NAME=yamnet.json
                      export SRC_VIDEO_NAME=H264_720p_30fps_FLAC.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux name=demux demux. ! queue ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw, format=NV12 ! qtivcomposer name=mixer sink_1::position="<50, 50>" sink_1::dimensions="<368, 64>" ! \
                      queue ! waylandsink fullscreen=true demux. ! queue ! flacparse ! flacdec ! queue ! audioconvert ! audioresample ! \
                      audiobuffersplit output-buffer-size=31200 ! queue ! qtimlaconverter  sample-rate=16000 feature=lmfe params="params,nfft=96,nhop=160,nmels=64,chunklen=0.96;" ! \
                      queue ! qtimltflite name=infeng model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME ! qtimlpostprocess settings="{\"confidence\": 10.0}" results=3 module=yamnet \
                      labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw,width=368,height=64 ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>
        </tbody>
      </table>
    </div>
  </Tab>
</Tabs>


# Depth Estimation
Source: https://imsdkdocs.qualcomm.com/supportedmodels/depth-estimation

QIM SDK supported models for Depth Estimation

***

<Tabs>
  <Tab title="TFLite Runtime">
    <div>
      <table>
        <colgroup>
          <col />

          <col />

          <col />
        </colgroup>

        <thead>
          <tr>
            <th>No.</th>
            <th>Model Name</th>
            <th>Description</th>
          </tr>
        </thead>

        <tbody>
          <tr>
            <td>1</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/depth_anything">Depth-Anything</a></td>
            <td>Foundation model for monocular depth estimation on diverse scenes.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/depth_anything) | depth\_anything\_float.tflite |
                      | <a href="../labels/monodepth.json">monodepth.json</a> | monodepth.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp depth_anything_float.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp monodepth.json               <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=depth_anything_float.tflite
                      export LABELS_NAME=monodepth.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -v --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! queue ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! tee name=split \
                      split. ! queue ! qtivcomposer name=mixer sink_1::alpha=0.5 ! video/x-raw,format=NV12 ! queue ! waylandsink sync=true fullscreen=true \
                      split. ! queue ! videoscale ! videoconvert ! qtimlvconverter ! queue ! \
                      qtimltflite delegate=gpu model=$HOME/Downloads/qimsdk_samples/models/depth_anything_float.tflite ! queue ! \
                      qtimlpostprocess module=midas-v2 labels=$HOME/Downloads/qimsdk_samples/labels/monodepth.json ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>2</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/depth_anything_v2">Depth-Anything-V2</a></td>
            <td>Improved foundation depth model with finer detail and accuracy.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/depth_anything_v2) | depth\_anything\_v2\_float.tflite |
                      | <a href="../labels/monodepth.json">monodepth.json</a> | monodepth.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp depth_anything_v2_float.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp monodepth.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4       <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=depth_anything_v2_float.tflite
                      export LABELS_NAME=monodepth.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -v --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! queue ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! tee name=split \
                      split. ! queue ! qtivcomposer name=mixer sink_1::alpha=0.5 ! video/x-raw,format=NV12 ! queue ! waylandsink sync=true fullscreen=true \
                      split. ! queue ! videoscale ! videoconvert ! qtimlvconverter ! queue ! \
                      qtimltflite delegate=gpu model=$HOME/Downloads/qimsdk_samples/models/depth_anything_float.tflite ! queue ! \
                      qtimlpostprocess module=midas-v2 labels=$HOME/Downloads/qimsdk_samples/labels/monodepth.json ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>
        </tbody>
      </table>
    </div>
  </Tab>
</Tabs>


# Image Classification
Source: https://imsdkdocs.qualcomm.com/supportedmodels/image-classification

QIM SDK supported models for Image Classification

***

<Tabs>
  <Tab title="TFLite Runtime">
    <div>
      <table>
        <colgroup>
          <col />

          <col />

          <col />
        </colgroup>

        <thead>
          <tr>
            <th>No.</th>
            <th>Model Name</th>
            <th>Description</th>
          </tr>
        </thead>

        <tbody>
          <tr>
            <td>1</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/resnext101?searchTerm=resnex">ResNeXt101</a></td>
            <td>Wide residual network with grouped convolutions for high accuracy.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/resnext101?searchTerm=resnex) | resnext101\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp resnext101_float.tflite             <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=resnext101_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 31.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/resnext101?searchTerm=resnex) | resnext101\_w8a8.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp resnext101_w8a8.tflite                <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                        <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4      <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=resnext101_w8a8.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 35.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>2</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/vit?searchTerm=vi">VIT</a></td>
            <td>Vision Transformer applying self-attention on image patches.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/vit?searchTerm=vi) | vit\_w8a8.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp vit_w8a8.tflite                    <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=vit_w8a8.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 35.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>3</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/efficientvit_b2_cls?searchTerm=EfficientViT-b2-cls">EfficientViT-b2-cls</a></td>
            <td>Memory-efficient vision transformer (B2 classification).</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/efficientvit_b2_cls?searchTerm=EfficientViT-b2-cls) | efficientvit\_b2\_cls\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp efficientvit_b2_cls_float.tflite   <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=efficientvit_b2_cls_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>4</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/efficientvit_l2_cls?searchTerm=EfficientViT-l2-cls">EfficientViT-l2-cls</a></td>
            <td>Larger memory-efficient vision transformer (L2 classification).</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/efficientvit_l2_cls?searchTerm=EfficientViT-l2-cls) | efficientvit\_l2\_cls\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp efficientvit_l2_cls_float.tflite   <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=efficientvit_l2_cls_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>5</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/efficientnet_b0?searchTerm=EfficientNet-B0">EfficientNet-B0</a></td>
            <td>Balanced accuracy/efficiency via compound scaling (B0 base).</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/efficientnet_b0?searchTerm=EfficientNet-B0) | efficientnet\_b0\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp efficientnet_b0_float.tflite      <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                    <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=efficientnet_b0_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>6</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/convnext_base?searchTerm=ConvNext-Base">ConvNext-Base</a></td>
            <td>Pure CNN architecture inspired by ViT design principles (Base).</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/convnext_base?searchTerm=ConvNext-Base) | convnext\_base\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp convnext_base_float.tflite         <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=convnext_base_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>7</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/convnext_tiny?searchTerm=ConvNext-Tiny">ConvNext-Tiny</a></td>
            <td>Compact ConvNeXt variant for efficient classification.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/convnext_tiny?searchTerm=ConvNext-Tiny) | convnext\_tiny\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp convnext_tiny_float.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                    <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=convnext_tiny_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>8</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/efficientnet_v2_s?searchTerm=EfficientNet-V2-s">EfficientNet-V2-s</a></td>
            <td>Faster training EfficientNet V2 small variant.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/efficientnet_v2_s?searchTerm=EfficientNet-V2-s) | efficientnet\_v2\_s\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp efficientnet_v2_s_float.tflite     <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=efficientnet_v2_s_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>9</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/beit?searchTerm=Beit">Beit</a></td>
            <td>BERT pre-training for image transformers.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/beit?searchTerm=Beit) | beit\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp beit_float.tflite                  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=beit_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>10</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/mnasnet05?searchTerm=MNASNet05">MNASNet05</a></td>
            <td>Neural architecture search model for mobile efficiency.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/mnasnet05?searchTerm=MNASNet05) | mnasnet05\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp mnasnet05_float.tflite              <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=mnasnet05_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>11</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/densenet121?searchTerm=Dense&chipsets=qualcomm-qcs6490-proxy">DenseNet-121</a></td>
            <td>Densely connected CNN with feature reuse across layers.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/densenet121?searchTerm=Dense\&chipsets=qualcomm-qcs6490-proxy) | densenet\_121\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp densenet_121_float.tflite           <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=densenet_121_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>12</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/googlenet?searchTerm=googl">GoogLeNet</a></td>
            <td>Inception-based deep CNN for efficient classification.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/googlenet?searchTerm=googl) | googlenet\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp googlenet_float.tflite              <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=googlenet_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/googlenet?searchTerm=googl) | googlenet\_w8a8.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp googlenet_w8a8.tflite               <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=googlenet_w8a8.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>13</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/resnet101?searchTerm=Resnet101">Resnet101</a></td>
            <td>Deep residual network with 101 layers.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/resnet101?searchTerm=Resnet101) | resnet101\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp resnet101_float.tflite                 <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                         <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4       <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=resnet101_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/resnet101?searchTerm=Resnet101) | resnet101\_w8a8.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp resnet101_w8a8.tflite               <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=resnet101_w8a8.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>14</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/resnet50?searchTerm=Resnet50">Resnet50</a></td>
            <td>Classic 50-layer residual network for image classification.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/resnet50?searchTerm=Resnet50) | resnet50\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp resnet50_float.tflite               <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=resnet50_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/resnet50?searchTerm=Resnet50) | resnet50\_w8a8.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp resnet50_w8a8.tflite                <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=resnet50_w8a8.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>15</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/inception_v3?searchTerm=incep">Inception-v3</a></td>
            <td>Multi-scale feature extraction with factorized convolutions.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/inception_v3?searchTerm=incep) | inception\_v3\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp inception_v3_float.tflite           <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=inception_v3_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/inception_v3?searchTerm=incep) | inception\_v3\_w8a8.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp inception_v3_w8a8.tflite            <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=inception_v3_w8a8.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>16</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/mobilenet_v2">MobileNet-v2</a></td>
            <td>Inverted residual blocks for mobile image classification.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/mobilenet_v2) | mobilenet\_v2\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp mobilenet_v2_float.tflite           <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=mobilenet_v2_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/mobilenet_v2) | mobilenet\_v2\_w8a8.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp mobilenet_v2_w8a8.tflite            <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=mobilenet_v2_w8a8.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>17</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/mobilenet_v3_large">MobileNet-v3-Large</a></td>
            <td>Optimized mobile CNN with SE modules (Large variant).</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/mobilenet_v3_large) | mobilenet\_v3\_large\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp mobilenet_v3_large_float.tflite     <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=mobilenet_v3_large_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>18</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/regnet?searchTerm=reg&chipsets=qualcomm-qcs6490-proxy">RegNet</a></td>
            <td>Regularized network design with quantized widths and depths.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/regnet?searchTerm=reg\&chipsets=qualcomm-qcs6490-proxy) | regnet\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp regnet_float.tflite                 <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=regnet_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/regnet?searchTerm=reg\&chipsets=qualcomm-qcs6490-proxy) | regnet\_w8a8.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp regnet_w8a8.tflite                  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=regnet_w8a8.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>19</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/resnet18?searchTerm=resnet18&chipsets=qualcomm-qcs6490-proxy">ResNet18</a></td>
            <td>Lightweight 18-layer residual network.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/resnet18?searchTerm=resnet18\&chipsets=qualcomm-qcs6490-proxy) | resnet18\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp resnet18_float.tflite               <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=resnet18_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/resnet18?searchTerm=resnet18\&chipsets=qualcomm-qcs6490-proxy) | resnet18\_w8a8.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp resnet18_w8a8.tflite                <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=resnet18_w8a8.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>20</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/resnext50?searchTerm=resnex&chipsets=qualcomm-qcs6490-proxy">ResNeXt50</a></td>
            <td>Compact grouped-convolution ResNet variant.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/resnext50?searchTerm=resnex\&chipsets=qualcomm-qcs6490-proxy) | resnext50\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp resnext50_float.tflite              <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=resnext50_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/resnext50?searchTerm=resnex\&chipsets=qualcomm-qcs6490-proxy) | resnext50\_w8a8.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp resnext50_w8a8.tflite               <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=resnext50_w8a8.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>21</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/shufflenet_v2?searchTerm=shuf&chipsets=qualcomm-qcs6490-proxy">Shufflenet-v2</a></td>
            <td>Channel-split shuffle network for efficient mobile inference.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/shufflenet_v2?searchTerm=shuf\&chipsets=qualcomm-qcs6490-proxy) | shufflenet\_v2\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp shufflenet_v2_float.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=shufflenet_v2_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/shufflenet_v2?searchTerm=shuf\&chipsets=qualcomm-qcs6490-proxy) | shufflenet\_v2\_w8a8.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp shufflenet_v2_w8a8.tflite           <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=shufflenet_v2_w8a8.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>22</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/squeezenet1_1?searchTerm=squee&chipsets=qualcomm-qcs6490-proxy">SqueezeNet-1.1</a></td>
            <td>AlexNet-level accuracy at 50x fewer parameters.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/squeezenet1_1?searchTerm=squee\&chipsets=qualcomm-qcs6490-proxy) | squeezenet\_1\_1\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp squeezenet_1_1_float.tflite         <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=squeezenet_1_1_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/squeezenet1_1?searchTerm=squee\&chipsets=qualcomm-qcs6490-proxy) | squeezenet\_1\_1\_w8a8.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp squeezenet_1_1_w8a8.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=squeezenet_1_1_w8a8.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>23</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/wideresnet50?searchTerm=wide&chipsets=qualcomm-qcs6490-proxy">WideResNet50</a></td>
            <td>Wide residual network with increased channel width.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/wideresnet50?searchTerm=wide\&chipsets=qualcomm-qcs6490-proxy) | wideresnet50\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp wideresnet50_float.tflite           <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=wideresnet50_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/wideresnet50?searchTerm=wide\&chipsets=qualcomm-qcs6490-proxy) | wideresnet50\_w8a8.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp wideresnet50_w8a8.tflite            <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=wideresnet50_w8a8.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>24</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/efficientnet_b4">EfficientNet-B4</a></td>
            <td>Higher-capacity EfficientNet for improved accuracy.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/efficientnet_b4) | efficientnet\_b4\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp efficientnet_b4_float.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=efficientnet_b4_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>25</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/dla102x">DLA-102-X</a></td>
            <td>Deep layer aggregation for hierarchical feature fusion.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/dla102x) | dla\_102\_x\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp dla_102_x_float.tflite              <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=dla_102_x_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/dla102x) | dla\_102\_x\_w8a8.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp dla_102_x_w8a8.tflite               <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=dla_102_x_w8a8.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>26</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/mobilenet_v3_small">MobileNet-v3-Small</a></td>
            <td>Ultra-compact mobile CNN (Small variant).</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/mobilenet_v3_small) | mobilenet\_v3\_small\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp mobilenet_v3_small_float.tflite                 <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                                  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4                <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=mobilenet_v3_small_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>27</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/efficientformer">EfficientFormer</a></td>
            <td>Ultra-fast transformer-style ViT for edge deployment.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/efficientformer) | efficientformer\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp efficientformer_float.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=efficientformer_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>28</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/gpunet">GPUNet</a></td>
            <td>GPU-optimized network via neural architecture search.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/gpunet) | gpunet\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/artifacts/videos/ai_demo_sample.mp4">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp gpunet_float.tflite                     <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4        <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=gpunet_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>29</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/sequencer2d">Sequencer2D</a></td>
            <td>2D LSTM-based alternative to attention for image classification.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/sequencer2d) | sequencer2d\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp sequencer2d_float.tflite                    <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                              <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4            <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=sequencer2d_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>30</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/levit?searchTerm=LeViT">LeViT</a></td>
            <td>Hybrid convolutional-attention ViT optimized for fast inference.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/levit) | levit\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/artifacts/videos/ai_demo_sample.mp4">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp levit_float.tflite                 <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=levit_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>31</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/mobile_vit">Mobile-VIT</a></td>
            <td>Lightweight hybrid CNN-Transformer for mobile devices.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/mobile_vit) | mobile\_vit\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/artifacts/videos/ai_demo_sample.mp4">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp mobile_vit_float.tflite             <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=mobile_vit_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=mobilenet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>32</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/nasnet">NASNet</a></td>
            <td>Neural architecture search network for mobile devices.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/nasnet) | nasnet\_float.tflite |
                      | <a href="../labels/mobilenet.json">mobilenet.json</a> | mobilenet.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/artifacts/videos/ai_demo_sample.mp4">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp nasnet_float.tflite                   <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp mobilenet.json                        <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4      <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=nasnet_float.tflite
                      export LABELS_NAME=mobilenet.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=class_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=mobilenet-softmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 51.0}" ! text/x-raw ! queue ! class_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>
        </tbody>
      </table>
    </div>
  </Tab>
</Tabs>


# Object Detection
Source: https://imsdkdocs.qualcomm.com/supportedmodels/object-detection

QIM SDK supported models for Object Detection

***

<Tabs>
  <Tab title="TFLite Runtime">
    <div>
      <table>
        <colgroup>
          <col />

          <col />

          <col />
        </colgroup>

        <thead>
          <tr>
            <th>No.</th>
            <th>Model Name</th>
            <th>Description</th>
          </tr>
        </thead>

        <tbody>
          <tr>
            <td>1</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/yolov11_det">YOLOv11-Detection</a></td>
            <td>Latest YOLO generation with enhanced feature extraction.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/yolov11_det) | yolov11\_det\_float.tflite |
                      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp yolov11_det_float.tflite       <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp yolov8.json                    <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4      <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=yolov11_det_float.tflite
                      export LABELS_NAME=yolov8.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/yolov11_det) | yolov11\_det\_w8a8.tflite |
                      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp yolov11_det_w8a8.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp yolov8.json                    <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4      <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=yolov11_det_w8a8.tflite
                      export LABELS_NAME=yolov8.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>2</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/yolox?searchTerm=yo">Yolo-X</a></td>
            <td>Anchor-free YOLO variant with decoupled detection head.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/yolox?searchTerm=yo) | yolox\_w8a8.tflite |
                      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp yolox_w8a8.tflite              <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp yolov8.json                    <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4      <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=yolox_w8a8.tflite
                      export LABELS_NAME=yolov8.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>3</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/yolov10_detection">YOLOv10-Detection</a></td>
            <td>NMS-free dual-label assignment for end-to-end detection.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/yolov10_detection) | yolov10\_detection\_w8a8.tflite |
                      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp yolov10_detection_w8a8.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp yolov8.json                    <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4      <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=yolov10_detection_w8a8.tflite
                      export LABELS_NAME=yolov8.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/yolov10_detection) | yolov10\_detection\_float.tflite |
                      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp yolov10_detection_float.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp yolov8.json                     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4       <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=yolov10_detection_float.tflite
                      export LABELS_NAME=yolov8.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>4</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/yolov8_det">YOLOv8-Detection</a></td>
            <td>Streamlined single-stage detector with CSP bottleneck backbone.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_float.tflite |
                      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp yolov8_det_float.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp yolov8.json                    <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4      <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=yolov8_det_float.tflite
                      export LABELS_NAME=yolov8.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_w8a8.tflite |
                      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp yolov8_det_w8a8.tflite         <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp yolov8.json                    <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4      <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=yolov8_det_w8a8.tflite
                      export LABELS_NAME=yolov8.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>5</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/yolov7">Yolo-v7</a></td>
            <td>High-accuracy detection with re-parameterizable convolutions.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/yolov7) | yolov7\_float.tflite |
                      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp yolov7_float.tflite                 <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp yolov8.json                         <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4           <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=yolov7_float.tflite
                      export LABELS_NAME=yolov8.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/yolov7) | yolov7\_w8a8.tflite |
                      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp yolov7_w8a8.tflite             <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp yolov8.json                    <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4      <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=yolov7_w8a8.tflite
                      export LABELS_NAME=yolov8.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>6</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/yolov5">Yolo-V5</a></td>
            <td>Real-time object detection using anchor-based detection heads.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/yolov5) | yolov5\_float.tflite |
                      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp yolov5_float.tflite            <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp yolov8.json                    <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4      <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=yolov5_float.tflite
                      export LABELS_NAME=yolov8.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>7</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/foot_track_net?searchTerm=person">Person-Foot-Detection</a></td>
            <td>Specialized foot and person localization for tracking pipelines.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/foot_track_net?searchTerm=person) | person\_foot\_detection\_float.tflite |
                      | <a href="../labels/foot_track_net.json">foot\_track\_net.json</a> | foot\_track\_net.json |
                      | <a href="../labels/foot_track_net_settings.json">foot\_track\_net\_settings.json</a> | foot\_track\_net\_settings.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp person_foot_detection_float.tflite       <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp foot_track_net.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp foot_track_net_settings.json             <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4                            <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=person_foot_detection_float.tflite
                      export LABELS_NAME_1=foot_track_net.json
                      export LABELS_NAME_2=foot_track_net_settings.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=qpd labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_1 settings=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_2 ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/foot_track_net?searchTerm=person) | person\_foot\_detection\_w8a8.tflite |
                      | <a href="../labels/foot_track_net.json">foot\_track\_net.json</a> | foot\_track\_net.json |
                      | <a href="../labels/foot_track_net_settings.json">foot\_track\_net\_settings.json</a> | foot\_track\_net\_settings.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">ai\_demo\_sample.mp4</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp person_foot_detection_w8a8.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp foot_track_net.json                      <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp foot_track_net_settings.json             <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4                            <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=person_foot_detection_w8a8.tflite
                      export LABELS_NAME_1=foot_track_net.json
                      export LABELS_NAME_2=foot_track_net_settings.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=qpd labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_1 settings=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_2 ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>8</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/detr_resnet101?searchTerm=detr">DETR-ResNet101</a></td>
            <td>Transformer-based detector with deeper ResNet-101 backbone.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/detr_resnet101) | detr\_resnet101\_float.tflite |
                      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp detr_resnet101_float.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp yolov8.json                  <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4    <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=detr_resnet101_float.tflite
                      export LABELS_NAME=yolov8.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>9</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/detr_resnet50?searchTerm=detr">DETR-ResNet50</a></td>
            <td>Transformer-based detector with ResNet-50 backbone.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/detr_resnet50) | detr\_resnet50\_float.tflite |
                      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp detr_resnet50_float.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=detr_resnet50_float.tflite
                      export LABELS_NAME=yolov8.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>10</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/conditional_detr_resnet50">Conditional-DETR-ResNet50</a></td>
            <td>Conditional cross-attention DETR for faster convergence.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/conditional_detr_resnet50) | conditional\_detr\_resnet50\_float.tflite |
                      | <a href="../labels/coco_labels_91.json">coco\_labels.json</a> | coco\_labels.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp conditional_detr_resnet50_float.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp coco_labels.json                        <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4               <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=conditional_detr_resnet50_float.tflite
                      export LABELS_NAME=coco_labels.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 50.0}" ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>11</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/rf_detr">RF-DETR</a></td>
            <td>Region-aware feature DETR with improved small-object detection.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/rf_detr) | rf\_detr\_float.tflite |
                      | <a href="../labels/coco_labels_91.json">coco\_labels.json</a> | coco\_labels.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp rf_detr_float.tflite            <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp coco_labels.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4       <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=rf_detr_float.tflite
                      export LABELS_NAME=coco_labels.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 70.0}" ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>12</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/rtmdet">RTMDet</a></td>
            <td>Real-time instance-level detector with dynamic label assignment.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/rtmdet) | rtmdet\_float.tflite |
                      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp rtmdet_float.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp yolov8.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=rtmdet_float.tflite
                      export LABELS_NAME=yolov8.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME settings="{\"confidence\": 50.0}" ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>13</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/yolor">Yolo-R</a></td>
            <td>Multi-scale object detection with implicit knowledge distillation.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/yolor) | yolor\_float.tflite |
                      | <a href="../labels/yolov8.json">yolov8.json</a> | yolov8.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp yolor_float.tflite         <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp yolov8.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=yolor_float.tflite
                      export LABELS_NAME=yolov8.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=yolov8 labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! text/x-raw ! queue ! obj_mux.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>
        </tbody>
      </table>
    </div>
  </Tab>
</Tabs>


# Pose Estimation
Source: https://imsdkdocs.qualcomm.com/supportedmodels/pose-estimation

QIM SDK supported models for Pose Estimation

***

<Tabs>
  <Tab title="TFLite Runtime">
    <div>
      <table>
        <colgroup>
          <col />

          <col />

          <col />
        </colgroup>

        <thead>
          <tr>
            <th>No.</th>
            <th>Model Name</th>
            <th>Description</th>
          </tr>
        </thead>

        <tbody>
          <tr>
            <td>1</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/hrnet_pose?searchTerm=HR">HRNetPose</a></td>
            <td>High-resolution representation network for 2D human pose keypoints.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Person/foot detection model](https://aihub.qualcomm.com/iot/models/foot_track_net) | `foot_track_net_w8a8.tflite` |
                      | <a href="../labels/foot_track_net.json">foot\_track\_net.json</a> | `foot_track_net.json` |
                      | <a href="../labels/foot_track_net_settings.json">foot\_track\_net\_settings.json</a> | `foot_track_net_settings.json` |
                      | [HRNet pose model](https://aihub.qualcomm.com/iot/models/hrnet_pose) | `hrnet_pose_w8a8.tflite` |
                      | <a href="../labels/hrnet.json">hrnet.json</a> | `hrnet.json` |
                      | <a href="../labels/hrnet_settings.json">hrnet\_settings.json</a> | `hrnet_settings.json` |
                      | Input video — use an encoded video file containing a single person in frame | `pose_sample.mp4` |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp foot_track_net_w8a8.tflite       <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp foot_track_net.json              <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp foot_track_net_settings.json     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp hrnet_pose_w8a8.tflite           <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp hrnet.json                       <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp hrnet_settings.json              <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp pose_sample.mp4                  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      mkdir -p $HOME/Downloads/qimsdk_samples/models $HOME/Downloads/qimsdk_samples/labels $HOME/Downloads/qimsdk_samples/media
                      export MODEL_NAME_1=foot_track_net_w8a8.tflite
                      export LABELS_NAME_1=foot_track_net.json
                      export LABELS_NAME_2=foot_track_net_settings.json
                      export MODEL_NAME_2=hrnet_pose_w8a8.tflite
                      export LABELS_NAME_3=hrnet.json
                      export LABELS_NAME_4=hrnet_settings.json
                      export SRC_VIDEO_NAME=pose_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                        qtimlvconverter name=stage_01_preproc mode=image-batch-non-cumulative \
                        qtimltflite name=stage_01_inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
                        external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" \
                        model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_1 \
                        qtimlpostprocess name=stage_01_postproc results=10 module=qpd labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_1 \
                        settings=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_2 \
                        qtimlvconverter name=stage_02_preproc mode=roi-batch-cumulative image-disposition=centre \
                        qtimltflite name=stage_02_inference delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
                        external-delegate-options="QNNExternalDelegate,backend_type=htp,htp_performance_mode=(string)2,log_level=(string)1;" \
                        model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME_2 \
                        qtimlpostprocess name=stage_02_postproc results=2 module=hrnet labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_3 \
                        settings=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME_4 \
                        filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                        v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                        tee name=t_split_1 \
                        t_split_1. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! stage_01_inference. stage_01_inference. ! queue ! \
                        stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! qtimetamux name=metamux_1 \
                        t_split_1. ! queue ! metamux_1. metamux_1. ! queue ! tee name=t_split_2 \
                        t_split_2. ! queue ! stage_02_preproc. stage_02_preproc. ! queue ! stage_02_inference. stage_02_inference. ! queue ! \
                        stage_02_postproc. stage_02_postproc. ! text/x-raw ! queue ! qtimetamux name=metamux_2 \
                        metamux_2. ! queue ! qtivoverlay ! queue ! waylandsink fullscreen=true sync=true \
                        t_split_2. ! queue ! metamux_2.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>
        </tbody>
      </table>
    </div>
  </Tab>
</Tabs>


# Semantic Segmentation
Source: https://imsdkdocs.qualcomm.com/supportedmodels/semantic-segmentation

QIM SDK supported models for Semantic Segmentation

***

<Tabs>
  <Tab title="TFLite Runtime">
    <div>
      <table>
        <colgroup>
          <col />

          <col />

          <col />
        </colgroup>

        <thead>
          <tr>
            <th>No.</th>
            <th>Model Name</th>
            <th>Description</th>
          </tr>
        </thead>

        <tbody>
          <tr>
            <td>1</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/deeplabv3_plus_mobilenet?searchTerm=DeepLabV3-Plus-MobileNet">DeepLabV3-Plus-MobileNet</a></td>
            <td>Atrous separable convolutions with encoder-decoder for segmentation.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/deeplabv3_plus_mobilenet?searchTerm=DeepLabV3-Plus-MobileNet) | deeplabv3\_plus\_mobilenet\_w8a8.tflite |
                      | <a href="../labels/dv3-argmax.json">dv3-argmax.json</a> | dv3-argmax.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp deeplabv3_plus_mobilenet_w8a8.tflite   <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp dv3-argmax.json                        <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4                     <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=deeplabv3_plus_mobilenet_w8a8.tflite
                      export LABELS_NAME=dv3-argmax.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=seg_mix sink_1::alpha=0.5 ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=deeplab-argmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw ! queue ! seg_mix.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>2</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/fcn_resnet50">FCN-ResNet50</a></td>
            <td>Fully convolutional network for dense pixel classification.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/fcn_resnet50) | fcn\_resnet50\_w8a8.tflite |
                      | <a href="../labels/dv3-argmax.json">dv3-argmax.json</a> | dv3-argmax.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp fcn_resnet50_w8a8.tflite                <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp dv3-argmax.json                         <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4                      <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=fcn_resnet50_w8a8.tflite
                      export LABELS_NAME=dv3-argmax.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=seg_mix sink_1::alpha=0.5 ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=deeplab-argmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw ! queue ! seg_mix.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>3</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/ffnet_122ns_lowres">FFNet-122NS-LowRes</a></td>
            <td>Fast Fusion Network for real-time semantic segmentation.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/ffnet_122ns_lowres) | ffnet\_122ns\_lowres\_w8a8.tflite |
                      | <a href="../labels/dv3-argmax.json">dv3-argmax.json</a> | dv3-argmax.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp ffnet_122ns_lowres_w8a8.tflite      <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp dv3-argmax.json                     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4                  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=ffnet_122ns_lowres_w8a8.tflite
                      export LABELS_NAME=dv3-argmax.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=seg_mix sink_1::alpha=0.5 ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=deeplab-argmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw ! queue ! seg_mix.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>4</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/ffnet_78s_lowres">FFNet-78S-LowRes</a></td>
            <td>Compact FFNet variant balancing speed and accuracy.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/ffnet_78s_lowres) | ffnet\_78s\_lowres\_w8a8.tflite |
                      | <a href="../labels/dv3-argmax.json">dv3-argmax.json</a> | dv3-argmax.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp ffnet_78s_lowres_w8a8.tflite        <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp dv3-argmax.json                     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4                  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=ffnet_78s_lowres_w8a8.tflite
                      export LABELS_NAME=dv3-argmax.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=seg_mix sink_1::alpha=0.5 ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=deeplab-argmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw ! queue ! seg_mix.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>5</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/pidnet?searchTerm=pid">PidNet</a></td>
            <td>Parallel information distillation network for real-time segmentation.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/pidnet?searchTerm=pid) | pidnet\_w8a8.tflite |
                      | <a href="../labels/dv3-argmax.json">dv3-argmax.json</a> | dv3-argmax.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp pidnet_w8a8.tflite                  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp dv3-argmax.json                     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4                  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=pidnet_w8a8.tflite
                      export LABELS_NAME=dv3-argmax.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=seg_mix sink_1::alpha=0.5 ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=deeplab-argmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw ! queue ! seg_mix.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>6</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/ddrnet23_slim?searchTerm=DDRNet23-Slim">DDRNet23-Slim</a></td>
            <td>Dual-resolution discrepancy network (slim) for fast segmentation.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 Float32 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/ddrnet23_slim?searchTerm=DDRNet23-Slim) | ddrnet23\_slim\_float.tflite |
                      | <a href="../labels/dv3-argmax.json">dv3-argmax.json</a> | dv3-argmax.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp ddrnet23_slim_float.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp dv3-argmax.json                     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4                  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=ddrnet23_slim_float.tflite
                      export LABELS_NAME=dv3-argmax.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=seg_mix sink_1::alpha=0.5 ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=deeplab-argmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw ! queue ! seg_mix.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>7</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/ffnet_40s?searchTerm=FFNet-40&chipsets=qualcomm-qcs6490-proxy">FFNet-40S</a></td>
            <td>Minimal FFNet for maximum throughput.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/ffnet_40s?searchTerm=FFNet-40\&chipsets=qualcomm-qcs6490-proxy) | ffnet\_40s\_w8a8.tflite |
                      | <a href="../labels/dv3-argmax.json">dv3-argmax.json</a> | dv3-argmax.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp ffnet_40s_w8a8.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp dv3-argmax.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4             <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=ffnet_40s_w8a8.tflite
                      export LABELS_NAME=dv3-argmax.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=seg_mix sink_1::alpha=0.5 ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=deeplab-argmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw ! queue ! seg_mix.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>8</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/ffnet_54s?searchTerm=FFNet-54&chipsets=qualcomm-qcs6490-proxy">FFNet-54S</a></td>
            <td>Small FFNet for low-latency segmentation.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/ffnet_54s?searchTerm=FFNet-54\&chipsets=qualcomm-qcs6490-proxy) | ffnet\_54s\_w8a8.tflite |
                      | <a href="../labels/dv3-argmax.json">dv3-argmax.json</a> | dv3-argmax.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp ffnet_54s_w8a8.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp dv3-argmax.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4             <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=ffnet_54s_w8a8.tflite
                      export LABELS_NAME=dv3-argmax.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=seg_mix sink_1::alpha=0.5 ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=deeplab-argmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw ! queue ! seg_mix.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>9</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/ffnet_78s?searchTerm=FFNet-78&chipsets=qualcomm-qcs6490-proxy">FFNet-78S</a></td>
            <td>Standard-resolution FFNet-78S segmentation model.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/ffnet_78s?searchTerm=FFNet-78\&chipsets=qualcomm-qcs6490-proxy) | ffnet\_78s\_w8a8.tflite |
                      | <a href="../labels/dv3-argmax.json">dv3-argmax.json</a> | dv3-argmax.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp ffnet_78s_w8a8.tflite          <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp dv3-argmax.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4             <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=ffnet_78s_w8a8.tflite
                      export LABELS_NAME=dv3-argmax.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=seg_mix sink_1::alpha=0.5 ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=deeplab-argmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw ! queue ! seg_mix.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>10</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/segformer_base">Segformer-Base</a></td>
            <td>Hierarchical transformer with lightweight MLP decoder.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/segformer_base) | segformer\_base\_w8a8.tflite |
                      | <a href="../labels/segformer.json">segformer.json</a> | segformer.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp segformer_base_w8a8.tflite         <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp segformer.json                     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4                 <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=segformer_base_w8a8.tflite
                      export LABELS_NAME=segformer.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=seg_mix sink_1::alpha=0.5 ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=deeplab-argmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw ! queue ! seg_mix.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>11</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/sinet">SINet</a></td>
            <td>Salient instance network for scene understanding.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 Float32 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/sinet) | sinet\_float.tflite |
                      | <a href="../labels/dv3-argmax.json">dv3-argmax.json</a> | dv3-argmax.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp sinet_float.tflite             <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp dv3-argmax.json                <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4             <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=sinet_float.tflite
                      export LABELS_NAME=dv3-argmax.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=seg_mix sink_1::alpha=0.5 ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=deeplab-argmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw ! queue ! seg_mix.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>12</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/pspnet">PSPNet</a></td>
            <td>Pyramid pooling module for global scene context aggregation.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/pspnet) | pspnet\_w8a8.tflite |
                      | <a href="../labels/dv3-argmax.json">dv3-argmax.json</a> | dv3-argmax.json |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/labels ~/Downloads/qimsdk_samples/media"
                        scp pspnet_w8a8.tflite                  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp dv3-argmax.json                     <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
                        scp ai_demo_sample.mp4                  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=pspnet_w8a8.tflite
                      export LABELS_NAME=dv3-argmax.json
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=seg_mix sink_1::alpha=0.5 ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! queue ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=deeplab-argmax labels=$HOME/Downloads/qimsdk_samples/labels/$LABELS_NAME ! video/x-raw ! queue ! seg_mix.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>
        </tbody>
      </table>
    </div>
  </Tab>
</Tabs>


# Super Resolution
Source: https://imsdkdocs.qualcomm.com/supportedmodels/super-resolution

QIM SDK supported models for Super Resolution

***

<Tabs>
  <Tab title="TFLite Runtime">
    <div>
      <table>
        <colgroup>
          <col />

          <col />

          <col />
        </colgroup>

        <thead>
          <tr>
            <th>No.</th>
            <th>Model Name</th>
            <th>Description</th>
          </tr>
        </thead>

        <tbody>
          <tr>
            <td>1</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/quicksrnetlarge?searchTerm=SR">QuickSRNetLarge</a></td>
            <td>Fast super-resolution network (large) for video upscaling.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/quicksrnetlarge) | quicksrnetlarge\_float.tflite |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
                        scp quicksrnetlarge_float.tflite     <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=quicksrnetlarge_float.tflite
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/quicksrnetlarge) | quicksrnetlarge\_w8a8.tflite |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
                        scp quicksrnetlarge_w8a8.tflite      <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=quicksrnetlarge_w8a8.tflite
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>2</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/quicksrnetmedium?searchTerm=SR">QuickSRNetMedium</a></td>
            <td>Fast super-resolution network (medium) balanced speed/quality.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/quicksrnetmedium) | quicksrnetmedium\_float.tflite |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
                        scp quicksrnetmedium_float.tflite    <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=quicksrnetmedium_float.tflite
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/quicksrnetmedium) | quicksrnetmedium\_w8a8.tflite |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
                        scp quicksrnetmedium_w8a8.tflite     <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=quicksrnetmedium_w8a8.tflite
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>3</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/quicksrnetsmall?searchTerm=SR">QuickSRNetSmall</a></td>
            <td>Lightweight super-resolution for real-time edge deployment.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/quicksrnetsmall) | quicksrnetsmall\_float.tflite |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
                        scp quicksrnetsmall_float.tflite     <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=quicksrnetsmall_float.tflite
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/quicksrnetsmall) | quicksrnetsmall\_w8a8.tflite |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
                        scp quicksrnetsmall_w8a8.tflite      <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=quicksrnetsmall_w8a8.tflite
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>4</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/xlsr?searchTerm=XLSR">XLSR</a></td>
            <td>Extreme lightweight super-resolution for mobile devices.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/xlsr) | xlsr\_float.tflite |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
                        scp xlsr_float.tflite                <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=xlsr_float.tflite
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/xlsr) | xlsr\_w8a8.tflite |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
                        scp xlsr_w8a8.tflite                 <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=xlsr_w8a8.tflite
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>5</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/sesr_m5?searchTerm=SESR">SESR-M5</a></td>
            <td>Single-image efficient super-resolution, 5-block variant.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/sesr_m5) | sesr\_m5\_float.tflite |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
                        scp sesr_m5_float.tflite             <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=sesr_m5_float.tflite
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/sesr_m5) | sesr\_m5\_w8a8.tflite |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
                        scp sesr_m5_w8a8.tflite              <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=sesr_m5_w8a8.tflite
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>6</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/real_esrgan_general_x4v3">Real-ESRGAN-General-x4v3</a></td>
            <td>General-purpose 4x upscaling with enhanced ESRGAN training.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/real_esrgan_general_x4v3) | real\_esrgan\_general\_x4v3\_w8a8.tflite |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
                        scp real_esrgan_general_x4v3_w8a8.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4       <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=real_esrgan_general_x4v3_w8a8.tflite
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/real_esrgan_general_x4v3) | real\_esrgan\_general\_x4v3\_float.tflite |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
                        scp real_esrgan_general_x4v3_float.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4        <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=real_esrgan_general_x4v3_float.tflite
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>7</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/real_esrgan_x4plus">Real-ESRGAN-x4plus</a></td>
            <td>Photo-realistic 4x super-resolution for natural images.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟢 W8A8 — Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/real_esrgan_x4plus) | real\_esrgan\_x4plus\_w8a8.tflite |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
                        scp real_esrgan_x4plus_w8a8.tflite   <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=real_esrgan_x4plus_w8a8.tflite
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" ! queue ! \
                      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>

                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/real_esrgan_x4plus) | real\_esrgan\_x4plus\_float.tflite |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
                        scp real_esrgan_x4plus_float.tflite  <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=real_esrgan_x4plus_float.tflite
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>

          <tr>
            <td>8</td>
            <td><a href="https://aihub.qualcomm.com/iot/models/esrgan">ESRGAN</a></td>
            <td>Enhanced super-resolution GAN for high-perceptual quality output.</td>
          </tr>

          <tr>
            <td>
              <Accordion title="Steps to Run on QIM SDK">
                <Accordion title="🟡 Float32 - Model Precision">
                  <Steps>
                    <Step title="Download Required Files">
                      | File | Save as |
                      | - | - |
                      | [Model](https://aihub.qualcomm.com/iot/models/esrgan) | esrgan\_float.tflite |
                      | <a href="https://github.com/qualcomm/sample-apps-for-qualcomm-linux/tree/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples">Input video</a> | ai\_demo\_sample.mp4 |

                      <Note>
                        If any downloaded file is a `.zip` archive, extract it on your host machine before copying:
                        `unzip filename.zip`
                      </Note>
                    </Step>

                    <Step title="Copy Files to Device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

                        ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/models ~/Downloads/qimsdk_samples/media"
                        scp esrgan_float.tflite              <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
                        scp ai_demo_sample.mp4  <user>@<device-ip>:~/Downloads/qimsdk_samples/media/
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Connect to device">
                      <CodeGroup>
                        ```bash SCP (SSH) theme={null}
                        # Run from your host machine — replace <user> and <device-ip>
                        ssh <user>@<device-ip>
                        ```
                      </CodeGroup>
                    </Step>

                    <Step title="Set Environment Variables">
                      ```bash Qualcomm Linux theme={null}
                      export MODEL_NAME=esrgan_float.tflite
                      export SRC_VIDEO_NAME=ai_demo_sample.mp4
                      ```
                    </Step>

                    <Step title="Run the Pipeline">
                      ```bash theme={null}
                      gst-launch-1.0 -e --gst-debug=2 \
                      filesrc location=$HOME/Downloads/qimsdk_samples/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
                      v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
                      tee name=t \
                      t. ! queue ! qtivcomposer name=mixer sink_0::position="<0, 0>" sink_0::dimensions="<960, 1080>" sink_1::position="<960, 0>" sink_1::dimensions="<960, 1080>" ! queue ! waylandsink fullscreen=true sync=true \
                      t. ! qtimlvconverter ! queue ! \
                      qtimltflite model=$HOME/Downloads/qimsdk_samples/models/$MODEL_NAME delegate=gpu ! queue ! \
                      qtimlpostprocess module=srnet ! video/x-raw ! queue ! mixer.
                      ```
                    </Step>
                  </Steps>
                </Accordion>
              </Accordion>
            </td>
          </tr>
        </tbody>
      </table>
    </div>
  </Tab>
</Tabs>