Skip to main content
This section covers QIM SDK AI sample applications that demonstrate custom preprocessing.

Prerequisites

Some of the steps in the pre-requisties will be removed from future releases once the necessary fixes are mainlined.
1

Set up Wi-Fi

Connect to the Wireless Access Point (Wi-Fi Router):
Check the connection and device status:
Login to the target deviceLocate the IP address of the device according to the type of network connection, using the UART console on the Linux host:For Ethernet:
For Wi-Fi:
Use the IP address from the Linux host to establish an SSH connection to the device:
Example:
Connect to the SSH shell using the following password:
2

Download Models and Artifacts

On the target device, obtain the download_artifacts.sh script, set executable permissions, and run it to download the model, media, and label files:
3

Enable qticamsrc

In the terminal of the target device, run the following command to enable the qticamsrc on Config #2:
4

Enable Audio and GPU Delegate

In the terminal of the target device, run the following command to enable audio:
To set the default devices for sink and source, get the device numbers from wpctl status and run the following command:
In the terminal of the target device, run the following command to enable the GPU delegate and backend:

AI Custom Preprocessing Applications


Object Detection (YOLOv8) with Custom Preprocessing

Common Description

gst-custom-preprocess-yolov8 is a GStreamer sample application that demonstrates how to run real-time object detection on Qualcomm platforms using a YOLOv8 model while substituting the built-in qtimlvconverter preprocessing stage with an application-provided (custom) preprocessing callback. Instead of relying on the converter element’s internal scale/format-conversion engine, the application sets engine=none on qtimlvconverter and connects to its process signal. The callback (gst_app_preprocess) manually converts the incoming NV12 video frame to an 8-bit RGB buffer, resizes it to the model’s expected input dimensions, and writes it directly into the tensor buffer consumed by inference. This pattern is useful when a model requires input preparation that the stock converter does not support (e.g. custom color-space conversion, non-standard resizing/letterboxing, or format packing). Detected objects are annotated with bounding boxes and rendered on the output (display, file, or an application-supplied sink).
Pipeline Diagram
YOLOv8 Object Detection Pipeline

Input and Output Capabilities

Input
The video source branch is fully configurable via --source / -s. Supported inputs (anything expressible as a GStreamer source bin) include: The custom preprocessing callback expects an NV12 input frame (it reads the Y/luma and interleaved U/V planes of the mapped GstVideoBlit to build the RGB model input), so any source branch must ultimately deliver video/x-raw,format=NV12.
Output
The output branch is selected via --output / -o: Any other value causes the application to abort with Unknown sink type.

Sample Model and Label Files

--model-base-path must point to a directory containing models/, labels/, and (for the default source/output) media/ subdirectories, e.g.:
If --model-base-path is not supplied, it defaults to $HOME.

Run the Application on the Target Device

  1. Copy (or ensure present on the target) the model, label, and sample media files in the layout described above.
  2. Run with defaults (file source, display output, model-base-path=$HOME):
  3. Run with an explicit model base path:
  4. Run against a custom source and write encoded output to file:
  5. Stop the application with Ctrl+C; the pipeline receives EOS and shuts down cleanly, or it stops automatically at end-of-stream for finite file sources.
  6. To display available options:

Configurations

Command line options (parsed with GOption): Fixed, compile-time pipeline configuration (not exposed on the command line):

Expected Output

Expected output

Pipeline Flow

  1. Source: The configured source branch decodes/produces NV12 video frames.
  2. Tee: Each frame is duplicated β€” one copy goes to the ML preprocessing branch, the other is held for later metadata attachment (qtimetamux).
  3. Custom preprocess (qtimlvconverter, engine=none): Because no built-in engine is configured, the element emits the process signal instead of transforming the frame itself. The connected callback, gst_app_preprocess:
    • Validates the ML tensor contract (UINT8 type, 4 dimensions, 3 channels).
    • Reads the Y (luma) and interleaved U/V (chroma) planes of the input GstVideoBlit.
    • Nearest-neighbor scales the frame to the model’s input width/height.
    • Converts each sampled NV12 pixel to 8-bit RGB (nv12_to_rgb).
    • Writes the resulting 3-channel RGB image into the output ML tensor buffer.
  4. Inference (qtimltflite): Runs yolov8_det_quantized.tflite through the external QNN delegate on the Hexagon HTP backend, producing raw bounding-box/score/class tensors.
  5. Postprocess (qtimlpostprocess, module=yolov8): Decodes the raw tensors into object bounding boxes using the yolov8 module β€” automatically dispatches to a mono-, dual-, or triple-tensor parsing path depending on the model’s output layout, applies a confidence threshold, and runs non-max suppression. Labels come from yolov8.json.
  6. Metadata mux (qtimetamux): Merges the postprocess results (as GStreamer metadata) back onto the original (tee’d) video frame.
  7. Overlay (qtivoverlay): Draws the bounding boxes/labels on top of the video frame.
  8. Sink: The annotated stream is sent to the selected output (display, video file, or appsink).
  9. Shutdown: SIGINT (Ctrl+C) or an EOS/ERROR bus message stops the GLib main loop; the pipeline transitions to NULL and all resources are released.

Palm Detection with Custom Preprocessing

Common Description

gst-custom-preprocess-palm-detect is a GStreamer sample application that demonstrates how to run real-time, palm detection on Qualcomm platforms while substituting the built-in qtimlvconverter preprocessing stage with an application-provided (custom) preprocessing callback. Instead of relying on the converter element’s internal scale/format-conversion engine, the application sets engine=none on qtimlvconverter and connects to its process signal. The callback (gst_app_preprocess) manually converts the incoming NV12 video frame to a float32 RGB buffer, resizes it to the model’s expected input dimensions, and writes it directly into the tensor buffer consumed by inference. This pattern is useful when a model requires input preparation that the stock converter does not support (e.g. custom color-space reduction, non-standard resizing/letterboxing, or format packing). It identifies the presence, location and rotation angle of any hands in the scene and attaches metadata, visualizes results with bounding boxes and rendered on the output (display, file, or an application-supplied sink).
Pipeline Diagram
Palm Detection Pipeline

Input and Output Capabilities

Input
The video source branch is fully configurable via --source / -s. Supported inputs (anything expressible as a GStreamer source bin) include: The custom preprocessing callback expects an NV12 input frame (it reads only the Y/luma plane of the mapped GstVideoBlit to build the grayscale model input), so any source branch must ultimately deliver video/x-raw,format=NV12.
Output
The output branch is selected via --output / -o: Any other value causes the application to abort with Unknown sink type.

Sample Model and Label Files

--model-base-path must point to a directory containing models/, labels/, and (for the default source/output) media/ subdirectories, e.g.:
If --model-base-path is not supplied, it defaults to $HOME.

Run the Application on the Target Device

  1. Copy (or ensure present on the target) the model, label, and sample media files in the layout described above.
  2. Run with defaults (camera source, display output, model-base-path=$HOME):
  3. Run with an explicit model base path:
  4. Run against a custom source and write encoded output to file:
  5. Stop the application with Ctrl+C; the pipeline receives EOS and shuts down cleanly, or it stops automatically at end-of-stream for finite file sources.
  6. To display available options:

Configurations

Command line options (parsed with GOption): Fixed, compile-time pipeline configuration (not exposed on the command line):

Expected Output

Palm Detection expected output

Pipeline Flow

  1. Source: The configured source branch decodes/produces NV12 video frames.
  2. Tee: Each frame is duplicated β€” one copy goes to the ML preprocessing branch, the other is held for later metadata attachment (qtimetamux).
  3. Custom preprocess (qtimlvconverter, engine=none): Because no built-in engine is configured, the element emits the process signal instead of transforming the frame itself. The connected callback, gst_app_preprocess:
    • Validates the ML tensor contract (UINT8 type, 4 dimensions, 1 channel).
    • Reads the Y (luma) plane of the input GstVideoBlit.
    • Nearest-neighbor scales it to the model’s input width/height.
    • Writes the resulting single-channel grayscale image into the output ML tensor buffer.
  4. Inference (qtimltflite): Runs palm_detection.tflite on the GPU delegate, producing raw score/bbox/landmark tensors.
  5. Postprocess (qtimlpostprocess, module=palmd): Decodes the raw tensors into face bounding boxes using the palmd module β€” applies a confidence threshold, reconstructs boxes from center/landmark offsets, squares them with margin, runs non-max suppression, and stabilizes boxes across frames. Labels come from palmd_labels.json.
  6. Metadata mux (qtimetamux): Merges the postprocess results (as GStreamer metadata) back onto the original (tee’d) video frame.
  7. Overlay (qtivoverlay): Draws the bounding boxes/labels on top of the video frame.
  8. Sink: The annotated stream is sent to the selected output (display, video file, or appsink).
  9. Shutdown: SIGINT (Ctrl+C) or an EOS/ERROR bus message stops the GLib main loop; the pipeline transitions to NULL and all resources are released.

Lightweight Face Detection with Custom Preprocessing

Common Description

gst-custom-preprocess-lightweight-face-detect is a GStreamer sample application that demonstrates how to run real-time, lightweight face detection on Qualcomm platforms while substituting the built-in qtimlvconverter preprocessing stage with an application-provided (custom) preprocessing callback. Instead of relying on the converter element’s internal scale/format-conversion engine, the application sets engine=none on qtimlvconverter and connects to its process signal. The callback (gst_app_preprocess) manually converts the incoming NV12 video frame to a single-channel grayscale buffer, resizes it to the model’s expected input dimensions, and writes it directly into the tensor buffer consumed by inference. This pattern is useful when a model requires input preparation that the stock converter does not support (e.g. custom color-space reduction, non-standard resizing/letterboxing, or format packing). Detected faces are annotated with bounding boxes and rendered on the output (display, file, or an application-supplied sink).
Pipeline Diagram
Lightweight Face Detection Pipeline

Input and Output Capabilities

Input
The video source branch is fully configurable via --source / -s. Supported inputs (anything expressible as a GStreamer source bin) include: The custom preprocessing callback expects an NV12 input frame (it reads only the Y/luma plane of the mapped GstVideoBlit to build the grayscale model input), so any source branch must ultimately deliver video/x-raw,format=NV12.
Output
The output branch is selected via --output / -o: Any other value causes the application to abort with Unknown sink type.

Sample Model and Label Files

--model-base-path must point to a directory containing models/, labels/, and (for the default source/output) media/ subdirectories, e.g.:
If --model-base-path is not supplied, it defaults to $HOME.

Run the Application on the Target Device

  1. Copy (or ensure present on the target) the model, label, and sample media files in the layout described above.
  2. Run with defaults (file source, display output, model-base-path=$HOME):
  3. Run with an explicit model base path:
  4. Run against a custom source and write encoded output to file:
  5. Stop the application with Ctrl+C; the pipeline receives EOS and shuts down cleanly, or it stops automatically at end-of-stream for finite file sources.
  6. To display available options:

Configurations

Command line options (parsed with GOption): Fixed, compile-time pipeline configuration (not exposed on the command line):

Expected Output

Lightweight Face Detection expected output

Pipeline Flow

  1. Source: The configured source branch decodes/produces NV12 video frames.
  2. Tee: Each frame is duplicated β€” one copy goes to the ML preprocessing branch, the other is held for later metadata attachment (qtimetamux).
  3. Custom preprocess (qtimlvconverter, engine=none): Because no built-in engine is configured, the element emits the process signal instead of transforming the frame itself. The connected callback, gst_app_preprocess:
    • Validates the ML tensor contract (UINT8 type, 4 dimensions, 1 channel).
    • Reads the Y (luma) plane of the input GstVideoBlit.
    • Nearest-neighbor scales it to the model’s input width/height.
    • Writes the resulting single-channel grayscale image into the output ML tensor buffer.
  4. Inference (qtimltflite): Runs face_det_lite_w8a8.tflite on the GPU delegate, producing raw score/bbox/landmark tensors.
  5. Postprocess (qtimlpostprocess, module=qfd): Decodes the raw tensors into face bounding boxes using the qfd module β€” applies a confidence threshold, reconstructs boxes from center/landmark offsets, squares them with margin, runs non-max suppression, and stabilizes boxes across frames. Labels come from qfd-labels.json.
  6. Metadata mux (qtimetamux): Merges the postprocess results (as GStreamer metadata) back onto the original (tee’d) video frame.
  7. Overlay (qtivoverlay): Draws the bounding boxes/labels on top of the video frame.
  8. Sink: The annotated stream is sent to the selected output (display, video file, or appsink).
  9. Shutdown: SIGINT (Ctrl+C) or an EOS/ERROR bus message stops the GLib main loop; the pipeline transitions to NULL and all resources are released.

Troubleshooting

If any model isn’t available after downloading the script file, you can download the model manually from IoT β€” Qualcomm AI Hub and push it to the target device:
For example:
Remount the file system with read/write permissions:For Qualcomm Linux:
If you cannot locate the qticamsrc plugin, ensure that the camera server is running and clear the GStreamer cache using the following commands:
To enable basic GStreamer logging, run the following before launching the application:
To increase verbosity for specific plugins, use a comma-separated list with log levels (1–7):
To redirect logs to a file for offline analysis: