Overview

qtimetatransform is a GStreamer metadata transformation element designed for real-time manipulation of video-associated metadata in multimedia and AI pipelines. In modern video processing systems, raw video frames are rarely the only data being processed. Additional information - such as object detections, classifications, timestamps, tracking IDs, or other machine‑learning outputs - is attached to video buffers as metadata. Downstream elements often require this metadata to be filtered, reshaped, synchronized, or adapted to match changes in the video stream or application requirements. qtimetatransform provides a structured and performant way to transform such metadata while preserving buffer synchronization and pipeline correctness. At a high level, this element:

Consumes metadata attached to incoming video buffers
Applies controlled transformations to that metadata
Outputs updated metadata that remains logically and temporally aligned with the transformed video stream

This makes qtimetatransform a foundational building block for vision pipelines where metadata must evolve alongside video frames.

Why Metadata Transformation Is Necessary

To understand why qtimetatransform exists, it is important to start from first principles.

What is Metadata in GStreamer?

In GStreamer, metadata is auxiliary information attached to buffers. Unlike raw pixel data, metadata does not represent image content itself, but describes or annotates that content. Examples include:

Bounding boxes produced by object detection models
Classification labels (e.g., “person”, “vehicle”)
Confidence scores
Frame-level timestamps or IDs
Custom application-specific annotations

Metadata travels with the buffer through the pipeline and is expected to remain valid and meaningful at every stage.

The Core Problem

Once video frames are transformed - resized, cropped, rotated, or even dropped - the associated metadata can become invalid unless it is explicitly updated. For example:

A bounding box defined in 1920×1080 coordinates is incorrect after the frame is resized to 1280×720.
Cropping a region of interest requires translating and clipping object coordinates.
Some metadata may no longer be relevant downstream and should be removed to reduce processing cost.

qtimetatransform exists to solve this consistency problem in a systematic and reproducible way.

High-Level Functionality

qtimetatransform operates on metadata only. It does not modify pixel data directly. Its responsibilities include:

Selecting which metadata types should be processed
Transforming metadata fields based on pipeline configuration
Ensuring temporal and logical consistency between metadata and video buffers
Emitting metadata in formats expected by downstream elements

The element is designed to be composable and deterministic, so downstream behavior is predictable and debuggable.

Hierarchy

GObject
   GstObject
      GstElement
         GstBaseTransform
            GstMetaTransform

Pad Templates

sink

Capabilities
`video/x-raw(ANY)`	`format: NA`
Availability: Always
Direction: sink

src

Capabilities
`video/x-raw(ANY)`	`format: NA`
Availability: Always
Direction: source

Element Properties

Property	Description
`module`	Module name that is going to be used for processing the buffer metas `Type: Enum "GstMetaTranformModules"` `Default: 0, "none"` `Range:` `(0): none - No module, default invalid mode` `(1): roi-person-merge - meta-transform-roi-person-merge` `(2): roi-label-moving-average - meta-transform-roi-label-moving-average` `(3): roi-auto-framing - meta-transform-roi-auto-framing` `(4): roi-palmd - meta-transform-roi-palmd` `Flags: readable, writable`
`module-params`	Parameters specific to the chosen module for processing/filtering/conversion of buffer metas. The format is in GstStructure string. `Type: String` `Default: null` `Flags: readable, writable`

Internal Architecture

qtimetatransform is implemented as a standard GStreamer element with two pads:

Sink Pad (input)
Receives video buffers along with attached metadata from upstream elements (e.g., ML inference, tracking, analytics).
Source Pad (output)
Outputs the same video buffers with transformed metadata attached.

No video frame copying or pixel manipulation occurs inside this element.

Processing Flow

The internal processing follows a strict, ordered sequence. Caps Negotiation
Caps negotiation establishes the expected buffer type and metadata compatibility between upstream and downstream elements. While qtimetatransform does not change video caps, it validates that:

Incoming buffers support metadata attachment
Required metadata types are present (if configured as mandatory)

This ensures early failure if the pipeline configuration is incompatible.

Modular Transformation Model

qtimetatransform is built around a modular transformation architecture.
Instead of implementing a single, fixed metadata transformation logic, the plugin acts as a runtime-selected container of transformation modules, each responsible for a specific metadata processing strategy. This design enables flexibility, extensibility, and clear separation of concerns. A module in qtimetatransform is:

A self-contained unit of metadata transformation logic
Implemented as a C module with a well-defined interface
Responsible for transforming one specific type or interpretation of metadata

Each module defines:

How metadata is parsed
How metadata is transformed
How metadata is emitted downstream

The core qtimetatransform element itself does not implement transformation logic directly.
Instead, it delegates all metadata processing to the selected module. Buffer and Metadata Reception
For each incoming buffer:

The buffer is received on the sink pad
Attached metadata structures are enumerated
Only metadata matching configured criteria is considered for transformation

Metadata that does not match transformation rules may be:

Passed through unchanged
Explicitly filtered out
Left untouched, depending on configuration

Metadata Transformation
Each selected metadata instance is transformed according to the configured rules. This may include:

Coordinate scaling and translation
Normalization or denormalization of values
Field filtering or renaming
Structural adaptation to downstream expectations

Transformations are applied in-place where safe, or via controlled metadata replacement when required. The element guarantees that metadata remains logically consistent relative to the associated frame. Buffer Forwarding
Once metadata processing is complete:

The original video buffer is forwarded
Updated metadata is attached
Buffer timestamps and ordering are preserved

Downstream elements receive a fully synchronized video–metadata pair.

Performance Characteristics

qtimetatransform is designed to be lightweight and CPU-efficient:

No frame copying
No GPU dependency
Minimal per-buffer overhead

Performance scales linearly with metadata size, not video resolution. This makes the element suitable for high-throughput pipelines and embedded deployments.

Usage

End-to-End Video Analytics with Metadata Transformation and Overlay This pipeline demonstrates a complete offline video analytics workflow where:

A recorded video is decoded
Objects are detected using a neural network
Detection results are attached to video buffers as metadata
That metadata is merged, transformed, and visualized
The final annotated video is re-encoded and saved to disk

The key challenge addressed here is ensuring that machine-learning metadata remains synchronized, meaningful, and correctly interpreted as it moves through a complex GStreamer pipeline that splits, recombines, and processes data in parallel.

Download Required Files

File	Download	Save as
YOLO model	Qualcomm Yolo Model	`yolov8_det_quantized.tflite`
YOLO labels	yolov8.json	`yolov8.json`
Input video	Input video	`Draw_1080p_180s_30FPS.mp4`

Copy files to device

ssh <user>@<device-ip> "mkdir -p $HOME/{models,labels,media}"
scp yolov8_det_quantized.tflite <user>@<device-ip>:$HOME/models/
scp yolov8.json                 <user>@<device-ip>:$HOME/labels/
scp Draw_1080p_180s_30FPS.mp4    <user>@<device-ip>:$HOME/media/

Connect to device

ssh <user>@<device-ip>

Set environment variables

Run below command on your device

mkdir -p $HOME/{models,labels,media}

Run the pipeline

gst-launch-1.0 -e --gst-debug=2 \
filesrc location=$HOME/media/Draw_1080p_180s_30FPS.mp4 ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! tee name=split \
split. ! queue ! qtimetamux name=metamux ! queue ! qtimetatransform module=<module> ! queue ! qtivoverlay ! v4l2h264enc capture-io-mode=4 output-io-mode=4 ! h264parse ! mp4mux ! filesink location=$HOME/media/output.mp4 \
split. ! queue ! qtimlvconverter ! queue ! qtimltflite model=$HOME/models/yolov8_det_quantized.tflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" ! queue ! \
qtimlpostprocess module=yolov8 labels=$HOME/labels/yolov8.json settings="{\"confidence\": 50.0}" ! text/x-raw ! metamux.

Group Auto-Framing Using ROI Transformation

The roi-person-merge and roi-auto-framing modules can be combined to implement intelligent group framing. Instead of tracking a single detected subject, this workflow merges all detected persons in a scene into one group region, then applies temporal smoothing so that the resulting framing window follows the group steadily rather than reacting abruptly to every detection change. The pipeline performs the following operations:

Runs YOLOv8 object detection on incoming frames.
Generates one ROI per detected person.
Merges all detected person ROIs into a single group ROI using roi-person-merge.
Applies temporal auto-framing using roi-auto-framing to generate a smooth framing window.
Supplies the resulting ROI to qtivsplit for display.

roi-person-merge

The roi-person-merge module combines multiple ROI metas that share the same label into a single enclosing ROI. Processing Behavior

Scans all attached GstVideoRegionOfInterestMeta entries
Selects ROIs matching the configured label
Computes a bounding rectangle enclosing all matching ROIs
Applies optional padding around the merged ROI
Removes the original matching ROIs
Attaches a single merged ROI

If no matching ROIs are found, the buffer is forwarded unchanged. Module Parameters

Parameter	Description
`label`	ROI label to merge (for example: `person`)
`padding`	Fractional padding applied around the merged ROI (`padding=0.1` adds 10% padding on each side)

Example

qtimetatransform module=roi-person-merge module-params="label=person,padding=0.1"

roi-auto-framing

The roi-auto-framing module generates a stabilized framing ROI that smoothly follows a subject or group over time. Processing Behavior

Consumes the target ROI
Applies temporal filtering and stabilization
Computes a smooth framing rectangle
Generates an auto-framing ROI
Removes previous ROI metas
Attaches only the final auto-framing ROI

If no input ROI is available, the module emits a full-frame ROI covering the complete field of view. Example

qtimetatransform module=roi-auto-framing

Example Pipeline

This pipeline detects people with YOLOv8, merges their ROIs into a single group region, applies temporal auto-framing, and displays the stabilized framing window using qtivsplit in single-roi-meta mode.

Download Required Files

File	Download	Save as
YOLO model	Qualcomm Yolo Model	`yolov8_det_w8a8.dlc`
YOLO labels	yolov8.json	`yolov8.json`

Copy files to device

ssh <user>@<device-ip> "mkdir -p $HOME/{models,labels}"
scp yolov8_det_w8a8.dlc <user>@<device-ip>:$HOME/models/
scp yolov8.json         <user>@<device-ip>:$HOME/labels/

Connect to device

ssh <user>@<device-ip>

Run the pipeline

gst-launch-1.0 -e \
qtiqmmfsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! tee name=t \
t. ! queue ! qtimlvconverter ! queue ! qtimlqairt backend=htp tensors="<boxes,scores,class_idx>" model=$HOME/models/yolov8_det_w8a8.dlc ! queue ! \
qtimlpostprocess module=yolov8 labels=$HOME/labels/yolov8.json settings="{\"confidence\":70.0}" ! text/x-raw ! qtimetamux name=mux ! \
qtimetatransform module=roi-person-merge module-params="label=person,padding=0.1" ! \
qtimetatransform module=roi-auto-framing ! tee name=display_tee \
display_tee. ! queue ! qtivsplit src_0::mode=single-roi-meta ! videoconvert ! fpsdisplaysink video-sink="waylandsink fullscreen=true" text-overlay=false \
display_tee. ! queue ! fakesink \
t. ! queue ! mux.

Overview

Supported AI Models

Plugins

Advanced Topics

qtimetatransform

Overview

Why Metadata Transformation Is Necessary

What is Metadata in GStreamer?

The Core Problem

High-Level Functionality

Hierarchy

Pad Templates

sink

src

Element Properties

Internal Architecture

Processing Flow

Modular Transformation Model

Performance Characteristics

Usage

Group Auto-Framing Using ROI Transformation

roi-person-merge

roi-auto-framing

Example Pipeline

​Overview

​Why Metadata Transformation Is Necessary

​What is Metadata in GStreamer?

​The Core Problem

​High-Level Functionality

​Hierarchy

​Pad Templates

​sink

​src

​Element Properties

​Internal Architecture

​Processing Flow

​Modular Transformation Model

​Performance Characteristics

​Usage

​Group Auto-Framing Using ROI Transformation

​roi-person-merge

​roi-auto-framing

​Example Pipeline

Overview

Why Metadata Transformation Is Necessary

What is Metadata in GStreamer?

The Core Problem

High-Level Functionality

Hierarchy

Pad Templates

sink

src

Element Properties

Internal Architecture

Processing Flow

Modular Transformation Model

Performance Characteristics

Usage

Group Auto-Framing Using ROI Transformation

roi-person-merge

roi-auto-framing

Example Pipeline