> ## Documentation Index
> Fetch the complete documentation index at: https://imsdkdocs.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# qtimetatransform

> Transforms video-associated metadata attached to GStreamer buffers in real time.

# Overview

`qtimetatransform` is a GStreamer metadata transformation element designed for real-time manipulation of **video-associated metadata** in multimedia and AI pipelines.

In modern video processing systems, raw video frames are rarely the only data being processed. Additional information - such as object detections, classifications, timestamps, tracking IDs, or other machine‑learning outputs - is attached to video buffers as metadata. Downstream elements often require this metadata to be **filtered, reshaped, synchronized, or adapted** to match changes in the video stream or application requirements.

`qtimetatransform` provides a structured and performant way to transform such metadata while preserving buffer synchronization and pipeline correctness.

At a high level, this element:

* Consumes metadata attached to incoming video buffers
* Applies controlled transformations to that metadata
* Outputs updated metadata that remains logically and temporally aligned with the transformed video stream

This makes `qtimetatransform` a foundational building block for vision pipelines where metadata must evolve alongside video frames.

![qtimetatransform\_overview](https://mintcdn.com/qimsdk/OnrxUB-8f1JMWTkp/plugin-reference/images/metatransform0.png?fit=max\&auto=format\&n=OnrxUB-8f1JMWTkp\&q=85\&s=71ec8d4ca0f64acec101b24fc92c23fe)

# Why Metadata Transformation Is Necessary

To understand why `qtimetatransform` exists, it is important to start from first principles.

### What is Metadata in GStreamer?

In GStreamer, **metadata** is auxiliary information attached to buffers. Unlike raw pixel data, metadata does not represent image content itself, but **describes or annotates that content**.

Examples include:

* Bounding boxes produced by object detection models
* Classification labels (e.g., “person”, “vehicle”)
* Confidence scores
* Frame-level timestamps or IDs
* Custom application-specific annotations

Metadata travels with the buffer through the pipeline and is expected to remain valid and meaningful at every stage.

### The Core Problem

Once video frames are transformed - resized, cropped, rotated, or even dropped - the associated metadata can become **invalid** unless it is explicitly updated.

For example:

* A bounding box defined in 1920×1080 coordinates is incorrect after the frame is resized to 1280×720.
* Cropping a region of interest requires translating and clipping object coordinates.
* Some metadata may no longer be relevant downstream and should be removed to reduce processing cost.

`qtimetatransform` exists to solve this consistency problem in a systematic and reproducible way.

# High-Level Functionality

`qtimetatransform` operates on metadata only. It does **not** modify pixel data directly.

Its responsibilities include:

* Selecting which metadata types should be processed
* Transforming metadata fields based on pipeline configuration
* Ensuring temporal and logical consistency between metadata and video buffers
* Emitting metadata in formats expected by downstream elements

The element is designed to be composable and deterministic, so downstream behavior is predictable and debuggable.

# Hierarchy

[GObject](https://docs.gtk.org/gobject/)\

   [GstObject](https://gstreamer.freedesktop.org/documentation/gstreamer/gstobject.html?gi-language=c)\

      [GstElement](https://gstreamer.freedesktop.org/documentation/gstreamer/gstelement.html?gi-language=c)\

         [GstBaseTransform](https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c)\

            GstMetaTransform

# Pad Templates

### sink

| Capabilities | |
| - | - |
| `video/x-raw(ANY)` | `format: NA` |
| Availability: *Always* | |
| Direction: *sink* | |

### src

| Capabilities | |
| - | - |
| `video/x-raw(ANY)` | `format: NA` |
| Availability: *Always* | |
| Direction: *source* | |

# Element Properties

| Property | Description |
| - | - |
| `module` | Module name that is going to be used for processing the buffer metas `Type: Enum "GstMetaTranformModules"` `Default: 0, "none"` `Range:`     `(0): none - No module, default invalid mode`     `(1): roi-person-merge - meta-transform-roi-person-merge`     `(2): roi-label-moving-average - meta-transform-roi-label-moving-average`     `(3): roi-auto-framing - meta-transform-roi-auto-framing`     `(4): roi-palmd - meta-transform-roi-palmd` `Flags: readable, writable` |
| `module-params` | Parameters specific to the chosen module for processing/filtering/conversion of buffer metas. The format is in GstStructure string. `Type: String` `Default: null` `Flags: readable, writable` |

## Internal Architecture

`qtimetatransform` is implemented as a standard GStreamer element with two pads:

* **Sink Pad (input)** \

  Receives video buffers along with attached metadata from upstream elements (e.g., ML inference, tracking, analytics).

* **Source Pad (output)** \

  Outputs the same video buffers with transformed metadata attached.

No video frame copying or pixel manipulation occurs inside this element.

# Processing Flow

The internal processing follows a strict, ordered sequence.

**Caps Negotiation** \

Caps negotiation establishes the expected buffer type and metadata compatibility between upstream and downstream elements.

While `qtimetatransform` does not change video caps, it validates that:

* Incoming buffers support metadata attachment
* Required metadata types are present (if configured as mandatory)

This ensures early failure if the pipeline configuration is incompatible.

### Modular Transformation Model

`qtimetatransform` is built around a **modular transformation architecture**.\
Instead of implementing a single, fixed metadata transformation logic, the plugin acts as a **runtime-selected container of transformation modules**, each responsible for a specific metadata processing strategy.

This design enables flexibility, extensibility, and clear separation of concerns.

A **module** in `qtimetatransform` is:

* A self-contained unit of metadata transformation logic
* Implemented as a C module with a well-defined interface
* Responsible for transforming one specific type or interpretation of metadata

Each module defines:

* How metadata is **parsed**
* How metadata is **transformed**
* How metadata is **emitted** downstream

The core `qtimetatransform` element itself does not implement transformation logic directly.\
Instead, it delegates all metadata processing to the selected module.

**Buffer and Metadata Reception** \

For each incoming buffer:

* The buffer is received on the sink pad
* Attached metadata structures are enumerated
* Only metadata matching configured criteria is considered for transformation

Metadata that does not match transformation rules may be:

* Passed through unchanged
* Explicitly filtered out
* Left untouched, depending on configuration

**Metadata Transformation** \

Each selected metadata instance is transformed according to the configured rules.

This may include:

* Coordinate scaling and translation
* Normalization or denormalization of values
* Field filtering or renaming
* Structural adaptation to downstream expectations

Transformations are applied **in-place where safe**, or via controlled metadata replacement when required.

The element guarantees that metadata remains logically consistent relative to the associated frame.

**Buffer Forwarding** \

Once metadata processing is complete:

* The original video buffer is forwarded
* Updated metadata is attached
* Buffer timestamps and ordering are preserved

Downstream elements receive a fully synchronized video–metadata pair.

# Performance Characteristics

`qtimetatransform` is designed to be lightweight and CPU-efficient:

* No frame copying
* No GPU dependency
* Minimal per-buffer overhead

Performance scales linearly with metadata size, not video resolution.

This makes the element suitable for high-throughput pipelines and embedded deployments.

# Usage

**End-to-End Video Analytics with Metadata Transformation and Overlay**

This pipeline demonstrates a **complete offline video analytics workflow** where:

* A recorded video is decoded
* Objects are detected using a neural network
* Detection results are attached to video buffers as metadata
* That metadata is **merged, transformed, and visualized**
* The final annotated video is re-encoded and saved to disk

The key challenge addressed here is ensuring that **machine-learning metadata remains synchronized, meaningful, and correctly interpreted** as it moves through a complex GStreamer pipeline that splits, recombines, and processes data in parallel.

![qtimetatransform\_usecase](https://mintcdn.com/qimsdk/OnrxUB-8f1JMWTkp/plugin-reference/images/metatransform1.png?fit=max\&auto=format\&n=OnrxUB-8f1JMWTkp\&q=85\&s=dcb0307b765cba5fc65537ece66a7011)

**1. Download Required Files**

| File | Download | Save as |
| - | - | - |
| YOLO model | [Qualcomm Yolo Model](https://aihub.qualcomm.com/iot/models/yolox) | `yolov8_det_quantized.tflite` |
| YOLO labels | [yolov8.json](../labels/yolov8.json) | `yolov8.json` |
| Input video | [Input video](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/raw/refs/heads/main/qualcomm-linux/artifacts/videos/demo_samples/) | `Draw_1080p_180s_30FPS.mp4` |

**2. Copy files to device**

```bash SCP (SSH) theme={null}
ssh <user>@<device-ip> "mkdir -p $HOME/{models,labels,media}"
scp yolov8_det_quantized.tflite <user>@<device-ip>:$HOME/models/
scp yolov8.json                 <user>@<device-ip>:$HOME/labels/
scp Draw_1080p_180s_30FPS.mp4    <user>@<device-ip>:$HOME/media/
```

**3. Connect to device**

```bash SCP (SSH) theme={null}
ssh <user>@<device-ip>
```

**4. Set environment variables**

Run below command on your device

```bash theme={null}
mkdir -p $HOME/{models,labels,media}
```

**5. Run the pipeline**

```bash theme={null}
gst-launch-1.0 -e --gst-debug=2 \
filesrc location=$HOME/media/Draw_1080p_180s_30FPS.mp4 ! qtdemux ! queue ! h264parse ! v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! tee name=split \
split. ! queue ! qtimetamux name=metamux ! queue ! qtimetatransform module=<module> ! queue ! qtivoverlay ! v4l2h264enc capture-io-mode=4 output-io-mode=4 ! h264parse ! mp4mux ! filesink location=$HOME/media/output.mp4 \
split. ! queue ! qtimlvconverter ! queue ! qtimltflite model=$HOME/models/yolov8_det_quantized.tflite delegate=external external-delegate-path=libQnnTFLiteDelegate.so external-delegate-options="QNNExternalDelegate,backend_type=htp;" ! queue ! \
qtimlpostprocess module=yolov8 labels=$HOME/labels/yolov8.json settings="{\"confidence\": 50.0}" ! text/x-raw ! metamux.
```

## Group Auto-Framing Using ROI Transformation

The `roi-person-merge` and `roi-auto-framing` modules can be combined to implement intelligent group framing. Instead of tracking a single detected subject, this workflow merges all detected persons in a scene into one group region, then applies temporal smoothing so that the resulting framing window follows the group steadily rather than reacting abruptly to every detection change.

The pipeline performs the following operations:

1. Runs YOLOv8 object detection on incoming frames.
2. Generates one ROI per detected person.
3. Merges all detected person ROIs into a single group ROI using `roi-person-merge`.
4. Applies temporal auto-framing using `roi-auto-framing` to generate a smooth framing window.
5. Supplies the resulting ROI to [`qtivsplit`](https://imsdkdocs.qualcomm.com/plugin-reference/qtivsplit) for display.

![Group Auto Framing](https://mintcdn.com/qimsdk/OnrxUB-8f1JMWTkp/plugin-reference/images/metatransform2.png?fit=max\&auto=format\&n=OnrxUB-8f1JMWTkp\&q=85\&s=506c10c1ba4844a59f5f71f850916468)

## roi-person-merge

The **roi-person-merge** module combines multiple ROI metas that share the same label into a single enclosing ROI.

**Processing Behavior**

* Scans all attached `GstVideoRegionOfInterestMeta` entries
* Selects ROIs matching the configured label
* Computes a bounding rectangle enclosing all matching ROIs
* Applies optional padding around the merged ROI
* Removes the original matching ROIs
* Attaches a single merged ROI

If no matching ROIs are found, the buffer is forwarded unchanged.

**Module Parameters**

| Parameter | Description |
| - | - |
| `label` | ROI label to merge (for example: `person`) |
| `padding` | Fractional padding applied around the merged ROI (`padding=0.1` adds 10% padding on each side) |

**Example**

```bash theme={null}
qtimetatransform module=roi-person-merge module-params="label=person,padding=0.1"
```

## roi-auto-framing

The **roi-auto-framing** module generates a stabilized framing ROI that smoothly follows a subject or group over time.

**Processing Behavior**

* Consumes the target ROI
* Applies temporal filtering and stabilization
* Computes a smooth framing rectangle
* Generates an `auto-framing` ROI
* Removes previous ROI metas
* Attaches only the final auto-framing ROI

If no input ROI is available, the module emits a full-frame ROI covering the complete field of view.

**Example**

```bash theme={null}
qtimetatransform module=roi-auto-framing
```

## Example Pipeline

This pipeline detects people with YOLOv8, merges their ROIs into a single group region, applies temporal auto-framing, and displays the stabilized framing window using [`qtivsplit`](https://imsdkdocs.qualcomm.com/plugin-reference/qtivsplit) in `single-roi-meta` mode.

**1. Download Required Files**

| File | Download | Save as |
| - | - | - |
| YOLO model | [Qualcomm Yolo Model](https://aihub.qualcomm.com/iot/models/yolox) | `yolov8_det_w8a8.dlc` |
| YOLO labels | [yolov8.json](../labels/yolov8.json) | `yolov8.json` |

**2. Copy files to device**

```bash SCP (SSH) theme={null}
ssh <user>@<device-ip> "mkdir -p $HOME/{models,labels}"
scp yolov8_det_w8a8.dlc <user>@<device-ip>:$HOME/models/
scp yolov8.json         <user>@<device-ip>:$HOME/labels/
```

**3. Connect to device**

```bash SCP (SSH) theme={null}
ssh <user>@<device-ip>
```

**4. Run the pipeline**

```bash theme={null}
gst-launch-1.0 -e \
qtiqmmfsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! tee name=t \
t. ! queue ! qtimlvconverter ! queue ! qtimlqairt backend=htp tensors="<boxes,scores,class_idx>" model=$HOME/models/yolov8_det_w8a8.dlc ! queue ! \
qtimlpostprocess module=yolov8 labels=$HOME/labels/yolov8.json settings="{\"confidence\":70.0}" ! text/x-raw ! qtimetamux name=mux ! \
qtimetatransform module=roi-person-merge module-params="label=person,padding=0.1" ! \
qtimetatransform module=roi-auto-framing ! tee name=display_tee \
display_tee. ! queue ! qtivsplit src_0::mode=single-roi-meta ! videoconvert ! fpsdisplaysink video-sink="waylandsink fullscreen=true" text-overlay=false \
display_tee. ! queue ! fakesink \
t. ! queue ! mux.
```