SDK Architecture and API Reference

Pipeline SDK Design

Overview This SDK delivers a streamlined, modern interfaces, engineered to facilitate the rapid development, execution, and management of sophisticated multimedia pipelines. By implementing a high-level abstraction layer, the SDK encapsulates complex underlying GStreamer based mechanisms — including element negotiation, signal management, event loops, and state transitions — allowing engineers to prioritize core application logic over boilerplate framework intricacies. Key Capabilities The wrapper empowers developers to architect advanced multimedia and AI-enhanced workflows with precision and efficiency. It supports the seamless integration of diverse components, such as video capture sources, signal processing nodes, machine learning inference engines, graphical overlays, and rendering outputs. Uniquely designed to decouple implementation from the underlying framework, the SDK eliminates the need for deep GStreamer expertise or compile-time dependencies on GStreamer headers, ensuring a clean, robust, and accessible development experience.

APIs

Public APIs:

Pipeline — build & control pipelines fluently.
Element — generic element wrapper with variadic property setting, and state control. It can be instantiated at create time as well as at runtime to update the pipeline. Some elements have special functions which can be controlled by dedicated element types. Those derived types can be used as a plain Element if the special functions are not needed.
- AppSrc — app-facing source for zero-copy buffer pull. Feeds application-produced buffers into the pipeline through a callback or a direct push API, and can signal EOS.
- AppSink — app-facing sink for zero-copy buffer push. Delivers pipeline output back to application code with zero copy.
- CamSrc — built-in camera element that exposes an API for capturing images at runtime.
- MLVConverter — ML video converter (qtimlvconverter). Exposes an API for custom pre-processing implementation in application code through a callback.
- MLPostprocess — ML postprocess element (qtimlpostprocess). Exposes an API for custom post-processing implementation in application code through a callback. The callback’s output type annotation selects the ML task (classification, object detection, pose estimation, depth estimation, segmentation, raw tensors).
- MLVideoONNXBin — ONNX inference bin (qtimlvideoonnxbin). Exposes an API for custom pre-processing and post-processing implementation through callbacks.
- MLVideoQNNBin — QNN inference bin (qtimlvideoqnnbin). Exposes an API for custom pre-processing and post-processing implementation through callbacks.
- MLVideoSNPEBin — SNPE inference bin (qtimlvideosnpebin). Exposes an API for custom pre-processing and post-processing implementation through callbacks.
- MLVideoTFLiteBin — TFLite inference bin (qtimlvideotflitebin). Exposes an API for custom pre-processing and post-processing implementation through callbacks.
Buffer — opaque buffer that bridges native GStreamer memory with timestamp helpers and zero-copy paths.
StreamFilter and derived concrete stream helpers: VideoFilter (all RAW formats - YUV, RGB, bayer), H264Filter, TensorFilter, TextFilter, AudioFilter, ImageFilter. These filters are used to set characteristics of the streams.
Logging controls — SetImsdkLogLevel() and SetImsdkGstLogMode() select the SDK’s log verbosity and whether GStreamer’s output is simplified or passed through raw. See Logging and Diagnostics.

Reference flows

Pipeline

Purpose A convenient wrapper around a Gst.Pipeline that:

Builds pipelines using element factory names or pre-constructed wrapper elements.
Manages pipeline lifecycle and execution state.
Performs delayed linking for dynamic/request-pad elements automatically via pad-added.

Key Responsibilities

Create and own the underlying Gst.Pipeline instance.
Add elements, assign properties, and insert caps filters.
Link elements (immediately when possible, deferred otherwise).
Manage the full pipeline lifecycle (prepare, start, wait, stop).
Install bus watches to handle ERROR, EOS, and state transitions.
Integrate with runtime shutdown notifications.

A minimal pipeline Add the elements in the order data flows through them, then run:

from qimsdk import Pipeline, VideoFilter

pipeline = (
    Pipeline("my-pipeline")
    .add("filesrc", "src", "location", "/home/user/media/video.mp4")
    .add("qtdemux", "demux")
    .add("h264parse", "parse")
    .add("v4l2h264dec", "decoder", "output-io-mode", 4, "capture-io-mode", 4)
    .add_stream_filter("vf", VideoFilter().format("NV12"))
    .add("waylandsink", "display", "fullscreen", True)
)

pipeline.execute()

execute() starts the pipeline, blocks until EOS/error/termination, and tears it down.

Adjacent elements are linked in add order. In the example above src → demux → parse → decoder → vf → display are linked simply because that is the order of the add() calls, so a linear graph needs no explicit linking. Use link() when the graph is not linear — a tee splitting into a display branch and an inference branch, or a qtimetamux rejoining them.

qtdemux is an example of the delayed linking described above: its src pads appear only once the container has been parsed, so the demux → parse link is completed automatically when the pad shows up. No pad-added handling is needed in application code.

Building the pipeline

Method	Description
`Pipeline(name: str)`	Creates an empty pipeline. `name` is the `Gst.Pipeline` name, used in log messages.
`classmethod Pipeline.from_yaml(name: str, config: str)`	Builds a complete pipeline from YAML configuration content. `config` is the YAML text itself, not a path — read the file first. See Describing a pipeline in YAML.
`Pipeline.add(factory_or_element, name=None, args, *props) -> Pipeline`	Creates an element from a factory name (or adds an externally constructed `Element`) under a unique instance name, and applies any trailing key/value properties immediately. Required for the specialized wrappers (`AppSrc`, `AppSink`, `CamSrc`, `MLVConverter`, `MLPostprocess`, the ML bins) so callbacks can be attached before the element joins the pipeline.
`Pipeline.add_stream_filter(name, caps=None) -> Pipeline`	Inserts a caps filter between the previous and next element. Accepts either `(name, caps)` or a single `StreamFilter` argument, in which case the element name is derived from the filter’s class. See StreamFilter.
`Pipeline.link(*names) -> Pipeline`	Links the named elements in the given order. Needed only for non-linear graphs.

add()’s optional key/value pairs are the element’s properties, validated as soon as the element is added — an unknown property raises ValueError naming the property and the element type, instead of being silently ignored:

# "fulscreen" is not a waylandsink property -> raises ValueError
pipeline.add("waylandsink", "display", "fulscreen", True)

Branching with explicit links Add order cannot express a branch, so name the chains explicitly. Element instance names are the identifiers.

pipeline = (
    Pipeline("branching-pipeline")
    .add("filesrc", "src", "location", path)
    .add("qtdemux", "demux")
    .add("h264parse", "parse")
    .add("v4l2h264dec", "decoder", "output-io-mode", 4, "capture-io-mode", 4)
    .add_stream_filter("vf", VideoFilter().format("NV12"))
    .add("tee", "split")
    .add("queue", "q1")
    .add("qtimlvconverter", "preprocessing")
    .add("qtimltflite", "inferencing", "model", model_path)
    .add("qtimlpostprocess", "postprocessing", "module", "yolov8")
    .add_stream_filter("mlf", TextFilter())
    .add("qtimetamux", "mlmuxer")
    .add("qtivoverlay", "overlay")
    .add("waylandsink", "display", "fullscreen", True)
    # trunk, up to the tee
    .link("src", "demux", "parse", "decoder", "vf", "split")
    # branch 1: raw video to the metadata muxer
    .link("split", "mlmuxer")
    # branch 2: inference path, rejoining at the muxer
    .link("split", "q1", "preprocessing", "inferencing", "postprocessing",
          "mlf", "mlmuxer", "overlay", "display")
)

pipeline.execute()

Elements whose src pads are SOMETIMES or REQUEST — qtdemux, tsdemux, tee, qtivsplit — cannot be linked at construction time, because the pads do not exist yet. The pipeline detects this from the pad templates, defers the link, and completes it from a pad-added handler it installs itself. Request pads on the sink side (for example qtimetamux, qtivcomposer) are requested and released automatically. No pad-added callback is needed in application code.

A link between two elements that both have only static pads and no common caps fails immediately with RuntimeError naming both elements. A deferred link that never resolves surfaces at run time instead; on error the pipeline logs which pads stayed unlinked.

Running the pipeline A pipeline is always in one of three states, and the lifecycle methods are the transitions between them:

State	Meaning
`NULL`	No resources allocated. Initial state, and the state after `stop()`. Devices are closed, buffer pools released.
`PAUSED`	Elements are allocated and caps are negotiated between linked pads, but the clock is not running and no data flows. Sinks have pre-rolled. This is the state in which elements can be added, removed, or relinked.
`PLAYING`	The clock is running and buffers flow through the graph.

execute() performs the whole NULL → PLAYING → NULL cycle, so most applications never call the individual transitions. prepare(), activate(), and deactivate() — the transitions through PAUSED — are covered in Advanced state control.

Method	Description
`Pipeline.execute() -> None`	`start()`, `wait()`, `stop()` in sequence. The normal entry point.
`Pipeline.start() -> Pipeline`	Moves the pipeline to `PLAYING`. Non-blocking: the state change is asynchronous and this returns without waiting for it. Raises `RuntimeError` if the transition fails.
`Pipeline.wait() -> Pipeline`	Blocks until the pipeline is done: `EOS` on the bus, `ERROR` on the bus, `stop()` from another thread, or `SIGINT`/`SIGTERM`.
`Pipeline.stop() -> Pipeline`	Moves the pipeline to `NULL` and releases resources. Blocks until teardown completes.
`Pipeline.eos(enabled: bool) -> Pipeline`	When enabled, `stop()` first sends an end-of-stream event through the pipeline and waits for it to come back from the sinks, so that data already inside the pipeline is processed before teardown. Off by default. See Draining the pipeline on shutdown.

start() is non-blocking, so it must be paired with wait() (or your own blocking logic). A script that calls start() and exits destroys the pipeline before buffers flow.

To run the pipeline while the application does other work, split execute() apart:

pipeline.start()   # returns immediately

# ... application work, runtime property changes, snapshots ...

pipeline.wait()    # block until EOS / error / Ctrl+C
pipeline.stop()

Draining the pipeline on shutdown By default stop() tears the pipeline down as fast as it can. Every element goes to NULL immediately and whatever buffers are still in flight — queued between elements, held inside an encoder, not yet written by a sink — are discarded. This is usually what you want: the pipeline stops without delay. Some pipelines must not lose those buffers, and for them eos(True) changes the shutdown sequence. Instead of stopping straight away, stop() sends an end-of-stream event into the pipeline. The event travels downstream element by element; each element finishes processing the data it has already accepted, passes the event on, and the sinks report it back on the bus. Only once that end-of-stream notification comes back does stop() tear the pipeline down. Nothing that had already entered the pipeline is dropped. The trade-off is shutdown latency: stop() now blocks until the last buffer has made it through the whole graph, which takes as long as the remaining data needs. Use eos(True) when the tail of the stream is part of the result:

Muxed recordings. Containers such as MP4 and MKV are written by a muxer (mp4mux, matroskamux) that writes its index and header metadata — the moov atom, in MP4’s case — only on end-of-stream. Cut it off early and the file is left incomplete, often unplayable rather than merely short.
Encoders with buffered frames. Encoders keep several frames in flight for B-frames, lookahead, and similar optimizations. End-of-stream is what makes the encoder flush those frames downstream; without it the tail of the recording is simply missing.
Any sink where losing the final buffered data is unacceptable — file sinks, muxed outputs, and applications that expect to receive every frame through AppSink.

Not every sink needs draining, though. A live network stream can usually just be cut off, and a receiver will not care that the last few buffers never arrived. The question is not whether a sink writes somewhere, but whether anything downstream depends on the data still sitting in the pipeline. Leave eos() off (the default) when the newest data is all that matters and stopping promptly is preferable — live preview to a display, a camera feed on screen, or writing self-contained files where each buffer is already complete on its own (JPEG frames through multifilesink, for example, which needs no finalization).

pipeline = (
    Pipeline("cam-encoder-pipeline")
    .add("qtiqmmfsrc", "source")
    .add_stream_filter("vf", VideoFilter().format("NV12")
                                          .resolution(1920, 1080)
                                          .framerate(30))
    .add("v4l2h264enc", "encoder",
         "output-io-mode", "dmabuf-import", "capture-io-mode", "dmabuf")
    .add("h264parse", "parser")
    .add("mp4mux", "muxer")
    .add("filesink", "sink", "location", f"{HOME_PATH}/media/output.mp4")
    .eos(True)          # drain, so mp4mux can finalize the container
)

pipeline.execute()

If the pipeline has already reached end-of-stream on its own — a file source that ran out of data, for instance — stop() detects it and tears down immediately instead of sending a second event and waiting again. Enabling eos(True) therefore costs nothing for pipelines that end naturally; it only matters when you stop a pipeline that is still running.

Advanced state control For reconfiguring a loaded pipeline — adding, removing, or relinking elements without returning to NULL.

Method	Description
`Pipeline.prepare() -> Pipeline`	`NULL` → `PAUSED`. Allocates resources and negotiates caps without running the clock. Blocking.
`Pipeline.activate() -> Pipeline`	`PAUSED` → `PLAYING`.
`Pipeline.deactivate() -> Pipeline`	`PLAYING` → `PAUSED`. Resources stay allocated.

Accessing elements at run time

Method	Description
`Pipeline.get(name: str, as_type=None) -> Any`	Returns the named element. With `as_type` omitted, returns the generic `Element` wrapper. With `as_type` set to a specialized class (for example `CamSrc`), returns that class’s `cast()` of the element instead. Raises `KeyError` if the name is not in the pipeline.

Unlike the C++ SDK’s get<T>(), Python has no compile-time generics, so the type is passed as an ordinary argument. To reach a specialized element’s own API (for example CamSrc.image_capture()), pass the class as as_type:

from qimsdk import CamSrc

# qtiqmmfsrc was added as "source"; CamSrc exposes its capture API.
cam = pipeline.get("source", CamSrc)
cam.image_capture()

Runtime property and per-pad configuration — get() reaches any element by name while the pipeline is loaded or playing:

# Change a property at run time.
pipeline.get("display").set("fullscreen", False)

# Per-pad properties on a request pad of qtivcomposer.
pipeline.get("composer").input(1).set("alpha", 0.5)

See the sample applications for complete image-capture pipelines.

get() returns a new Element wrapper on every call, each referencing the same underlying Gst.Element. There is no cached identity to keep in sync — call it again whenever you need the element.

Describing a pipeline in YAML The graph — elements, properties, stream filters, and links — can be described in YAML instead of in code. This separates what the pipeline is from what the application does with it: the same script can run a different topology, a different model, or a different source by pointing it at another configuration file. That makes it useful where the pipeline is not fixed up front:

Deploying one script across variants — the same application against a built-in camera, a USB camera, or an RTSP stream, switching source and format per device.
Tuning without code changes — resolution, framerate, model path, or delegate changed on the target, without touching the script.
Keeping topology under review — the pipeline shape lives in a file that can be diffed and versioned on its own.

Loading from YAML does not give up application logic: the elements are ordinary pipeline members afterwards, so get() reaches them as usual.

pipeline:
  elements:
    - type: filesrc
      name: src
      location: ~/media/video.mp4

    - type: qtdemux
      name: demux

    - type: h264parse
      name: parse

    - type: v4l2h264dec
      name: decoder
      output-io-mode: 4
      capture-io-mode: 4

    - type: filter
      name: videofilter
      video:
        format: NV12

    - type: waylandsink
      name: display
      fullscreen: true

Everything lives under a single root key, pipeline:, which takes three independent child keys:

pipeline:
  eos: true             # optional — EOS-on-shutdown, same as .eos(True) in code
  elements:             # required — the elements, in add order
    - type: ...
  links:                # optional — explicit links, for non-linear graphs
    - [ ... ]

Key	Required	Description
`eos`	No	Boolean. Equivalent to calling `eos()`. Defaults to false when omitted.
`elements`	Yes	Sequence of elements. Add order is link order, exactly as with `add()`.
`links`	No	Sequence of chains, each a list of element names. Equivalent to one `link()` call per entry. Omit it for linear pipelines.

Within elements:, each entry needs type (the factory name) and name (the instance name); every remaining key is applied as a property of that element. An entry with type: filter inserts a stream filter instead, described by a video:, image:, h264:, tensor:, text:, audio:, or raw caps: block. links: mirrors link() — one sequence per chain, so the branching pipeline shown earlier becomes:

  links:
    - [split, mlmuxer]
    - [source, videostream, split, q1, preprocessing, inferencing,
       postprocessing, mlf, mlmuxer, overlay, display]

from_yaml() takes the configuration content itself, so read the file first:

from qimsdk import Pipeline

with open(config_path, "r", encoding="utf-8") as f:
    config = f.read()

pipeline = Pipeline.from_yaml("demo-pipeline", config)
pipeline.execute()

The names in the configuration are the handles the application uses afterwards. Given a config that declares a qtimlpostprocess element named postprocessing, the application can attach its post-processing callback without the topology being hard-coded:

from qimsdk import Pipeline, MLPostprocess, ObjectDetections

with open(config_path, "r", encoding="utf-8") as f:
    config = f.read()

pipeline = Pipeline.from_yaml("demo-pipeline", config)

def detection_callback(mlframe, mlparams, detections: ObjectDetections):
    ...
    return True

postprocessing = pipeline.get("postprocessing", MLPostprocess)
postprocessing.set_handler(detection_callback)

pipeline.execute()

Element

Purpose Generic wrapper around a Gst.Element with fluent, variadic property setting. Can be instantiated at create time or used at runtime to update a live pipeline. Serves as the base class for all specialized element types. Key Responsibilities

Create GStreamer elements from a factory name.
Provide fluent, variadic property setting for all common types and StreamFilter (caps).
Implement pad linking/unlinking with error checking.
Support state publication to pipeline (sync, deactivate, stop).
Expose per-pad configuration through Port.
Serve as the base class for specialized wrappers.

from qimsdk import Element

display = Element("waylandsink", "display")
display.set("fullscreen", True)
display.set("sync", False)

Method	Description
`Element(factory: str, name: str \| None = None)`	Creates the element from a GStreamer factory name.
`Element.set(args, *props) -> Element`	Sets one or more properties. Accepts key/value pairs positionally, keyword arguments, or both, and returns `self` for chaining.
`Element.link(downstream, src_pad="src", sink_pad="sink") -> Element`	Links this element’s src pad to `downstream`’s sink pad.
`Element.unlink(downstream=None, src_pad="src", sink_pad="sink") -> Element`	Unlinks this element’s src pad from `downstream`’s sink pad.
`Element.sync() -> Element`	Synchronizes the element’s state with its parent pipeline’s state. For runtime link/unlink.
`Element.deactivate() -> Element`	Sets the element’s state to `PAUSED`.
`Element.stop() -> Element`	Moves the element to `NULL` state. For runtime link/unlink.
`Element.input(name_or_id=0, id=None) -> Port`	Returns a reference to an input port, for per-pad configuration.
`Element.output(name_or_id=0, id=None) -> Port`	Returns a reference to an output port, for per-pad configuration.

Per-pad configuration with Port Some elements are configured per pad rather than per element — a compositor’s inputs each have their own position, size, and alpha. input() and output() return a Port for one pad, which carries the same fluent, validated set():

# Second input of qtivcomposer: blend at 50% alpha.
pipeline.get("composer").input(1).set("alpha", 0.5)

Method	Description
`Port.set(args, *props) -> Port`	Sets one or more pad properties, same as `Element.set()`.

AppSrc

Purpose AppSrc wraps the appsrc element, the entry point for feeding application-produced buffers into a pipeline. It manages the need-data/enough-data signal plumbing so the application only has to supply buffers, either from a callback or by pushing them directly. Key Responsibilities

Validate that the wrapped element is indeed an appsrc.
Register and dispatch the underlying need-data and enough-data signals.
Let the application supply buffers on demand through set_buffer_producer(), or push them directly with push_buffer().
Signal end-of-stream on this element’s output.
Manage lifetime of the connected signal handlers.

from qimsdk import AppSrc, VideoFilter

appsrc = AppSrc("src")
appsrc.set("is-live", True, "block", True)
appsrc.set("format", AppSrc.Format.TIME, "do-timestamp", True)
appsrc.set("caps", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))

Method	Description
`AppSrc(name: str \| None = None)`	Instantiates an application source element with an optional instance name.
`classmethod AppSrc.cast(element) -> AppSrc`	Wraps an existing `Element` (for example one returned by `Pipeline.get()`) as an `AppSrc`.
`AppSrc.set(args, *props) -> AppSrc`	Sets one or more properties, same as `Element.set()`.
`AppSrc.set_caps(caps) -> AppSrc`	Sets the `caps` property from a `StreamFilter`, a raw caps string, or a `Gst.Caps` object.
`AppSrc.set_buffer_producer(producer) -> AppSrc`	Attaches a callback executed when more data is needed. Data control is driven by the pipeline.
`AppSrc.set_enough_handler(enough) -> AppSrc`	Attaches a callback executed when the data source should be stopped. Data control is driven by the pipeline.
`AppSrc.push_buffer(buffer: Buffer) -> bool`	Pushes a buffer into the AppSrc/pipeline stream. Data control is driven by the application.
`AppSrc.end_of_stream() -> None`	Sends an end-of-stream event to the AppSrc/pipeline stream.

AppSrc.Format mirrors appsrc’s own format property — an IntEnum with DEFAULT, BYTES, TIME, BUFFERS, PERCENT.

AppSink

Purpose AppSink wraps the appsink element, the exit point that delivers pipeline output back to application code. It pulls samples through appsink’s action signals and hands them to the application as Buffer. Key Responsibilities

Validate that the wrapped element is indeed an appsink.
Pull samples via appsink’s action signals and convert them into Buffer.
Deliver each buffer to the application through set_buffer_consumer().
Deliver the first (preroll) sample separately through set_preroll_handler().
Notify the application of end-of-stream through set_eos_handler().
Manage lifetime of the connected signal handlers.

from qimsdk import AppSink

appsink = AppSink("sink")
appsink.set("emit-signals", True, "max-buffers", 5, "drop", True)
appsink.set_buffer_consumer(lambda buffer: consume(buffer))

Method	Description
`AppSink(name: str \| None = None)`	Instantiates an application sink element with an optional instance name.
`classmethod AppSink.cast(element) -> AppSink`	Wraps an existing `Element` (for example one returned by `Pipeline.get()`) as an `AppSink`.
`AppSink.set(args, *props) -> AppSink`	Sets one or more properties, same as `Element.set()`.
`AppSink.set_buffer_consumer(consumer) -> AppSink`	Attaches a listener for buffers pulled from the pipeline.
`AppSink.set_preroll_handler(preroll) -> AppSink`	Attaches a listener for the first (preroll) sample only.
`AppSink.set_eos_handler(eos) -> AppSink`	Attaches a listener for end-of-stream.

CamSrc

Purpose CamSrc wraps the qtiqmmfsrc camera source element and exposes its still/burst image-capture action signals as ordinary method calls, so an application can trigger a snapshot without touching GStreamer signals directly. Key Responsibilities

Validate that the wrapped element is indeed a qtiqmmfsrc.
Trigger still or burst image capture through image_capture().
Cancel any pending capture request through cancel_capture().

Method	Description
`CamSrc(name: str \| None = None)`	Instantiates a camera source element with an optional instance name.
`classmethod CamSrc.cast(element) -> CamSrc`	Wraps an existing `Element` (for example one returned by `Pipeline.get()`) as a `CamSrc`.
`CamSrc.image_capture(args, *kwargs) -> bool`	Captures still or burst images. Accepts `count`, `mode`, and `metadata` positionally or as keyword arguments.
`CamSrc.cancel_capture() -> bool`	Cancels any capture requests still pending.

Reach a qtiqmmfsrc already in the pipeline through Pipeline.get()’s as_type argument to use its capture API:

from qimsdk import CamSrc

cam = pipeline.get("source", CamSrc)
cam.image_capture()

MLVConverter

Purpose MLVConverter wraps the qtimlvconverter element, which converts raw video frames into the input tensor format expected by a model. Beyond the properties available on the generic Element, it lets the application replace the built-in conversion with its own implementation through a callback. Key Responsibilities

Validate that the wrapped element is a qtimlvconverter.
Convert incoming video frames into model input tensors.
Optionally delegate the conversion to an application-supplied pre-process callback.

from qimsdk import MLVConverter

preprocessing = MLVConverter("preprocessing")
# engine=none disables internal preprocessing path in qtimlvconverter.
preprocessing.set("engine", "none")
preprocessing.set_handler(preprocess_callback)

Method	Description
`MLVConverter(name: str \| None = None)`	Instantiates a `qtimlvconverter` element with an optional instance name.
`classmethod MLVConverter.cast(element) -> MLVConverter`	Wraps an existing `Element` (for example one returned by `Pipeline.get()`) as an `MLVConverter`.
`MLVConverter.set(args, *props) -> MLVConverter`	Sets one or more properties, same as `Element.set()`.
`MLVConverter.set_handler(handler) -> MLVConverter`	Registers the external pre-process callback. Requires `engine=none` to take effect.

MLPostprocess

Purpose MLPostprocess wraps the qtimlpostprocess element, which decodes model output tensors into ML metadata attached to the frame. It lets the application replace the built-in decoding module with its own implementation through a callback. Key Responsibilities

Validate that the wrapped element is a qtimlpostprocess.
Decode output tensors into ML metadata (classifications, detections, poses, depth maps, segmentations, or raw tensors).
Optionally delegate the decoding to an application-supplied post-process callback.
Select the callback’s ML task automatically from its output-parameter type annotation.

Post-process callback set_handler() takes a single callback and, unlike a per-task setter, infers which ML task it handles from the type annotation on the callback’s output parameter. The callback takes either 3 parameters (mlframe, mlparams, results) or 4 (mlpostprocess, mlframe, mlparams, results); the output parameter must be annotated with one of the marker types below.

Output annotation	ML task
`ImageClassifications`	Image classification
`AudioClassifications`	Audio classification
`ObjectDetections`	Object detection
`Poses`	Pose estimation
`DepthMaps`	Depth estimation
`Segmentations`	Image segmentation
`Tensors`	Raw output tensors

from qimsdk import MLPostprocess, ImageClassifications

def classification_callback(mlframe, mlparams, classifications: ImageClassifications):
    ...
    return True

postprocessing = MLPostprocess("postprocessing")
postprocessing.set("results", 1)
postprocessing.set_handler(classification_callback)

Method	Description
`MLPostprocess(name: str \| None = None)`	Instantiates a `qtimlpostprocess` element with an optional instance name.
`classmethod MLPostprocess.cast(element) -> MLPostprocess`	Wraps an existing `Element` (for example one returned by `Pipeline.get()`) as an `MLPostprocess`.
`MLPostprocess.set(args, *props) -> MLPostprocess`	Sets one or more properties, same as `Element.set()`.
`MLPostprocess.set_handler(handler, kind: str \| None = None) -> MLPostprocess`	Registers a callback that decodes tensors into the ML task selected by `kind`, or inferred from the callback’s output-parameter type annotation if `kind` is omitted.

ML inference bins

Purpose The ML video bins wrap a full pre-process → inference → post-process chain into a single element, so a whole AI branch can be added to the pipeline as one node instead of being assembled from separate elements. Four bins are available, one per inference runtime:

Python class	GStreamer element	Runtime
`MLVideoONNXBin`	`qtimlvideoonnxbin`	ONNX Runtime
`MLVideoQNNBin`	`qtimlvideoqnnbin`	Qualcomm® AI Engine Direct (QNN)
`MLVideoSNPEBin`	`qtimlvideosnpebin`	Snapdragon® Neural Processing Engine (SNPE)
`MLVideoTFLiteBin`	`qtimlvideotflitebin`	LiteRT (TensorFlow Lite)

All four expose an identical API and differ only in the wrapped element and the runtime-specific properties. Each accepts a custom pre-process callback, a custom post-process callback, or both. Key Responsibilities

Validate that the wrapped element matches the expected bin factory.
Run pre-processing, inference, and post-processing inside a single element.
Optionally delegate pre-processing to an application-supplied callback via set_preprocess_handler().
Optionally delegate post-processing to an application-supplied callback via set_postprocess_handler().

Properties Because the bin embeds several stages, its properties are prefixed by the stage they configure — preprocess-*, inference-*, and postprocess-*. For example inference-model on the bin corresponds to model on a standalone inference element, and postprocess-labels corresponds to labels on a standalone qtimlpostprocess.

Set preprocess-engine to none to disable the internal pre-processing path when you supply your own pre-process callback.

The bin’s callbacks are set_preprocess_handler() and set_postprocess_handler() — distinct names from the standalone MLVConverter.set_handler() and MLPostprocess.set_handler(), since a bin exposes both stages on one object.

Usage Custom post-processing — the bin handles pre-processing and inference internally, the application decodes the output tensors:

from qimsdk import MLVideoTFLiteBin, VideoFilter, ObjectDetections

def detection_callback(mlframe, mlparams, detections: ObjectDetections):
    ...
    return True

mlbin = MLVideoTFLiteBin("mlbin")
mlbin.set("inference-delegate", "external",
          "inference-external-delegate-path", "libQnnTFLiteDelegate.so",
          "inference-external-delegate-options",
          "QNNExternalDelegate,backend_type=htp;",
          "inference-model", f"{HOME_PATH}/models/yolov8_det_quantized.tflite")
mlbin.set_postprocess_handler(detection_callback)

pipeline = (
    Pipeline("mlbin-external-detection")
    .add("filesrc", "src", "location", f"{HOME_PATH}/media/video.mp4")
    .add("qtdemux", "demux")
    .add("h264parse", "parse")
    .add("v4l2h264dec", "decoder", "output-io-mode", 4, "capture-io-mode", 4)
    .add_stream_filter("vf", VideoFilter().format("NV12"))
    .add(mlbin)
    .add("qtivoverlay", "overlay")
    .add("waylandsink", "display", "fullscreen", True)
)

pipeline.execute()

Custom pre-processing — the application converts frames into input tensors, the bin runs inference and its built-in post-process module:

mlbin = MLVideoTFLiteBin("mlbin")

# preprocess-engine=none disables internal preprocessing path in mlbin.
mlbin.set("preprocess-engine", "none")
mlbin.set("inference-delegate", "external")
mlbin.set("inference-external-delegate-path", "libQnnTFLiteDelegate.so")
mlbin.set("inference-external-delegate-options",
          "QNNExternalDelegate,backend_type=htp;")
mlbin.set("inference-model", f"{HOME_PATH}/models/yolov8_det_quantized.tflite")
mlbin.set("postprocess-module", "yolov8")
mlbin.set("postprocess-labels", f"{HOME_PATH}/labels/yolov8.json")

def preprocess_callback(blits, output) -> bool:
    ...
    return True

mlbin.set_preprocess_handler(preprocess_callback)

Swapping runtimes is a matter of changing the class and the runtime-specific properties; the callback registration is unchanged.

from qimsdk import MLVideoQNNBin, MLVideoSNPEBin, MLVideoONNXBin

qnn_bin = MLVideoQNNBin("mlbin")     # qtimlvideoqnnbin
snpe_bin = MLVideoSNPEBin("mlbin")   # qtimlvideosnpebin
onnx_bin = MLVideoONNXBin("mlbin")   # qtimlvideoonnxbin

APIs The table below applies to each of MLVideoONNXBin, MLVideoQNNBin, MLVideoSNPEBin, and MLVideoTFLiteBin; substitute the concrete class name for <Bin>.

Method	Description
`<Bin>(name: str \| None = None)`	Creates the corresponding `qtimlvideo*bin` element with an optional instance name.
`<Bin>.set(args, *props) -> <Bin>`	Sets one or more properties, same as `Element.set()`. Properties are stage-prefixed: `preprocess-`, `inference-`, `postprocess-*`.
`<Bin>.set_preprocess_handler(handler) -> <Bin>`	Registers the external pre-process callback. Requires `preprocess-engine=none` to take effect.
`<Bin>.set_postprocess_handler(handler, kind: str \| None = None) -> <Bin>`	Registers the external post-process callback. Accepts the same callback shapes as `MLPostprocess.set_handler()`.

Buffer

Purpose Unified buffer abstraction over native GStreamer memory. Provides read/write access, timestamp metadata (pts, dts, duration), resize support, and zero-copy transfer between AppSink and AppSrc. Key Responsibilities

Own and manage mapping/unmapping of underlying Gst.Buffer or Gst.Sample.
Provide read/write access to buffer memory.
Support resizing and writable allocation for AppSrc.
Carry timestamps (PTS/DTS/duration) and apply them back to Gst.Buffer.
Enable zero-copy extraction using take_gst_buffer().

def on_buffer(buf):
    data = buf.data()
    pts = buf.pts

appsink.set_buffer_consumer(on_buffer)

Method	Description
`Buffer(gst_buffer=None, size=None)`	Wraps an existing `Gst.Buffer`, or allocates a new one of `size` bytes.
`classmethod Buffer.from_readable_sample(sample) -> Buffer`	Wraps the buffer inside a `Gst.Sample`, as delivered by `AppSink`’s preroll handler.
`Buffer.data() -> memoryview \| None`	Maps the buffer for reading (or writing, if writable) and returns a `memoryview` onto it.
`Buffer.size() -> int`	Returns the buffer’s size in bytes.
`Buffer.resize(size: int) -> None`	Reallocates the buffer to `size` bytes, preserving existing data.
`Buffer.is_writable() -> bool`	Returns whether the buffer can be mapped for writing.
`Buffer.is_readonly() -> bool`	Returns `True` if the buffer is valid but not writable.
`Buffer.valid() -> bool`	Returns `True` if the buffer wraps a `Gst.Buffer`.
`Buffer.take_gst_buffer() -> Any \| None`	Transfers ownership of the underlying `Gst.Buffer` and clears this wrapper, enabling zero-copy hand-off (used internally by `AppSrc.push_buffer()`).
`Buffer.set_pts(pts_ns)` / `Buffer.set_dts(dts_ns)` / `Buffer.set_duration(duration_ns)`	Sets the buffer’s timestamp fields, in nanoseconds.
`Buffer.pts` / `Buffer.dts` / `Buffer.duration`	Read-only properties for the buffer’s timestamp fields; `None` if the buffer is empty.

StreamFilter

Purpose StreamFilter describes the characteristics of a media stream between two pipeline stages so they can negotiate concrete parameters rather than relying on defaults. Without explicit parameters, adjacent components may fall back to suboptimal formats, trigger implicit conversions, or produce mismatched configurations. Specialized subtypes provide fluent, type-safe setters for each domain. Raw caps strings are also accepted for cases not covered by typed setters or when copying directly from gst-launch output. When to Use StreamFilter

When two adjacent stages can interoperate in multiple ways, and you want to select explicit parameters instead of relying on defaults.
When you need to optimize latency or performance by avoiding implicit conversions (e.g., choose a hardware-friendly pixel format).
When you want predictable behavior across platforms (e.g., lock color range/primaries/transfer/matrix).
When you integrate tensors for AI/ML and must precisely specify dimensions and element types.
When you need a human-readable summary of the stream parameters for logs and debugging.

A StreamFilter is not an Element and is never added to a pipeline with add(). It is always attached between stages with Pipeline.add_stream_filter(name, filter), which internally creates a capsfilter element and applies the filter’s caps to it. The base StreamFilter class itself only exposes a raw-caps-string constructor and to_string(); the fluent per-domain setters (.format(), .resolution(), .framerate(), …) live on the typed subclasses such as VideoFilter. Usage Typed subclass

from qimsdk import Pipeline, VideoFilter

pipeline = (
    Pipeline("video-pipeline")
    .add("videotestsrc", "src", "pattern", "ball")
    .add_stream_filter("filter", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
    .add("waylandsink", "sink")
)

pipeline.execute()

Raw caps string

from qimsdk import Pipeline, StreamFilter

pipeline = (
    Pipeline("video-pipeline-streamfilter-string")
    .add("videotestsrc", "src", "pattern", "ball")
    .add_stream_filter("filter", StreamFilter(
        "video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1"))
    .add("waylandsink", "sink")
)

pipeline.execute()

APIs

Method	Description
`StreamFilter(caps: str)`	Creates a filter from a raw caps or media-type string, e.g. `"video/x-raw,format=NV12"`.
`StreamFilter.to_string() -> str`	Serializes the current filter expression to a caps string, for logging/debugging.

VideoFilter

Purpose Configures a raw video stream (YUV, RGB, Bayer) between a camera source and any downstream processing or display element. Supports format, resolution, framerate, colorimetry, interlace mode, and pixel aspect ratio. Key Responsibilities

Produce caps with standard raw video attributes: format, resolution, framerate, colorimetry, interlace mode, pixel aspect ratio, etc.

APIs

Method	Description
`format(fmt: str \| VideoFormat)`	Sets the stream’s pixel format, for example: `NV12`, `YUY2`, `RGB`, `RGBA`, etc. Determines how pixel data is laid out in memory; essential for hardware compatibility and color-conversion paths.
`resolution(w: int, h: int)`	Specifies the frame dimensions in pixels.
`framerate(num: int \| float, den: int = 1)`	Sets the frame rate as a fraction, for example: `30/1`, `30000/1001` frames-per-second. You can pass only `num` if you do not need a fractional frame rate, like `30.0` fps.
`colorimetry(value: str)`	Defines the colorimetry standard, for example: `bt601`, `bt709`, `bt2020`.
`range(value: str)`	Selects the quantization range, for example: `full` or `limited`.
`interlace(mode: str)`	Sets the interlacing mode, for example: `progressive`, `interleaved`, etc. Progressive is standard for modern pipelines; interlaced is rare but still used in some camera/legacy systems.
`pixel_aspect_ratio(num: int, den: int = 1)`	Specifies the pixel aspect ratio, also called PAR. Normally `1/1`; used for anamorphic or non-square-pixel formats.
`add(expr: str)`	Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API.

Usage We can achieve the same results in two different ways: Typed (enum) API - if you want IDE autocompletion and guardrails against typos. Ideal for most application code.

from qimsdk import VideoFilter, VideoFormat

v1 = (
    VideoFilter()
    .format(VideoFormat.NV12)
    .resolution(1920, 1080)
    .framerate(30)
    .colorimetry("bt709")
    .range("limited")
    .interlace("progressive")
)

String API - if you need quick iteration or to pass values not (yet) covered by the enum — e.g., a freshly added GStreamer value on a target platform.

from qimsdk import VideoFilter

v2 = (
    VideoFilter()
    .format("NV12")
    .resolution(1920, 1080)
    .framerate(30)
    .colorimetry("bt709")
    .range("limited")
    .interlace("progressive")
)

Raw caps string - very useful for setting an arbitrary stream (caps) filter, or if you want to lift an existing caps line from logs, gst-launch, or a pad template and drop it as-is — great for debugging, prototyping, or reproducing pipelines exactly.

from qimsdk import StreamFilter

v3 = StreamFilter("video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1,colorimetry=bt709,range=limited")

H264Filter

Purpose Configures an encoded H.264 stream between an encoder and a downstream parser or muxer. Supports resolution, framerate, stream-format, alignment, profile, and level. Key Responsibilities

Specify resolution, framerate, stream-format, alignment, profile, level.
Insert codec_data buffers.
Attach arbitrary name/value string properties.

APIs

Method	Description
`resolution(w: int, h: int)`	Specifies the frame dimensions in pixels.
`framerate(num: int \| float, den: int = 1)`	Sets the frame rate as a fraction.
`profile(profile: str)`	Sets the H.264 profile caps field, for example `baseline`, `main`, `high`.
`level(level: str)`	Sets the H.264 level caps field.
`stream_format(fmt: str)`	Sets the H.264 stream-format caps field, for example `byte-stream`, `avc`.
`alignment(value: str)`	Sets the H.264 alignment caps field, for example `au`, `nal`.
`codec_data(value: str)`	Sets the H.264 `codec_data` caps field.
`set(key: str, value: str)`	Shorthand for `add(f"{key}={value}")`.
`add(expr: str)`	Appends an arbitrary custom caps field using a `key=value` expression.

Usage

from qimsdk import Pipeline, H264Filter

h264f = (
    H264Filter()
    .resolution(1920, 1080)
    .framerate(30)
    .profile("baseline")
    .stream_format("byte-stream")
    .alignment("au")
)

pipeline = Pipeline("h264-pipeline").add_stream_filter("h264f", h264f)

TensorFilter

Purpose Configures an ML tensor stream with explicit data type and shape (e.g., UINT8, NHWC). Not needed in a full inference pipeline — when preprocess, inference, and postprocess elements are all present, tensor parameters are negotiated automatically. Use TensorFilter when:

The preprocessing stage does not have model dimension information
You need exact tensor data output from the inference plugin

Key Responsibilities

Declare element type (e.g., UINT8, FLOAT32) for tensors.
Specify shape (dimensions). Supports:
- Single tensor: a flat list of ints (e.g., 1, 520, 520, 3).
- Multiple tensors: a list of lists (e.g., [[1,3,H,W], [1,32]]) when needed.

APIs

Method	Description
`type(t: TensorFilter.Type \| str)`	Specify tensor data type, for example: `UINT8`, `FLOAT32`.
`dimensions(*dims)`	Specify tensor dimensions. Pass ints directly for a single tensor, or one or more `list`/`tuple` arguments for multiple tensors.
`add(expr: str)`	Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API.

Usage We can achieve the same results in two different ways: Typed (enum) API - if you want IDE autocompletion and guardrails against typos. Ideal for most application code.

from qimsdk import TensorFilter

t1 = (
    TensorFilter()
    .type(TensorFilter.Type.UINT8)
    .dimensions(1, 520, 520, 3)       # NHWC, for example
)

String API - if you need quick iteration or to pass values not (yet) covered by the enum.

from qimsdk import TensorFilter

t2 = (
    TensorFilter()
    .type("FLOAT32")
    .dimensions(1, 3, 640, 640)       # NCHW, for example
)

Raw caps string - very useful for setting an arbitrary stream (caps) filter, or if you want to lift an existing caps line from logs, gst-launch, or a pad template and drop it as-is.

from qimsdk import StreamFilter

t3 = StreamFilter("neural-network/tensors,type=UINT8,dimensions=<<1,520,520,3>>")

TextFilter

Purpose Configures a raw text stream (text/x-raw) for pipelines that carry textual data such as metadata, subtitles, or logs. Minimal by design — attach only the fields you need using .add("key=value"). Key Responsibilities

Provide a lightweight caps descriptor for text (text/x-raw) without enforcing a schema.
Let callers attach arbitrary properties (e.g., encoding/charset, language, packetization hints) using .add("key=value").

APIs

Method	Description
`add(expr: str)`	Appends an arbitrary custom caps field using a `"key=value"` expression. Escape-hatch for advanced/experimental fields not covered by the typed API.

Usage

from qimsdk import TextFilter

txt = TextFilter().add("charset=utf8")

ImageFilter

Overview Used exclusively with CamSrc (qtiqmmfsrc) to enable and link its dedicated image capture pad. Without ImageFilter, the image pad is not requested, not activated, and produces no buffers. It is a structural enabler, not a processing stage — it is not intended for general image processing or frame extraction from video sources. ImageFilter is used to:

Enable the image output pad exposed by CamSrc (qtiqmmfsrc)
Attach that pad to the pipeline so the source can activate image capture
Provide a concrete sink target required by GStreamer for pad activation

It is not intended for:

Generic image processing
Frame extraction from arbitrary video sources
Replacement of VideoFilter or AppSink

Relationship to CamSrc (qtiqmmfsrc)

CamSrc (qtiqmmfsrc) exposes a dedicated image pad that remains inactive unless linked.
ImageFilter explicitly exists to satisfy this requirement.
Without ImageFilter, the image pad:
- Will not be requested
- Will not be activated
- Will not produce buffers

This makes ImageFilter a structural enabler, not a processing stage. Usage

from qimsdk import ImageFilter

imagefilter1 = (
    ImageFilter()
    .format("JPEG")
    .resolution(3840, 2160)
)

AudioFilter

Purpose Configures a raw PCM audio stream with sample format, channel count, sample rate, and layout. Use this to ensure audio converters and sinks negotiate exactly the format your hardware supports rather than inserting an implicit resampler. Key Responsibilities

Set audio sample format (e.g., S16LE, F32LE) to match HW/SW capabilities.
Define channels and rate for correct resampling/mixing.
Choose layout (interleaved vs non-interleaved) to align with downstream expectations.

APIs

Method	Description
`format(fmt: str \| AudioFormat)`	Specifies the audio sample format, e.g. `S16LE`, `F32LE`.
`channels(n: int)`	Specifies the number of audio channels.
`rate(hz: int)`	Specifies the audio sample rate in Hz.
`layout(layout: str \| AudioLayout)`	Specifies the audio layout, `interleaved` or `non-interleaved`.
`add(expr: str)`	Appends an arbitrary custom caps field using a `"key=value"` expression. Escape-hatch for advanced/experimental fields not covered by the typed API.

Usage We can achieve the same results in two different ways: Typed (enum) API - if you want IDE autocompletion and guardrails against typos. Ideal for most application code.

from qimsdk import AudioFilter, AudioFormat, AudioLayout

a1 = (
    AudioFilter()
    .format(AudioFormat.S16LE)
    .rate(48000)
    .channels(2)
    .layout(AudioLayout.INTERLEAVED)
)

String API - if you need quick iteration or to pass values not (yet) covered by the enums.

from qimsdk import AudioFilter

a2 = (
    AudioFilter()
    .format("F32LE")
    .rate(44100)
    .channels(1)
    .layout("non-interleaved")
)

Raw caps string - very useful for setting an arbitrary stream (caps) filter, or if you want to lift an existing caps line from logs, gst-launch, or a pad template and drop it as-is.

from qimsdk import StreamFilter

a3 = StreamFilter("audio/x-raw,format=S16LE,rate=48000,channels=2,layout=interleaved")

Logging

The SDK can either simplify GStreamer’s log stream into a compact, uniform format or step aside and let GStreamer log raw, with GST_DEBUG fully in control. Both modes, the verbosity levels, the integration with Python’s logging module, and the ordering constraint on configuring them are covered in Logging and Diagnostics.

QIM SDK Nodes

All nodes are executed by GStreamer elements under the hood. Most elements are handled through the Element base class. However, elements that provide additional functionality — such as application buffer interfaces in appsink and appsrc, or image-capture signals in camera sources like qtiqmmfsrc — have a specialized Python wrapper class. If you do not need these extra capabilities (for example, image capture), you can still use these elements via the generic Element base class without any limitations for standard data processing. Below is a table listing all supported elements and the corresponding QIM SDK Python wrapper classes used to work with them. Sources

GStreamer	QIM SDK Python Class
`qtiqmmfsrc`	`CamSrc` (also usable as `Element`)
`v4l2src`	`Element`
`filesrc`	`Element`
`multifilesrc`	`Element`
`appsrc`	`AppSrc`
`appsink`	`AppSink`
`qtisocketsrc`	`Element`

GStreamer	QIM SDK Python Class
`qtimlaconverter`	`Element`
`qtimlvconverter`	`MLVConverter` (also usable as `Element`)
`qtimlqnn`	`Element`
`qtimlsnpe`	`Element`
`qtimltflite`	`Element`
`qtimlpostprocess`	`MLPostprocess` (also usable as `Element`)
`qtimlvideoonnxbin`	`MLVideoONNXBin` (also usable as `Element`)
`qtimlvideoqnnbin`	`MLVideoQNNBin` (also usable as `Element`)
`qtimlvideosnpebin`	`MLVideoSNPEBin` (also usable as `Element`)
`qtimlvideotflitebin`	`MLVideoTFLiteBin` (also usable as `Element`)
`qtimlaic`	`Element`
`qtimetamux`	`Element`
`qtimlmetaextractor`	`Element`
`qtimlmetaparser`	`Element`
`qtimldemux`	`Element`
`qtimetatransform`	`Element`
`mlonnx`	`Element`

Multimedia

GStreamer	QIM SDK Python Class
`qtiobjtracker`	`Element`
`qtismartvencbin`	`Element`
`qtivoverlay`	`Element`
`qtivsplit`	`Element`
`qtivcomposer`	`Element`
`qtibatch`	`Element`
`qticamreproc`	`Element`
`qtic2adec`	`Element`
`qtic2aenc`	`Element`
`qtic2vdec`	`Element`
`qtic2venc`	`Element`
`qtijpegenc`	`Element`
`qtivtransform`	`Element`
`qticvimgpyramid`	`Element`
`qticvoptclflow`	`Element`
`qtidfs`	`Element`
`v4l2h264dec`	`Element`
`v4l2h264enc`	`Element`
`v4l2h265dec`	`Element`
`v4l2h265enc`	`Element`
`tee`	`Element`
`h264parse`	`Element`
`rtpptdemux`	`Element`
`rtph264depay`	`Element`
`queue`	`Element`
`qtdemux`	`Element`
`tsdemux`	`Element`

Common Application Design

Create pipeline with config file

The SDK supports fully declarative pipeline construction using YAML. A YAML pipeline describes elements and links between them, which are translated at runtime into a GStreamer pipeline. TopLevel Structure A YAML pipeline file has a single top-level key:

pipeline:
  elements: [...]
  links: [...]

Elements Section elements is an ordered list of pipeline elements. Each element entry defines:

The GStreamer element type
A unique name used for linking
Optional properties
Optional filter configuration

Common Fields

- type: <element-type>
  name: <unique-name>
  <property>: <value>

type: GStreamer element factory name, or filter
name: Unique identifier within the pipeline
properties: Any remaining key/value pairs are passed directly to the element as GStreamer properties

Filter Elements Filters are declared using type: filter and are further specialized by their configuration block. Stream Filter

- type: capsfilter
  name: raw_caps
  caps: "video/x-raw,format=NV12,width=1280,height=720,framerate=30/1"

Video Filter

- type: filter
  name: vf
  video:
    format: NV12
    width: 1920
    height: 1080
    framerate: 30

Links Section links defines explicit connection paths between elements. Each entry is an ordered list of element names. Elements are linked sequentially from left to right.

links:
  - [src, demux, parse, decoder, vf, split]

Example yaml files YAML config file

pipeline:
  elements:
    - type: qtiqmmfsrc
      name: source
      camera: "0"

    - type: filter
      name: filter
      video:
        format: NV12
        width: 1920
        height: 1080
        framerate: 30.0

    - type: tee
      name: split

    - type: qtimlvconverter
      name: preprocessing

    - type: qtimltflite
      name: inferencing
      delegate: gpu
      model: /etc/models/model.tflite

    - type: qtimlpostprocess
      name: postprocessing

    - type: qtimetamux
      name: mlmuxer

    - type: qtivoverlay
      name: overlay

    - type: waylandsink
      name: display
      sync: true
      async: true
      fullscreen: true

  links:
    - [source, filter, split, preprocessing, inferencing, postprocessing, mlmuxer, overlay, display]
    - [split, mlmuxer]

Multi-Model Daisy Chain

Below is shown a logical diagram of an AI daisy chain pipeline with sequential and parallel inferencing. The same could be handled by a straight sequential pipeline of inference bins. But we must set manually the source for every inference bin.

Try me

Several .add(...) calls in this example had missing/mismatched quotes in the source PDF (e.g. a stray closing paren instead of a closing quote, or a missing leading quote). These have been corrected below where unambiguous. One inline comment truncated at the page margin was completed using the identical sentence found in the prose above.

Element ml_car("qtimlvideotflitebin", "car")
  .add("performance-profile", "high-performance")
  .add("inference-delegate", "dsp")
  .add("inference-model", "/opt/car_detect.dlc")
  .add("inference-layers", "< /model.22/Mul_2, /model.22/Sigmoid >")
  .add("postprocess-threshold", 60.0)
  .add("postprocess-results", 4)
  .add("postprocess-module", "yolov8")
  .add("postprocess-labels", "/opt/car_detect_model.labels");
  // The same could be handle by straight sequential pipeline of inference bins. But we must set manually source to every inference bin.

Element ml_plate("qtimlvideotflitebin", "plated")
  .add("source", 1)
  .add("image-disposition", "centre")
  .add("performance-profile", "high-performance")
  .add("inference-delegate", "dsp")
  .add("inference-model", "/opt/plate_detect.dlc")
  .add("inference-layers", "< /model.22/Mul_2, /model.22/Sigmoid >")
  .add("postprocess-threshold", 40.0)
  .add("postprocess-results", 1)
  .add("postprocess-module", "yolov8")
  .add("postprocess-labels", "/opt/plate_detect_model.labels");

Element ml_anpr("qtimlvideotflitebin", "anpr")
  .add("source", 2)
  .add("image-disposition", "centre")
  .add("inference-delegate", "dsp")
  .add("inference-model", "/opt/anpr.dlc")
  .add("inference-layers", "< /decode_head2/classifier/Conv, /decode_head/classifier/Conv >")
  .add("postprocess-threshold", 15.0)
  .add("postprocess-results", 1)
  .add("postprocess-module", "anpr")
  .add("postprocess-labels", "/opt/anpr_model.labels");

Element ml_siglip("qtimlvideotflitebin", "siglip")
  .add("source", 1)
  .add("image-disposition", "centre");
  .add("inference-delegate", "dsp");
  .add("inference-model", "/opt/car_make_model.dlc");
  .add("inference-layers", "< /classifier/Gemm >");
  .add("postprocess-threshold", 15.0);
  .add("postprocess-results", 1);
  .add("postprocess-module", "mobilenet");
  .add("postprocess-labels", "/opt/car_make_model.labels");

Pipeline pipeline("SmartVehicleIdentification");
pipeline.add("camsrc", "source", "camera", "0")
            .add_stream_filter("filter", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
            .add(ml_car)
            .add(ml_plate)
            .add(ml_anpr)
            .add(ml_siglip)
            .add("qtivoverlay", "overlay")
            .add("waylandsink", "display", "sync", "true", "async", "true", "fullscreen")
            .execute();

Custom Postprocessing for YOLOv8 (Detection)

QIM SDK implements AI model post-processing through dedicated modules that take tensors as input and produce predictions. All remaining complexity — such as batching, daisy-chaining, image mask support — is encapsulated within the post-processing plugin. This makes the post-processing module API an ideal integration point for application-specific post-processing logic. As a result, applications do not need to handle batching, chaining, image masks, or ML metadata themselves. We will implement a specialized post-processing module to manage bindings with the application.

Custom postprocessing callback is attached to a qtimlpostprocess element via MLPostprocess.set_handler(...). Typical pipeline segment:

Preprocess frame/tensors (qtimlvconverter)
Run ML inference (qtimltflite)
Convert raw tensors in custom callback (qtimlpostprocess)

Supported output types

ImageClassifications
ObjectDetections
Poses
DepthMaps
Segmentations
Tensors

Minimal usage

from qimsdk import MLPostprocess, ObjectDetections

def my_callback(mlframe, mlparams, out: ObjectDetections):
    ...
    return True

postprocessing = MLPostprocess("postprocessing")
postprocessing.set_handler(my_callback)

pipeline = (
    Pipeline("my-pipeline")
    .add("qtimltflite", "inferencing", ...)
    .add(postprocessing)
    .add_stream_filter("mlf", TextFilter())
    ...
)

Input and output explained mlframe (read-only model tensors) Passed to every callback. Its tensors list holds one entry per model output tensor, each exposing:

type: tensor element type
dimensions: tensor shape
data: raw tensor memory (as a memoryview)
size: tensor bytes

Common access pattern:

t = mlframe.tensors[0]
data = t.data

mlparams (runtime metadata and context) mlparams carries key/value fields provided by the pipeline and preprocessing/postprocess context. Commonly used keys:

input-tensor-width
input-tensor-height
input-tensor-region (an (x, y, width, height) region)

Example:

source_width = mlparams.get("input-tensor-width")
source_height = mlparams.get("input-tensor-height")
region = mlparams.get("input-tensor-region")

Callback signatures

Callbacks annotated with ImageClassifications, AudioClassifications, ObjectDetections, Poses, DepthMaps, or Segmentations fill the corresponding output metadata container.
A callback annotated with Tensors fills an output Tensors object for downstream tensor stages.
mlframe is read-only and must not be modified.

Classification callback

def classification_callback(mlframe, mlparams, out: ImageClassifications):
    ...
    return True

Output: append classification items to out. Field meanings:

name: class label text
confidence: prediction confidence
color: optional UI/overlay color
extra: optional metadata map

Object detection callback

def detection_callback(mlframe, mlparams, out: ObjectDetections):
    ...
    return True

Output: append detection items to out. Field meanings:

left, top, right, bottom: normalized box coordinates
name, confidence, color: class info
landmarks: optional keypoints
extra: optional metadata map

Implemented example includes confidence thresholding, region coordinate transform, and NMS. Segmentation callback

def segmentation_callback(mlframe, mlparams, segmentations: Segmentations):
    ...
    return True

Input parameters

mlframe: read-only model output tensors.
mlparams: runtime metadata (for example: input-tensor-width, input-tensor-height, input-tensor-region).
segmentations: output container to fill.

Output contract

labels: per-pixel semantic labels.
colors: per-pixel colors; index-aligned with labels.
n_rows, n_columns: segmentation map dimensions.
extra: optional metadata fields.

Pose estimation callback

def pose_callback(mlframe, mlparams, out: Poses):
    ...
    return True

Output: append pose items to out. Field meanings:

keypoints: list of keypoints (name, x, y, confidence, color)
links: list of keypoint links describing skeleton connections
name, confidence, extra: pose-level metadata

Depth estimation callback

def depth_callback(mlframe, mlparams, out: DepthMaps):
    ...
    return True

Output: append depth map items to out. Field meanings:

values: per-pixel depth values
colors: per-pixel pseudo-color values
n_rows, n_columns: depth map shape
extra: optional metadata map

Tensor-to-tensor callback

def tensor_callback(mlframe, mlparams, out: Tensors):
    ...
    return True

out is owned and writable; callback typically:

validates input tensors from mlframe
copies/selects required tensors into out
updates output tensor dimensions/order if needed

Semantic callbacks (classification, detection, pose, depth, segmentation)

def callback(mlframe, mlparams, output: OutputType) -> bool:
    ...

mlframe: input tensors from inference (read-only)
mlparams: runtime metadata (input sizes, region, etc.)
output: semantic objects to be filled by callback
return True on valid output
return False on validation/decode failure

Tensor callback

def callback(mlframe, mlparams, output: Tensors) -> bool:
    ...

mlframe: input tensors from inference (read-only)
mlparams: runtime metadata/context
output: writable output tensors for downstream tensor stage
return True on valid output
return False on validation/decode failure

Copy-ready callback skeletons Classification

def classification_callback(mlframe, mlparams, out: ImageClassifications):
    # decode tensors -> classes
    return True

Detection

def detection_callback(mlframe, mlparams, out: ObjectDetections):
    # decode tensors -> detections
    return True

Pose estimation

def pose_callback(mlframe, mlparams, out: Poses):
    # decode tensors -> poses
    return True

Depth estimation

def depth_callback(mlframe, mlparams, out: DepthMaps):
    # decode tensors -> depthmaps
    return True

Segmentation

def segmentation_callback(mlframe, mlparams, out: Segmentations):
    # decode tensors -> segmentations
    return True

Tensors

def tensor_callback(mlframe, mlparams, out: Tensors):
    # map/copy selected tensors from mlframe -> out
    return True

Try me

Check application source code on GitHub: qimsdk_ref_external_postprocess_detection_yolov8.pyPre-built application on device: /usr/bin/qimsdk_ref_external_postprocess_detection_yolov8.py

Download Required Files

File	Save as
Model	yolov8_det_quantized.tflite
yolov8.json	yolov8.json

If the downloaded model file is a .zip archive, extract it on your host machine before copying: unzip filename.zipSome AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn’t always sufficient.

Copy Files to Device

Also copy an MP4 (H.264) file for the pipeline’s video input, or update the pipeline’s location property to point to your own file.

# Replace <user> and <device-ip> with your device credentials.
# ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
scp video.mp4                   <user>@<device-ip>:~/Downloads/qimsdk_samples/media/

Run the application

python3 /usr/bin/qimsdk_ref_external_postprocess_detection_yolov8.py

The application runs inference on the video file and uses the custom, application-defined postprocessing callback to decode detections, which are then overlaid and displayed fullscreen.To stop the application, press CTRL + C.

Getting Started

Examples

Design

Plugins

Pipeline SDK Design

APIs

Pipeline

Element

AppSrc

AppSink

CamSrc

MLVConverter

MLPostprocess

ML inference bins

Buffer

StreamFilter

VideoFilter

H264Filter

TensorFilter

TextFilter

ImageFilter

AudioFilter

Logging

QIM SDK Nodes

Common Application Design

Create pipeline with config file

Multi-Model Daisy Chain

Custom Postprocessing for YOLOv8 (Detection)

Download Required Files

​Pipeline SDK Design

​APIs

​Pipeline

​Element

​AppSrc

​AppSink

​CamSrc

​MLVConverter

​MLPostprocess

​ML inference bins

​Buffer

​StreamFilter

​VideoFilter

​H264Filter

​TensorFilter

​TextFilter

​ImageFilter

​AudioFilter

​Logging

​QIM SDK Nodes

​Common Application Design

​Create pipeline with config file

​Multi-Model Daisy Chain

​Custom Postprocessing for YOLOv8 (Detection)

​Download Required Files

Pipeline SDK Design

APIs

Pipeline

Element

AppSrc

AppSink

CamSrc

MLVConverter

MLPostprocess

ML inference bins

Buffer

StreamFilter

VideoFilter

H264Filter

TensorFilter

TextFilter

ImageFilter

AudioFilter

Logging

QIM SDK Nodes

Common Application Design

Create pipeline with config file

Multi-Model Daisy Chain

Custom Postprocessing for YOLOv8 (Detection)

Download Required Files