SDK Architecture and API Reference

Pipeline SDK Design

Overview This SDK delivers a streamlined, modern interfaces, engineered to facilitate the rapid development, execution, and management of sophisticated multimedia pipelines. By implementing a high-level abstraction layer, the SDK encapsulates complex underlying GStreamer based mechanisms — including element negotiation, signal management, event loops, and state transitions — allowing engineers to prioritize core application logic over boilerplate framework intricacies. Key Capabilities The wrapper empowers developers to architect advanced multimedia and AI-enhanced workflows with precision and efficiency. It supports the seamless integration of diverse components, such as video capture sources, signal processing nodes, machine learning inference engines, graphical overlays, and rendering outputs. Uniquely designed to decouple implementation from the underlying framework, the SDK eliminates the need for deep GStreamer expertise or compile-time dependencies on GStreamer headers, ensuring a clean, robust, and accessible development experience.

APIs

Public APIs:

Pipeline — build & control pipelines fluently.
Element — generic element wrapper with variadic property setting, and state control. [it can be instantiated at create time as well as at runtime to update the pipeline.] Some element has special functions which can be controlled by dedicated elements types. Those elements derived from Element can be use as Element if special functions are not used.
- AppSrc — app-facing source for zero-copy buffer pull. [Providing the buffer from application to pipeline using standard lambdas, push APIs, EOS etc]
- AppSink — app-facing sink for zero-copy buffer push [Providing the buffer from pipeline to application with zero copy]
- CamSrc - Build-in camera element which expose API for capture images [provides functionality to take snapshots at runtime]
- MLVConverter - ML video converter (qtimlvconverter). Expose API for custom pre-processing implementation at application utilizing standard lambdas.
- MLPostprocess - ML postprocess element (qtimlpostprocess). Expose API for custom post-processing implementation at application utilizing standard lambdas. The callback is typed per ML task (classification, object detection, pose estimation, depth estimation, segmentation, raw tensors).
- MLVideoONNXBin - ONNX inference bin (qtimlvideoonnxbin). Expose API for custom pre-processing and post-processing implementation at application utilizing standard lambdas.
- MLVideoQNNBin - QNN inference bin (qtimlvideoqnnbin). Expose API for custom pre-processing and post-processing implementation at application utilizing standard lambdas.
- MLVideoSNPEBin - SNPE inference bin (qtimlvideosnpebin). Expose API for custom pre-processing and post-processing implementation at application utilizing standard lambdas.
- MLVideoTFLiteBin - TFLite inference bin (qtimlvideotflitebin). Expose API for custom pre-processing and post-processing implementation at application utilizing standard lambdas.
Buffer — opaque buffer that bridges Native Buffer with timestamp helpers and zero-copy paths.
StreamFilter and derived concrete stream helpers: VideoFilter (all RAW formats - YUV, RGB, bayer), H264Filter, TensorFilter, TextFilter, AudioFilter, ImageFilter. these filters are used to set characteristics of the streams.
Logging controls — SetImsdkLogLevel() and SetImsdkGstLogMode() select the SDK’s log verbosity and whether GStreamer’s output is simplified or passed through raw. See Logging and Diagnostics.

Reference flows

Pipeline

Purpose A convenient, exception-safe wrapper around a GstPipeline that:

Builds pipelines using element factory names or pre-constructed wrapper elements.
Manages pipeline lifecycle and execution state.
Performs delayed linking for dynamic/request-pad elements automatically via pad-added.

Key Responsibilities

Create and own the underlying GstPipeline instance.
Add elements, assign properties, and insert caps filters.
Link elements (immediately when possible, deferred otherwise).
Manage the full pipeline lifecycle (prepare, start, wait, stop).
Install bus watches to handle ERROR, EOS, and state transitions.
Integrate with runtime shutdown notifications.

A minimal pipeline Add the elements in the order data flows through them, then run:

Pipeline pipeline("my-pipeline");

pipeline
    .add("filesrc", "src", "location", "/home/user/media/video.mp4")
    .add("qtdemux", "demux")
    .add("h264parse", "parse")
    .add("v4l2h264dec", "decoder", "output-io-mode", 4, "capture-io-mode", 4)
    .add_stream_filter("vf", VideoFilter().format("NV12"))
    .add("waylandsink", "display", "fullscreen", true)
    .execute();

execute() starts the pipeline, blocks until EOS/error/termination, and tears it down.

Adjacent elements are linked in add order. In the example above src → demux → parse → decoder → vf → display are linked simply because that is the order of the add() calls, so a linear graph needs no explicit linking. Use link() when the graph is not linear — a tee splitting into a display branch and an inference branch, or a qtimetamux rejoining them.

qtdemux is an example of the delayed linking described above: its src pads appear only once the container has been parsed, so the demux → parse link is completed automatically when the pad shows up. No pad-added handling is needed in application code.

Building the pipeline

Method	Description
`Pipeline(const std::string &name)`	Creates an empty pipeline. `name` is the `GstPipeline` name, used in log messages.
`Pipeline(const std::string &name, const std::string &config)`	Builds a complete pipeline from a YAML description. `config` is the YAML content, not a path. See Describing a pipeline in YAML.
`Pipeline &add(String factory, String name, ...)`	Creates an element from the factory name and adds it under a unique instance name. Trailing arguments are property key/value pairs, applied immediately.
`Pipeline &add(const Element &element)`	Adds an externally constructed element. Required for the specialized wrappers (`AppSrc`, `AppSink`, `CamSrc`, `MLVConverter`, `MLPostprocess`, the ML bins) so callbacks can be attached before the element joins the pipeline.
`Pipeline &add_stream_filter(String name, const StreamFilter &filter)`	Inserts a caps filter between the previous and next element. When neighbouring elements have several formats in common, this fixates the one you want instead of leaving the choice to caps negotiation. See StreamFilter.
`Pipeline &link(names...)`	Links the named elements in the given order. Needed only for non-linear graphs.

The optional key/value pairs passed to add() are the element’s properties, and they are validated as soon as the element is added. Setting a property the element does not have throws std::invalid_argument naming the property, the element instance, and the element type — so a typo surfaces immediately at that add() call instead of being silently ignored:

// "fulscreen" is not a waylandsink property -> throws std::invalid_argument
pipeline.add("waylandsink", "display", "fulscreen", true);

Branching with explicit links Add order cannot express a branch, so name the chains explicitly. Element instance names are the identifiers.

pipeline
    .add("filesrc", "src", "location", path)
    .add("qtdemux", "demux")
    .add("h264parse", "parse")
    .add("v4l2h264dec", "decoder", "output-io-mode", 4, "capture-io-mode", 4)
    .add_stream_filter("vf", VideoFilter().format("NV12"))
    .add("tee", "split")
    .add("queue", "q1")
    .add("qtimlvconverter", "preprocessing")
    .add("qtimltflite", "inferencing", "model", model_path)
    .add("qtimlpostprocess", "postprocessing", "module", "yolov8")
    .add_stream_filter("mlf", TextFilter())
    .add("qtimetamux", "mlmuxer")
    .add("qtivoverlay", "overlay")
    .add("waylandsink", "display", "fullscreen", true)
    // trunk, up to the tee
    .link("src", "demux", "parse", "decoder", "vf", "split")
    // branch 1: raw video to the metadata muxer
    .link("split", "mlmuxer")
    // branch 2: inference path, rejoining at the muxer
    .link("split", "q1", "preprocessing", "inferencing", "postprocessing",
          "mlf", "mlmuxer", "overlay", "display");

Elements whose src pads are SOMETIMES or REQUEST — qtdemux, tsdemux, tee, qtivsplit — cannot be linked at construction time, because the pads do not exist yet. The pipeline detects this from the pad templates, defers the link, and completes it from a pad-added handler it installs itself. Request pads on the sink side (for example qtimetamux, qtivcomposer) are requested and released automatically. No pad-added callback is needed in application code.

A link between two elements that both have only static pads and no common caps fails immediately with std::runtime_error naming both elements. A deferred link that never resolves surfaces at run time instead; on error the pipeline logs which pads stayed unlinked.

Running the pipeline A pipeline is always in one of three states, and the lifecycle methods are the transitions between them:

State	Meaning
`NULL`	No resources allocated. Initial state, and the state after `stop()`. Devices are closed, buffer pools released.
`PAUSED`	Elements are allocated and caps are negotiated between linked pads, but the clock is not running and no data flows. Sinks have pre-rolled. This is the state in which elements can be added, removed, or relinked.
`PLAYING`	The clock is running and buffers flow through the graph.

execute() performs the whole NULL → PLAYING → NULL cycle, so most applications never call the individual transitions. prepare(), activate(), and deactivate() — the transitions through PAUSED shown above — are covered in Advanced state control.

Method	Description
`void execute()`	`start()`, `wait()`, `stop()` in sequence. The normal entry point.
`Pipeline &start()`	Sets the pipeline to `PLAYING`. Non-blocking: the state change is asynchronous and this returns without waiting for it. Throws `std::runtime_error` if the transition fails.
`Pipeline &wait()`	Blocks until the pipeline is done: `EOS` on the bus, `ERROR` on the bus, `stop()` from another thread, or `SIGINT`/`SIGTERM`.
`Pipeline &stop()`	Sets the pipeline to `NULL` and releases resources. Blocks until teardown completes.
`Pipeline &eos(bool enabled)`	When enabled, `stop()` first sends an end-of-stream event through the pipeline and waits for it to come back from the sinks, so that data already inside the pipeline is processed before teardown. Off by default. See Draining the pipeline on shutdown.

start() is non-blocking, so it must be paired with wait() (or your own blocking logic). A main() that calls start() and returns destroys the pipeline before buffers flow.

To run the pipeline while the application does other work, split execute() apart:

pipeline.start();       // returns immediately

// ... application work, runtime property changes, snapshots ...

pipeline.wait();        // block until EOS / error / Ctrl+C
pipeline.stop();

Draining the pipeline on shutdown By default stop() tears the pipeline down as fast as it can. Every element goes to NULL immediately and whatever buffers are still in flight — queued between elements, held inside an encoder, not yet written by a sink — are discarded. This is usually what you want: the pipeline stops without delay. Some pipelines must not lose those buffers, and for them eos(true) changes the shutdown sequence. Instead of stopping straight away, stop() sends an end-of-stream event into the pipeline. The event travels downstream element by element; each element finishes processing the data it has already accepted, passes the event on, and the sinks report it back on the bus. Only once that end-of-stream notification comes back does stop() tear the pipeline down. Nothing that had already entered the pipeline is dropped. The trade-off is shutdown latency: stop() now blocks until the last buffer has made it through the whole graph, which takes as long as the remaining data needs. Use eos(true) when the tail of the stream is part of the result:

Muxed recordings. Containers such as MP4 and MKV are written by a muxer (mp4mux, matroskamux) that writes its index and header metadata — the moov atom, in MP4’s case — only on end-of-stream. Cut it off early and the file is left incomplete, often unplayable rather than merely short.
Encoders with buffered frames. Encoders keep several frames in flight for B-frames, lookahead, and similar optimizations. End-of-stream is what makes the encoder flush those frames downstream; without it the tail of the recording is simply missing.
Any sink where losing the final buffered data is unacceptable — file sinks, muxed outputs, and applications that expect to receive every frame through AppSink.

Not every sink needs draining, though. A live network stream can usually just be cut off, and a receiver will not care that the last few buffers never arrived. The question is not whether a sink writes somewhere, but whether anything downstream depends on the data still sitting in the pipeline. Leave eos() off (the default) when the newest data is all that matters and stopping promptly is preferable — live preview to a display, a camera feed on screen, or writing self-contained files where each buffer is already complete on its own (JPEG frames through multifilesink, for example, which needs no finalization).

Pipeline pipeline("cam-encoder-pipeline");
pipeline
    .add("qtiqmmfsrc", "source")
    .add_stream_filter("vf", VideoFilter().format("NV12")
                                          .resolution(1920, 1080)
                                          .framerate(30))
    .add("v4l2h264enc", "encoder",
         "output-io-mode", "dmabuf-import", "capture-io-mode", "dmabuf")
    .add("h264parse", "parser")
    .add("mp4mux", "muxer")
    .add("filesink", "sink", "location", HOME_PATH + "/media/output.mp4")
    .eos(true)          // drain, so mp4mux can finalize the container
    .execute();

If the pipeline has already reached end-of-stream on its own — a file source that ran out of data, for instance — stop() detects it and tears down immediately instead of sending a second event and waiting again. Enabling eos(true) therefore costs nothing for pipelines that end naturally; it only matters when you stop a pipeline that is still running.

Advanced state control For reconfiguring a loaded pipeline — adding, removing, or relinking elements without returning to NULL.

Method	Description
`Pipeline &prepare()`	`NULL` → `PAUSED`. Allocates resources and negotiates caps without running the clock. Blocking.
`Pipeline &activate()`	`PAUSED` → `PLAYING`.
`Pipeline &deactivate()`	`PLAYING` → `PAUSED`. Resources stay allocated.

Accessing elements at run time

Method	Description
`Element get(String name)`	Returns a wrapper for the named element, for runtime property changes and per-pad configuration. Throws `std::runtime_error` if the name is not in the pipeline.
`T get<T>(String name)`	Returns a typed wrapper, exposing that type’s element-specific API — for example `get<CamSrc>("source")` for the image-capture calls. Use it when the element was added by factory name but its specialized interface is needed.

Runtime property and per-pad configuration — get() reaches any element by name while the pipeline is loaded or playing:

// Change a property at run time.
pipeline.get("display").set("fullscreen", false);

// Per-pad properties on a request pad of qtivcomposer.
pipeline.get("composer").input(1).set("alpha", 0.5);

Typed access to an element’s own API — get<T>() returns the specialized wrapper, so element-specific calls become available on an element that was added by factory name:

// qtiqmmfsrc was added as "source"; CamSrc exposes its capture API.
auto cam = pipeline.get<CamSrc>("source");
cam.image_capture();

See the sample applications for complete image-capture pipelines.

get() returns an element wrapper by value; each call yields a new wrapper referencing the same GstElement. Bind it with auto, not auto&.

Describing a pipeline in YAML The graph — elements, properties, stream filters, and links — can be described in YAML instead of in code. This separates what the pipeline is from what the application does with it: the same binary can run a different topology, a different model, or a different source by pointing it at another configuration file. That makes it useful where the pipeline is not fixed at build time:

Deploying one binary across variants — the same application against a built-in camera, a USB camera, or an RTSP stream, switching source and format per device.
Tuning without rebuilding — resolution, framerate, model path, or delegate changed on the target, which matters when a rebuild means a full cross-compile and reflash.
Keeping topology under review — the pipeline shape lives in a file that can be diffed and versioned on its own.

Loading from YAML does not give up application logic: the elements are ordinary pipeline members afterwards, so get() and get<T>() reach them as usual.

pipeline:
  elements:
    - type: filesrc
      name: src
      location: ~/media/video.mp4

    - type: qtdemux
      name: demux

    - type: h264parse
      name: parse

    - type: v4l2h264dec
      name: decoder
      output-io-mode: 4
      capture-io-mode: 4

    - type: filter
      name: videofilter
      video:
        format: NV12

    - type: waylandsink
      name: display
      fullscreen: true

Everything lives under a single root key, pipeline:, which takes three independent child keys:

pipeline:
  eos: true             # optional — EOS-on-shutdown, same as eos(true) in code
  elements:             # required — the elements, in add order
    - type: ...
  links:                # optional — explicit links, for non-linear graphs
    - [ ... ]

Key	Required	Description
`eos`	No	Boolean. Equivalent to calling `eos(bool)`. Defaults to false when omitted.
`elements`	Yes	Sequence of elements. Add order is link order, exactly as with `add()`.
`links`	No	Sequence of chains, each a list of element names. Equivalent to one `link()` call per entry. Omit it for linear pipelines.

Within elements:, each entry needs type (the factory name) and name (the instance name); every remaining key is applied as a property of that element. An entry with type: filter inserts a stream filter instead, described by a video:, image:, h264:, tensor:, text:, audio:, or raw caps: block. Any of those blocks may carry an add: sequence of key=value strings, appended to the filter’s caps through add() — the same escape-hatch the typed filters expose in code. It is accepted both inside the filter block and alongside it:

    - type: filter
      name: videofilter
      video:
        format: NV12
        add:
          - colorimetry=bt709

links: mirrors link() — one sequence per chain, so the branching pipeline shown earlier becomes:

  links:
    - [split, mlmuxer]
    - [source, videostream, split, q1, preprocessing, inferencing,
       postprocessing, mlf, mlmuxer, overlay, display]

The constructor takes the YAML content, so read the file first:

std::ifstream input(config_path);
std::ostringstream buffer;
buffer << input.rdbuf();

Pipeline pipeline("demo-pipeline", buffer.str());
pipeline.execute();

The names in the configuration are the handles the application uses afterwards. Given a config that declares a qtimlpostprocess element named postprocessing, the application can claim its post-processing without the topology being hard-coded:

Pipeline pipeline("demo-pipeline", buffer.str());

auto postprocessing = pipeline.get<MLPostprocess>("postprocessing");
postprocessing.set_handler([](const MLFrame& frame, const MLParam& params,
                              MLDetections& detections) {
  return decode_detection(frame, params, detections);
});

pipeline.execute();

Element

Purpose Generic wrapper around a GstElement with fluent, variadic property setting. Can be instantiated at create time or used at runtime to update a live pipeline. Serves as the base class for all specialized element types. Key Responsibilities

Create GStreamer elements from a factory name.
Provide fluent, variadic property setting for all common types and StreamFilter (caps).
Implement pad linking/unlinking with error checking.
Support state publication to pipeline (sync, deactivate, stop).
Expose per-pad configuration through Port.
Serve as the base class for specialized wrappers.

Element display("waylandsink", "display");
display.set("fullscreen", true);
display.set("sync", false);

Any element can be constructed directly and handed to Pipeline::add(), or reached later by name with Pipeline::get(). Elements cannot be copied — only moved — so pass them by reference or move them where ownership has to change.

`qti::Element`	Description
`Element(const std::string &factory, const std::string &name = {})`	Creates the element from a GStreamer factory name. The instance name is optional but recommended, since it is how the element is addressed in `link()`, `get()`, and log messages.
`Element &set(String prop, Value value, ...)`	Sets one or more properties. Accepts any number of key/value pairs in a single call and returns `*this`, so calls can be chained.
`Element &link(Element &downstream, String src_pad = "src", String sink_pad = "sink")`	Links this element’s src pad to `downstream`’s sink pad. Throws `std::runtime_error` naming both pads if the link fails.
`Element &unlink(Element &downstream, String src_pad = "src", String sink_pad = "sink")`	Unlinks the named pad pair between this element and `downstream`.
`Element &unlink(String src_pad = "src")`	Unlinks whatever is connected to the given src pad, without naming the peer.
`Element &sync()`	Synchronizes the element’s state with its parent pipeline’s state. Call this after adding an element to a pipeline that is already running.
`Element &deactivate()`	Moves the element to `PAUSED`.
`Element &stop()`	Moves the element to `NULL`.
`Port input(unsigned int id)`	Returns the sink pad at the given index, for per-pad configuration.
`Port input(String name_or_type, unsigned int id)`	Returns the sink pad matching a pad-template name and index.
`Port output(unsigned int id)`	Returns the src pad at the given index.
`Port output(String name_or_type, unsigned int id)`	Returns the src pad matching a pad-template name and index.

Setting properties set() is variadic, so a whole configuration can go in one call, and property names are validated immediately — an unknown property throws std::invalid_argument naming the property and the element type rather than being silently ignored:

Element decoder("v4l2h264dec", "decoder");
decoder.set("output-io-mode", 4, "capture-io-mode", 4);

Besides the usual scalar types and strings, set() accepts a StreamFilter wherever an element takes a caps property, and plain enums (converted to their underlying integer):

AppSrc appsrc("appsrc");
appsrc.set("caps", VideoFilter().format("NV12")
                                .resolution(1920, 1080)
                                .framerate(30));

Per-pad configuration with Port Some elements are configured per pad rather than per element — a compositor’s inputs each have their own position, size, and alpha. input() and output() return a Port for one pad, which carries the same fluent, validated set():

// Second input of qtivcomposer: blend at 50% alpha.
pipeline.get("composer").input(1).set("alpha", 0.5);

`qti::Port`	Description
`Port &set(String prop, Value value, ...)`	Sets one or more pad properties. Validated like `Element::set()`, and additionally accepts `std::vector<int>` and `{a, b}` initializer lists for array-valued pad properties such as positions and dimensions.

The id selects which pad, and name_or_type narrows it to a pad template when an element has several families of pads. A template containing %u is formatted with the index ("image_%u" with id 0 gives image_0); otherwise the name and index are joined with an underscore ("sink" with id 1 gives sink_1). Without a name, the pad is matched by numeric suffix first and by position among same-direction pads second. An unresolvable pad throws std::runtime_error.

A Port refers to a pad of a live element and cannot be copied, only moved. Get it, set the properties, and let it go — as in the one-liner above — rather than storing it beyond the lifetime of the element.

Runtime changes Because Pipeline::get() returns an Element for any element in the pipeline, the same set() works while the pipeline is playing:

pipeline.get("display").set("fullscreen", false);

sync(), deactivate(), and stop() exist for reconfiguring a live pipeline. An element added to an already-running pipeline starts in NULL while the pipeline is PLAYING, so it will not process data until its state is brought up to match — that is what sync() does. deactivate() and stop() take a single element down without touching the rest of the graph.

Runtime relinking is an advanced use of the API. Pad linking, unlinking, and per-element state changes on a live pipeline must respect GStreamer’s rules about when pads may be changed; the wrappers report failures as exceptions but do not remove the need to get the sequence right.

AppSrc

Purpose AppSrc wraps the appsrc element, the entry point for feeding application-produced buffers into a pipeline. It manages the need-data/enough-data signal plumbing so the application only has to supply buffers, either from a callback or by pushing them directly. Key Responsibilities

Validate that the wrapped element is indeed an appsrc.
Register and dispatch the underlying need-data and enough-data signals.
Let the application supply buffers on demand through set_buffer_producer(), or push them directly with push_buffer().
Signal end-of-stream on this element’s output.
Manage lifetime of the connected signal handlers.

AppSrc appsrc("src");
appsrc.set("is-live", true)
    .set("block", true)
    .set("format", AppSrc::Format::TIME)
    .set("do-timestamp", true)
    .set("caps", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30));

`qti::AppSrc`	Description
`AppSrc(const std::string &name = {})`	Creates an `appsrc` element with an optional instance name.
`AppSrc &set(String prop, Value value, ...)`	Sets one or more properties, same as `Element::set()`.
`AppSrc &set_buffer_producer(std::function<bool(qti::Buffer &)> producer)`	Registers a callback invoked whenever appsrc needs more data. Fill the buffer and return `true` to push it, `false` to skip this round.
`AppSrc &set_enough_handler(std::function<void()> enough)`	Registers a callback invoked when appsrc reports it is holding enough queued data.
`bool push_buffer(qti::Buffer &buffer)` / `bool push_buffer(qti::Buffer &&buffer)`	Pushes a buffer into the stream directly, outside the `need-data` callback. Returns `true` once it is accepted downstream.
`void end_of_stream()`	Sends an end-of-stream event on this element’s output.

AppSrc::Format mirrors appsrc’s own format property (DEFAULT, BYTES, TIME, BUFFERS, PERCENT); TIME, used above, is the usual choice for timestamped media.

push_buffer() transfers a buffer’s underlying GstBuffer without copying when the Buffer already owns one — which is exactly the case for a buffer just pulled from an AppSink. A Buffer filled with raw data instead (as inside a set_buffer_producer() callback) is copied into a new GstBuffer on push.

AppSink

Purpose AppSink wraps the appsink element, the exit point that delivers pipeline output back to application code. It pulls samples through appsink’s action signals and hands them to the application as Buffer. Key Responsibilities

Validate that the wrapped element is indeed an appsink.
Pull samples via appsink’s action signals and convert them into Buffer.
Deliver each buffer to the application through set_buffer_consumer().
Deliver the first (preroll) sample separately through set_preroll_handler().
Notify the application of end-of-stream through set_eos_handler().
Manage lifetime of the connected signal handlers.

AppSink appsink("sink");
appsink.set("emit-signals", true)
    .set("max-buffers", 5)
    .set("drop", true)
    .set_buffer_consumer([](Buffer buf) {
        // consume buf.data(), buf.size(), buf.pts()
    });

`qti::AppSink`	Description
`AppSink(const std::string &name = {})`	Creates an `appsink` element with an optional instance name.
`AppSink &set(String prop, Value value, ...)`	Sets one or more properties, same as `Element::set()`.
`AppSink &set_buffer_consumer(std::function<void(qti::Buffer)> consumer)`	Registers a callback invoked for every buffer pulled from the pipeline.
`AppSink &set_preroll_handler(std::function<bool(qti::Buffer &&)> preroll)`	Registers a callback invoked for the first (preroll) sample only; return `false` to report an error instead of accepting it.
`AppSink &set_eos_handler(std::function<void()> eos)`	Registers a callback invoked when appsink receives end-of-stream.

Bridging two pipelines Because AppSink and AppSrc are ordinary elements, they can sit in two independent Pipeline objects and be wired together in application code — useful when one graph produces frames (say, from a camera) and another consumes them (say, encoding or display) on its own clock:

AppSrc appsrc("src");
appsrc.set("is-live", true)
    .set("block", true)
    .set("format", AppSrc::Format::TIME)
    .set("do-timestamp", true)
    .set("caps", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30));

AppSink appsink("sink");
appsink.set("emit-signals", true)
    .set("max-buffers", 5)
    .set("drop", true)
    .set_buffer_consumer([&](Buffer b) {
        appsrc.push_buffer(std::move(b));
    });

Pipeline pipeline1("p1");
pipeline1.add("videotestsrc", "testsrc", "is-live", true, "pattern", "ball")
    .add_stream_filter("videostream", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
    .add(appsink);

Pipeline pipeline2("p2");
pipeline2.add(appsrc)
    .add("waylandsink", "display", "fullscreen", true);

Each buffer handed to set_buffer_consumer() on pipeline1 is pushed straight into pipeline2 through appsrc.push_buffer(), without a copy.

CamSrc

Purpose CamSrc wraps the qtiqmmfsrc camera source element and exposes its still/burst image-capture action signals as ordinary method calls, so an application can trigger a snapshot without touching GStreamer signals directly. Key Responsibilities

Validate that the wrapped element is indeed a qtiqmmfsrc.
Trigger still or burst image capture through image_capture().
Cancel any pending capture request through cancel_capture().

`qti::CamSrc`	Description
`CamSrc(const std::string &name = {})`	Creates a `qtiqmmfsrc` element with an optional instance name.
`bool image_capture(unsigned int count = 1)`	Captures `count` still images.
`bool image_capture(CaptureMode mode, unsigned int count = 1)`	Captures `count` images using `CaptureMode::kStill` or `CaptureMode::kBurst`.
`bool image_capture(CaptureMode mode, unsigned int count, const std::vector<void*> &metadata_ptrs)`	Same as above, attaching a per-request metadata payload to each capture.
`bool cancel_capture()`	Cancels any capture requests still pending.

A running pipeline built from a plain factory name still has a qtiqmmfsrc element in it; use Pipeline::get<CamSrc>() to reach it as a CamSrc and call its capture API:

auto cam = pipeline.get<CamSrc>("source");
cam.image_capture();

See examples/test_camera_and_capture for a complete pipeline that branches a qtiqmmfsrc into a live display and a multifilesink for captured images.

MLVConverter

Purpose MLVConverter wraps the qtimlvconverter element, which converts raw video frames into the input tensor format expected by a model. Beyond the properties available on the generic Element, it lets the application replace the built-in conversion with its own implementation through a lambda. Key Responsibilities

Validate that the wrapped element is a qtimlvconverter.
Convert incoming video frames into model input tensors.
Optionally delegate the conversion to an application-supplied pre-process callback.
Provide fluent, variadic property setting that returns MLVConverter& for chaining.

Pre-process callback The callback receives the source frames as MLVideoBlits (each entry carries the image planes plus the source quadrilateral, destination rectangle, alpha and rotation) and must fill the tensors of the MLFrame output. Returning false marks the conversion as failed.

using TensorsPreprocessCallback =
    std::function<bool(const MLVideoBlits& blits, MLFrame& output)>;

Set engine to none to disable the internal pre-processing path when you supply your own callback — otherwise the built-in conversion runs instead.

Usage

MLVConverter preprocessing("preprocessing");

// engine=none disables internal preprocessing path in qtimlvconverter.
preprocessing.set("engine", "none");

preprocessing.set_handler(
    [](const MLVideoBlits& blits, MLFrame& output) {
      if (blits.entries.empty() || output.tensors.empty())
        return false;

      const MLVideoBlit& blit = blits.entries.front();
      return convert_nv12_to_nhwc_i8(blit.image, &blit,
                                     output.tensors.front());
    });

pipeline.add(preprocessing);

`qti::MLVConverter`	Description
`MLVConverter(const std::string &name = {})`	Creates a `qtimlvconverter` element with an optional instance name.
`MLVConverter &set(String prop, Value value, ...)`	Sets one or more properties, same as `Element::set()`.
`MLVConverter &set_handler(TensorsPreprocessCallback handler)`	Registers the external pre-process callback. Requires `engine=none` to take effect.

MLPostprocess

Purpose MLPostprocess wraps the qtimlpostprocess element, which decodes model output tensors into ML metadata attached to the frame. It lets the application replace the built-in decoding module with its own implementation through a lambda. Key Responsibilities

Validate that the wrapped element is a qtimlpostprocess.
Decode output tensors into ML metadata (classifications, detections, poses, depth maps, segmentations), or pass the raw tensors through for a further stage.
Optionally delegate the decoding to an application-supplied post-process callback.
Select the callback type automatically from the lambda’s third parameter.
Provide fluent, variadic property setting that returns MLPostprocess& for chaining.

Post-process callbacks set_handler() is overloaded once per ML task. The overload is selected by the type of the third parameter of the callback, so the same method name registers any of the supported result types. Each callback receives the model output tensors as MLFrame, the element parameters as MLParam, and an output container to fill. Returning false marks the frame as failed.

Callback type	Signature	Output
`ClassificationPostprocessCallback`	`bool(const MLFrame&, const MLParam&, MLClassifications&)`	Class name, confidence, color
`ObjectDetectionPostprocessCallback`	`bool(const MLFrame&, const MLParam&, MLDetections&)`	Bounding boxes, confidence, landmarks
`PoseEstimationPostprocessCallback`	`bool(const MLFrame&, const MLParam&, MLPoses&)`	Keypoints and keypoint links
`DepthEstimationPostprocessCallback`	`bool(const MLFrame&, const MLParam&, MLDepthMaps&)`	Per-cell depth values and colors
`SegmentationPostprocessCallback`	`bool(const MLFrame&, const MLParam&, MLSegmentations&)`	Per-cell labels and colors
`TensorsPostprocessCallback`	`bool(const MLFrame&, const MLParam&, MLFrame&)`	Raw output tensors, for daisy-chaining into a next stage

Use MLParam::get() to read element parameters inside the callback. For example params.get("input-tensor-region", region) returns the Region the frame was scaled into, which is needed to map model coordinates back to frame coordinates.

Usage

static const std::vector<LabelEntry> labels =
    load_labels(HOME_PATH + "/labels/resnet101.json");

MLPostprocess postprocessing("postprocessing");
postprocessing
    .set("results", 1)
    .set_handler([](const MLFrame& frame, const MLParam& params,
                    MLClassifications& classifications) {
      return decode_top1_classification(frame, params, labels,
                                        classifications,
                                        /*confidence_threshold=*/51.0f, "");
    });

pipeline.add(postprocessing);

`qti::MLPostprocess`	Description
`MLPostprocess(const std::string &name = {})`	Creates a `qtimlpostprocess` element with an optional instance name.
`MLPostprocess &set(String prop, Value value, ...)`	Sets one or more properties, same as `Element::set()`.
`MLPostprocess &set_handler(ClassificationPostprocessCallback handler)`	Registers a callback that decodes tensors into classifications.
`MLPostprocess &set_handler(ObjectDetectionPostprocessCallback handler)`	Registers a callback that decodes tensors into detections.
`MLPostprocess &set_handler(PoseEstimationPostprocessCallback handler)`	Registers a callback that decodes tensors into poses.
`MLPostprocess &set_handler(DepthEstimationPostprocessCallback handler)`	Registers a callback that decodes tensors into depth maps.
`MLPostprocess &set_handler(SegmentationPostprocessCallback handler)`	Registers a callback that decodes tensors into segmentations.
`MLPostprocess &set_handler(TensorsPostprocessCallback handler)`	Registers a callback that produces raw output tensors, for chaining into another ML stage.

ML inference bins

Purpose The ML video bins wrap a full pre-process → inference → post-process chain into a single element, so a whole AI branch can be added to the pipeline as one node instead of being assembled from separate elements and queues. Four bins are available, one per inference runtime:

QIM SDK C++ class	GStreamer element	Runtime
`MLVideoONNXBin`	`qtimlvideoonnxbin`	ONNX Runtime
`MLVideoQNNBin`	`qtimlvideoqnnbin`	Qualcomm® AI Engine Direct (QNN)
`MLVideoSNPEBin`	`qtimlvideosnpebin`	Snapdragon® Neural Processing Engine (SNPE)
`MLVideoTFLiteBin`	`qtimlvideotflitebin`	LiteRT (TensorFlow Lite)

All four expose an identical API and differ only in the wrapped element and the runtime-specific properties. Each inherits from both the pre-process and the post-process base, so it accepts a custom pre-process callback, a custom post-process callback, or both. Key Responsibilities

Validate that the wrapped element matches the expected bin factory.
Run pre-processing, inference, and post-processing inside a single element.
Optionally delegate pre-processing to an application-supplied callback via set_preprocess_handler().
Optionally delegate post-processing to an application-supplied callback via set_postprocess_handler().
Provide fluent, variadic property setting that returns the concrete bin type for chaining.

Properties Because the bin embeds several stages, its properties are prefixed by the stage they configure — preprocess-*, inference-*, and postprocess-*. For example inference-model on the bin corresponds to model on a standalone inference element, and postprocess-labels corresponds to labels on a standalone qtimlpostprocess.

Set preprocess-engine to none to disable the internal pre-processing path when you supply your own pre-process callback.

Usage Custom post-processing — the bin handles pre-processing and inference internally, the application decodes the output tensors.

static const std::vector<LabelEntry> labels =
    load_labels(HOME_PATH + "/labels/yolov8.json");

MLVideoTFLiteBin mlbin("mlbin");
mlbin
    .set("inference-delegate", "external",
         "inference-external-delegate-path", "libQnnTFLiteDelegate.so",
         "inference-external-delegate-options",
         "QNNExternalDelegate,backend_type=htp;",
         "inference-model", HOME_PATH + "/models/yolov8_det_quantized.tflite")
    .set_postprocess_handler(
        [&](const MLFrame& frame, const MLParam& params,
            MLDetections& detections) {
          return decode_detection(frame, detections, params, labels,
                                  /*confidence_threshold=*/0.70f);
        });

Pipeline pipeline("mlbin-external-detection");
pipeline
    .add("filesrc", "src", "location", HOME_PATH + "/media/video.mp4")
    .add("qtdemux", "demux")
    .add("h264parse", "parse")
    .add("v4l2h264dec", "decoder", "output-io-mode", 4, "capture-io-mode", 4)
    .add_stream_filter("vf", VideoFilter().format("NV12"))
    .add(mlbin)
    .add("qtivoverlay", "overlay")
    .add("waylandsink", "display", "fullscreen", true);

pipeline.execute();

Custom pre-processing — the application converts frames into input tensors, the bin runs inference and its built-in post-process module.

MLVideoTFLiteBin mlbin("mlbin");

// preprocess-engine=none disables internal preprocessing path in mlbin.
mlbin.set("preprocess-engine", "none");
mlbin.set("inference-delegate", "external");
mlbin.set("inference-external-delegate-path", "libQnnTFLiteDelegate.so");
mlbin.set("inference-external-delegate-options",
          "QNNExternalDelegate,backend_type=htp;");
mlbin.set("inference-model",
          HOME_PATH + "/models/yolov8_det_quantized.tflite");
mlbin.set("postprocess-module", "yolov8");
mlbin.set("postprocess-labels", HOME_PATH + "/labels/yolov8.json");

mlbin.set_preprocess_handler(
    [](const MLVideoBlits& blits, MLFrame& output) {
      if (blits.entries.empty() || output.tensors.empty())
        return false;

      const MLVideoBlit& blit = blits.entries.front();
      return convert_nv12_to_nhwc_i8(blit.image, &blit,
                                     output.tensors.front());
    });

Swapping runtimes is a matter of changing the class and the runtime-specific properties; the callback registration is unchanged.

MLVideoQNNBin   qnn_bin("mlbin");     // qtimlvideoqnnbin
MLVideoSNPEBin  snpe_bin("mlbin");    // qtimlvideosnpebin
MLVideoONNXBin  onnx_bin("mlbin");    // qtimlvideoonnxbin

APIs The table below applies to each of MLVideoONNXBin, MLVideoQNNBin, MLVideoSNPEBin, and MLVideoTFLiteBin; substitute the concrete class name for <Bin>.

`qti::<Bin>`	Description
`<Bin>(const std::string &name = {})`	Creates the corresponding `qtimlvideo*bin` element with an optional instance name.
`<Bin> &set(String prop, Value value, ...)`	Sets one or more properties, same as `Element::set()`. Properties are stage-prefixed: `preprocess-`, `inference-`, `postprocess-*`.
`<Bin> &set_preprocess_handler(TensorsPreprocessCallback handler)`	Registers the external pre-process callback. Requires `preprocess-engine=none` to take effect.
`<Bin> &set_postprocess_handler(Callback handler)`	Registers the external post-process callback. Accepts the same task-specific callback types as `MLPostprocess::set_handler()`.

The bins cannot be copied, only moved — the same as the other element wrappers.

Buffer

Purpose Unified buffer abstraction over native GStreamer memory. Provides read/write access, timestamp metadata (pts, dts, duration), resize support, and zero-copy transfer between AppSink and AppSrc. Key Responsibilities

Own and manage mapping/unmapping of underlying GstBuffer or GstSample.
Provide read/write access to buffer memory.
Support resizing and writable allocation for AppSrc.
Carry timestamps (PTS/DTS/duration) and apply them back to GstBuffer.
Enable zero-copy extraction using take_gst_buffer().

appsink.set_buffer_consumer([](Buffer buf) {
    auto* data = buf.data();
    auto  pts  = buf.pts();
});

A Buffer arrives either from an AppSink — wrapping the sample the pipeline just produced — or is allocated by the application to be filled and pushed through an AppSrc. Which of the two it is determines whether the payload is writable: samples coming out of the pipeline are read-only. APIs

`qti::Buffer`	Description
`Buffer()`	Creates an empty wrapper, referencing no data. `valid()` returns `false` until it is filled.
`Buffer(size_t size)`	Allocates a writable buffer with a payload of `size` bytes, for pushing into an `AppSrc`.
`static Buffer from_readable_sample(void *gst_sample_opaque)`	Wraps an existing `GstSample` and exposes its payload and metadata as a `Buffer`.
`uint8_t data()` / `const uint8_t data() const`	Returns a pointer to the payload. The non-const overload requires a writable buffer.
`size_t size() const`	Returns the payload size in bytes.
`void resize(size_t n)`	Resizes the payload storage.
`void set_pts(uint64_t ns)` / `void set_dts(uint64_t ns)` / `void set_duration(uint64_t ns)`	Sets the presentation timestamp, decode timestamp, or duration, in nanoseconds.
`uint64_t pts() const` / `uint64_t dts() const` / `uint64_t duration() const`	Reads the corresponding timestamp field, in nanoseconds.
`bool is_writable() const`	Returns `true` when the underlying memory can be modified in place.
`bool is_readonly() const`	Returns `true` when the buffer came from an `AppSink` and its memory cannot be modified.
`bool valid() const`	Returns `true` when the wrapper currently references valid data.
`void *take_gst_buffer()`	Transfers ownership of the underlying `GstBuffer` to the caller and clears this wrapper. This is the zero-copy hand-off path used by `AppSrc::push_buffer()`.
`void refill_for_appsrc(size_t n)`	Prepares the buffer’s storage for reuse inside an `AppSrc` producer callback, resizing it to `n` bytes.

Like the element wrappers, a Buffer cannot be copied, only moved — ownership of the underlying memory is unique. Pass it by reference, or std::move() it where ownership has to change, as in the AppSink → AppSrc bridge above.

StreamFilter

Purpose StreamFilter describes the characteristics of a media stream between two pipeline stages so they can negotiate concrete parameters rather than relying on defaults. Without explicit parameters, adjacent components may fall back to suboptimal formats, trigger implicit conversions, or produce mismatched configurations. Specialized subtypes provide fluent, type-safe setters for each domain. Raw caps strings are also accepted for cases not covered by typed setters or when copying directly from gst-launch output. When to Use StreamFilter

When two adjacent stages can interoperate in multiple ways, and you want to select explicit parameters instead of relying on defaults.
When you need to optimize latency or performance by avoiding implicit conversions (e.g., choose a hardware-friendly pixel format).
When you want predictable behavior across platforms (e.g., lock color range/primaries/transfer/matrix).
When you integrate tensors for AI/ML and must precisely specify dimensions and element types.
When you need a human-readable summary of the stream parameters for logs and debugging.

A StreamFilter is not an Element and is never added to a pipeline with add(). It is always attached between stages with Pipeline::add_stream_filter(unique_name, filter), which internally creates a capsfilter element and applies the filter’s caps to it. The base StreamFilter class itself only exposes a raw-caps-string constructor and to_string(); the fluent per-domain setters (.format(), .resolution(), .framerate(), …) live on the typed subclasses such as VideoFilter. Usage Typed subclass

Pipeline pipeline("video-pipeline");
pipeline.add("videotestsrc", "src", "pattern", "ball")
        .add_stream_filter("filter", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
        .add("waylandsink", "sink")
        .execute();

Raw caps string

Pipeline pipeline("video-pipeline-streamfilter-string");
pipeline.add("videotestsrc", "src", "pattern", "ball")
        .add_stream_filter("filter", qti::StreamFilter(
            "video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1"))
        .add("waylandsink", "sink")
        .execute();

APIs

`qti::StreamFilter`	Description
`StreamFilter()`	Creates an empty filter expression.
`StreamFilter(const std::string &caps_or_mediatype)`	Creates a filter from a raw caps or media-type string, e.g. `"video/x-raw,format=NV12"`.
`std::string to_string() const`	Serializes the current filter expression to a caps string, for logging/debugging.

VideoFilter

Purpose Configures a raw video stream (YUV, RGB, Bayer) between a camera source and any downstream processing or display element. Supports format, resolution, framerate, colorimetry, interlace mode, and pixel aspect ratio. Key Responsibilities

Produce caps with standard raw video attributes: format, resolution, framerate, colorimetry, interlace mode, pixel aspect ratio, etc.

APIs

Member	Description
`format(const std::string &fmt)` / `format(Format fmt)`	Sets the stream’s pixel format, for example `NV12`, `YUY2`, `RGB`, `RGBA`. Determines how pixel data is laid out in memory; essential for hardware compatibility and color-conversion paths. Accepts a string or a `VideoFilter::Format` enumerator.
`resolution(int width, int height)`	Specifies the frame dimensions in pixels.
`framerate(int num, int den = 1)` / `framerate(float num)`	Sets the frame rate as a fraction, for example `30/1` or `30000/1001` frames-per-second. Pass a single value when you do not need a fractional frame rate, like `30.0` fps.
`colorimetry(const std::string &value)`	Defines the colorimetry standard, for example `bt601`, `bt709`, `bt2020`.
`range(const std::string &value)`	Selects the quantization range, `full` or `limited`.
`interlace(const std::string &mode)`	Sets the interlacing mode, for example `progressive` or `interleaved`. Progressive is standard for modern pipelines; interlaced is still used in some camera/legacy systems.
`pixel_aspect_ratio(int num, int den = 1)`	Specifies the pixel aspect ratio, also called PAR. Normally `1/1`; used for anamorphic or non-square-pixel formats.
`add(const std::string &expr)`	Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API.

VideoFilter::Format enumerates the raw video formats (video/x-raw), JPEG (image/jpeg), and the Bayer patterns (video/x-bayer: BGGR, RGGB, GBRG, GRBG, MONO). Usage We can achieve the same results in three different ways: Typed (enum) API - if you want compile-time safety, IDE autocompletion, and guardrails against typos. Ideal for SDK consumers and most production code.

auto v1 = VideoFilter()
                .format(VideoFilter::Format::NV12)
                .resolution(1920, 1080)
                .framerate(30)
                .colorimetry("bt709")
                .range("limited")
                .interlace("progressive");

String API - if you need quick iteration or to pass values not (yet) covered by enums — e.g., enabling a freshly added GStreamer value on a target platform without immediately updating headers.

auto v2 = VideoFilter()
                .format("NV12")
                .resolution(1920, 1080)
                .framerate(30)
                .colorimetry("bt709")
                .range("limited")
                .interlace("progressive");

Raw caps string - very useful for setting arbitrary stream (caps) filter or you want to lift an existing caps line from logs, gst-launch, or a pad template and drop it as-is — great for debugging, prototyping, or reproducing pipelines exactly.

auto v3 = StreamFilter("video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1,colorimetry=bt709,range=limited");

H264Filter

Purpose Configures an encoded H.264 stream between an encoder and a downstream parser or muxer. Supports resolution, framerate, stream-format, alignment, profile, and level. Key Responsibilities

Specify resolution, framerate, stream-format, alignment, profile, level.
Insert codec_data buffers.
Attach arbitrary name/value string properties.

APIs

Member	Description
`resolution(int width, int height)`	Specifies the frame dimensions in pixels.
`framerate(int num, int den = 1)` / `framerate(float num)`	Sets the frame rate as a fraction, or as a single value when a fractional rate is not needed.
`profile(const std::string &p)`	Sets the H.264 profile caps field, for example `baseline`, `main`, `high`.
`level(const std::string &lvl)`	Sets the H.264 level caps field.
`stream_format(const std::string &f)`	Sets the H.264 stream-format caps field, for example `byte-stream`, `avc`.
`alignment(const std::string &a)`	Sets the H.264 alignment caps field, for example `au`, `nal`.
`codec_data(const std::string &bytes)`	Sets the H.264 `codec_data` caps field from a byte string.
`set(const std::string &key, const std::string &val)`	Shorthand for `add(key + "=" + val)`.
`add(const std::string &expr)`	Appends an arbitrary custom caps field using a `key=value` expression.

Usage

auto h264f = H264Filter().resolution(1920, 1080)
                         .framerate(30)
                         .profile("baseline")
                         .stream_format("byte-stream")
                         .alignment("au");

Pipeline pipeline("h264-pipeline");
pipeline.add_stream_filter("h264f", h264f);

TensorFilter

Purpose Configures an ML tensor stream with explicit data type and shape (e.g., UINT8, NHWC). Not needed in a full inference pipeline — when preprocess, inference, and postprocess elements are all present, tensor parameters are negotiated automatically. Use TensorFilter when:

The preprocessing stage does not have model dimension information
You need exact tensor data output from the inference plugin

Key Responsibilities

Declare element type (e.g., UINT8, FLOAT32) for tensors.
Specify shape (dimensions). Supports:
- Single tensor: a 1D list (e.g., 1, 520, 520, 3).
- Multiple tensors: array-of-arrays (e.g., { {1,3,H,W}, {1,32} }) when needed.

APIs

Member	Description
`type(const std::string &t)` / `type(Type t)`	Specifies the tensor data type, for example `UINT8` or `FLOAT32`. Accepts a string or a `TensorFilter::Type` enumerator (`UINT8`, `UINT16`, `UINT32`, `INT8`, `INT16`, `INT32`, `FLOAT16`, `FLOAT32`).
`dimensions(UInts... dims)`	Specifies a single tensor’s dimensions as a variadic list of integers, for example `dimensions(1, 520, 520, 3)`.
`dimensions(const std::vector<int> &one_tensor_dims)`	Specifies a single tensor’s dimensions as a vector.
`dimensions(const std::vector<std::vector<int>> &many_tensors_dims)`	Specifies several tensors at once, one inner vector per tensor.
`add(const std::string &expr)`	Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API.

Usage We can achieve the same results in three different ways: Typed (enum) API - if you want compile-time safety, IDE autocompletion, and guardrails against typos. Ideal for SDK consumers and most production code.

auto t1 = TensorFilter()
                .type(TensorFilter::Type::UINT8)
                .dimensions(1, 520, 520, 3);      // NHWC, for example

String API - if you need quick iteration or to pass values not (yet) covered by enums — e.g., enabling a freshly added GStreamer value on a target platform without immediately updating headers.

auto t2 = TensorFilter()
                .type("FLOAT32")
                .dimensions(1, 3, 640, 640);      // NCHW, for example

auto t3 = StreamFilter("neural-network/tensors,type=UINT8,dimensions=<<1,520,520,3>>");

TextFilter

Purpose Configures a raw text stream (text/x-raw) for pipelines that carry textual data such as metadata, subtitles, or logs. Minimal by design — attach only the fields you need using .add(“key=value”). Key Responsibilities

Provide a lightweight caps descriptor for text (text/x-raw) without enforcing a schema.
Let callers attach arbitrary properties (e.g., encoding/charset, language, packetization hints) using .add("key=value").

APIs

Member	Description
`add(const std::string &expr)`	Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API.

Usage

auto txt = TextFilter().add("charset=utf8");

ImageFilter

Overview Used exclusively with CamSrc (qtiqmmfsrc) to enable and link its dedicated image capture pad. Without ImageFilter, the image pad is not requested, not activated, and produces no buffers. It is a structural enabler, not a processing stage — it is not intended for general image processing or frame extraction from video sources. ImageFilter is used to:

Enable the image output pad exposed by CamSrc (qtiqmmfsrc)
Attach that pad to the pipeline so the source can activate image capture
Provide a concrete sink target required by GStreamer for pad activation

It is not intended for:

Generic image processing
Frame extraction from arbitrary video sources
Replacement of VideoFilter or AppSink

Relationship to CamSrc (qtiqmmfsrc)

CamSrc (qtiqmmfsrc) exposes a dedicated image pad that remains inactive unless linked.
ImageFilter explicitly exists to satisfy this requirement.
Without ImageFilter, the image pad:
- Will not be requested
- Will not be activated
- Will not produce buffers

This makes ImageFilter a structural enabler, not a processing stage. APIs

Member	Description
`format(const std::string &fmt)`	Sets the image format for the capture pad, for example `JPEG`.
`resolution(int width, int height)`	Specifies the captured image dimensions in pixels.
`framerate(int num, int den = 1)` / `framerate(float num)`	Sets the capture pad’s frame rate.
`add(const std::string &expr)`	Appends an arbitrary custom caps field using a `key=value` expression.

Usage

auto imagefilter1 = ImageFilter()
             .format("JPEG")
             .resolution(3840, 2160);

AudioFilter

Purpose Configures a raw PCM audio stream with sample format, channel count, sample rate, and layout. Use this to ensure audio converters and sinks negotiate exactly the format your hardware supports rather than inserting an implicit resampler. Key Responsibilities

Set audio sample format (e.g., S16LE, F32LE) to match HW/SW capabilities.
Define channels and rate for correct resampling/mixing.
Choose layout (interleaved vs non-interleaved) to align with downstream expectations.

APIs

Member	Description
`format(const std::string &fmt)` / `format(Format f)`	Sets the audio sample format, for example `S16LE`, `F32LE`. Accepts a string or an `AudioFilter::Format` enumerator.
`channels(int n)`	Specifies the number of audio channels.
`rate(int hz)`	Specifies the audio sample rate in Hz.
`layout(const std::string &l)` / `layout(Layout l)`	Specifies the audio layout. Accepts a string or `AudioFilter::Layout::INTERLEAVED` / `NON_INTERLEAVED`.
`add(const std::string &expr)`	Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API.

AudioFilter::Format covers the signed/unsigned integer formats (S8 through U32BE, including the packed 18/20/24-bit variants) and the float formats F32LE, F32BE, F64LE, F64BE. Usage We can achieve the same results in three different ways: Typed (enum) API - if you want compile-time safety, IDE autocompletion, and guardrails against typos. Ideal for SDK consumers and most production code.

auto a1 = AudioFilter()
                .format(AudioFilter::Format::S16LE)
                .rate(48000)
                .channels(2)
                .layout(AudioFilter::Layout::INTERLEAVED);

String API - if you need quick iteration or to pass values not (yet) covered by enums — e.g., enabling a freshly added GStreamer value on a target platform without immediately updating headers.

auto a2 = AudioFilter()
                .format("F32LE")
                .rate(44100)
                .channels(1)
                .layout("non-interleaved");

auto a3 = StreamFilter("audio/x-raw,format=S16LE,rate=48000,channels=2,layout=interleaved");

Logging

The SDK can either simplify GStreamer’s log stream into a compact, uniform format or step aside and let GStreamer log raw, with GST_DEBUG fully in control. Both modes, the verbosity levels, and the ordering constraint on configuring them are covered in Logging and Diagnostics.

QIM SDK Nodes

All nodes are executed by GStreamer elements under the hood. Most elements are handled through the Element base class. However, elements that provide additional functionality — such as application buffer interfaces in appsink and appsrc, or image-capture signals in camera sources like qtiqmmfsrc — require extended interfaces. If you do not need these extra capabilities (for example, image capture), you can still use these elements via the generic Element base class without any limitations for standard data processing. Below is a table listing all supported elements and the corresponding QIM SDK wrapper classes used to work with them. Sources

GStreamer	QIM SDK C++ Class
`qtiqmmfsrc`	`qti::CamSrc` (also usable as `qti::Element`)
`v4l2src`	`qti::Element`
`filesrc`	`qti::Element`
`multifilesrc`	`qti::Element`
`appsrc`	`qti::AppSrc`
`appsink`	`qti::AppSink`
`qtisocketsrc`	`qti::Element`

GStreamer	QIM SDK C++ Class
`qtimlaconverter`	`qti::Element`
`qtimlvconverter`	`qti::MLVConverter` (also usable as `qti::Element`)
`qtimlqnn`	`qti::Element`
`qtimlsnpe`	`qti::Element`
`qtimltflite`	`qti::Element`
`qtimlpostprocess`	`qti::MLPostprocess` (also usable as `qti::Element`)
`qtimlvideoonnxbin`	`qti::MLVideoONNXBin` (also usable as `qti::Element`)
`qtimlvideoqnnbin`	`qti::MLVideoQNNBin` (also usable as `qti::Element`)
`qtimlvideosnpebin`	`qti::MLVideoSNPEBin` (also usable as `qti::Element`)
`qtimlvideotflitebin`	`qti::MLVideoTFLiteBin` (also usable as `qti::Element`)
`qtimlaic`	`qti::Element`
`qtimetamux`	`qti::Element`
`qtimlmetaextractor`	`qti::Element`
`qtimlmetaparser`	`qti::Element`
`qtimldemux`	`qti::Element`
`qtimetatransform`	`qti::Element`
`mlonnx`	`qti::Element`

Multimedia

GStreamer	QIM SDK C++ Class
`qtiobjtracker`	`qti::Element`
`qtismartvencbin`	`qti::Element`
`qtivoverlay`	`qti::Element`
`qtivsplit`	`qti::Element`
`qtivcomposer`	`qti::Element`
`qtibatch`	`qti::Element`
`qticamreproc`	`qti::Element`
`qtic2adec`	`qti::Element`
`qtic2aenc`	`qti::Element`
`qtic2vdec`	`qti::Element`
`qtic2venc`	`qti::Element`
`qtijpegenc`	`qti::Element`
`qtivtransform`	`qti::Element`
`qticvimgpyramid`	`qti::Element`
`qticvoptclflow`	`qti::Element`
`qtidfs`	`qti::Element`
`v4l2h264dec`	`qti::Element`
`v4l2h264enc`	`qti::Element`
`v4l2h265dec`	`qti::Element`
`v4l2h265enc`	`qti::Element`
`tee`	`qti::Element`
`h264parse`	`qti::Element`
`rtpptdemux`	`qti::Element`
`rtph264depay`	`qti::Element`
`queue`	`qti::Element`
`qtdemux`	`qti::Element`
`tsdemux`	`qti::Element`

Common Application Design

Create pipeline with config file

The SDK supports fully declarative pipeline construction using YAML. A YAML pipeline describes elements and links between them, which are translated at runtime into a GStreamer pipeline. TopLevel Structure A YAML pipeline file has a single top-level key:

pipeline:
  elements: [...]
  links: [...]

Elements Section elements is an ordered list of pipeline elements. Each element entry defines:

The GStreamer element type
A unique name used for linking
Optional properties
Optional filter configuration

Common Fields

- type: <element-type>
  name: <unique-name>
  <property>: <value>

type: GStreamer element factory name, or filter
name: Unique identifier within the pipeline
properties: Any remaining key/value pairs are passed directly to the element as GStreamer properties

Filter Elements Filters are declared using type: filter and are further specialized by their configuration block. Stream Filter

- type: capsfilter
  name: raw_caps
  caps: "video/x-raw,format=NV12,width=1280,height=720,framerate=30/1"

Video Filter

- type: filter
  name: vf
  video:
    format: NV12
    width: 1920
    height: 1080
    framerate: 30

Links Section links defines explicit connection paths between elements. Each entry is an ordered list of element names. Elements are linked sequentially from left to right.

links:
  - [src, demux, parse, decoder, vf, split]

Example yaml files YAML config file

pipeline:
  elements:
    - type: qtiqmmfsrc
      name: source
      camera: "0"

    - type: filter
      name: filter
      video:
        format: NV12
        width: 1920
        height: 1080
        framerate: 30.0

    - type: tee
      name: split

    - type: qtimlvconverter
      name: preprocessing

    - type: qtimltflite
      name: inferencing
      delegate: gpu
      model: /etc/models/model.tflite

    - type: qtimlpostprocess
      name: postprocessing

    - type: qtimetamux
      name: mlmuxer

    - type: qtivoverlay
      name: overlay

    - type: waylandsink
      name: display
      sync: true
      async: true
      fullscreen: true

  links:
    - [source, filter, split, preprocessing, inferencing, postprocessing, mlmuxer, overlay, display]
    - [split, mlmuxer]

Multi-Model Daisy Chain

Below is shown a logical diagram of an AI daisy chain pipeline with sequential and parallel inferencing. The same could be handled by a straight sequential pipeline of inference bins. But we must set manually the source for every inference bin.

Try me

Several .add(...) calls in this example had missing/mismatched quotes in the source PDF (e.g. a stray closing paren instead of a closing quote, or a missing leading quote). These have been corrected below where unambiguous. One inline comment truncated at the page margin was completed using the identical sentence found in the prose above.

Element ml_car("qtimlvideotflitebin", "car")
  .add("performance-profile", "high-performance")
  .add("inference-delegate", "dsp")
  .add("inference-model", "/opt/car_detect.dlc")
  .add("inference-layers", "< /model.22/Mul_2, /model.22/Sigmoid >")
  .add("postprocess-threshold", 60.0)
  .add("postprocess-results", 4)
  .add("postprocess-module", "yolov8")
  .add("postprocess-labels", "/opt/car_detect_model.labels");
  // The same could be handle by straight sequential pipeline of inference bins. But we must set manually source to every inference bin.

Element ml_plate("qtimlvideotflitebin", "plated")
  .add("source", 1)
  .add("image-disposition", "centre")
  .add("performance-profile", "high-performance")
  .add("inference-delegate", "dsp")
  .add("inference-model", "/opt/plate_detect.dlc")
  .add("inference-layers", "< /model.22/Mul_2, /model.22/Sigmoid >")
  .add("postprocess-threshold", 40.0)
  .add("postprocess-results", 1)
  .add("postprocess-module", "yolov8")
  .add("postprocess-labels", "/opt/plate_detect_model.labels");

Element ml_anpr("qtimlvideotflitebin", "anpr")
  .add("source", 2)
  .add("image-disposition", "centre")
  .add("inference-delegate", "dsp")
  .add("inference-model", "/opt/anpr.dlc")
  .add("inference-layers", "< /decode_head2/classifier/Conv, /decode_head/classifier/Conv >")
  .add("postprocess-threshold", 15.0)
  .add("postprocess-results", 1)
  .add("postprocess-module", "anpr")
  .add("postprocess-labels", "/opt/anpr_model.labels");

Element ml_siglip("qtimlvideotflitebin", "siglip")
  .add("source", 1)
  .add("image-disposition", "centre");
  .add("inference-delegate", "dsp");
  .add("inference-model", "/opt/car_make_model.dlc");
  .add("inference-layers", "< /classifier/Gemm >");
  .add("postprocess-threshold", 15.0);
  .add("postprocess-results", 1);
  .add("postprocess-module", "mobilenet");
  .add("postprocess-labels", "/opt/car_make_model.labels");

Pipeline pipeline("SmartVehicleIdentification");
pipeline.add("camsrc", "source", "camera", "0")
            .add_stream_filter("filter", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
            .add(ml_car)
            .add(ml_plate)
            .add(ml_anpr)
            .add(ml_siglip)
            .add("qtivoverlay", "overlay")
            .add("waylandsink", "display", "sync", "true", "async", "true", "fullscreen")
            .execute();

Custom Postprocessing with YOLOv8 (Detection)

QIM SDK implements AI model post-processing through dedicated modules that take tensors as input and produce predictions. All remaining complexity — such as batching, daisy-chaining, image mask support — is encapsulated within the post-processing plugin. This makes the post-processing module API an ideal integration point for application-specific post-processing logic. As a result, applications do not need to handle batching, chaining, image masks, or ML metadata themselves. We will implement a specialized post-processing module to manage bindings with the application.

Custom postprocessing callback is attached to a qtimlpostprocess element via MLPostprocess::set_handler(...). Typical pipeline segment:

Preprocess frame/tensors (qtimlvconverter)
Run ML inference (qtimltflite)
Convert raw tensors in custom callback (qtimlpostprocess)

Supported callback types

ClassificationPostprocessCallback
ObjectDetectionPostprocessCallback
PoseEstimationPostprocessCallback
DepthEstimationPostprocessCallback
SegmentationPostprocessCallback
TensorsPostprocessCallback

Minimal usage

MLPostprocess postprocessing("postprocessing");
postprocessing.set_handler(my_callback);

pipeline.add("qtimltflite", "inferencing", ...)
 .add(postprocessing)
 .add_stream_filter("mlf", TextFilter())
 ...;

Input and output explained MLFrameView (read-only model tensors) Used by classification/detection/pose/depth/segmentation callbacks. Each TensorView contains:

type: tensor element type (MLTensorType)
dimensions: tensor shape
data: raw tensor memory (const uint8_t*)
size: tensor bytes

Common access pattern:

const auto& t = frame.tensors[0];
const float* data = static_cast<const float*>(t.data);

MLParam (runtime metadata and context) MLParam carries key/value fields provided by pipeline and preprocessing/postprocess context. Commonly used keys:

input-tensor-width
input-tensor-height
input-tensor-region (mapped to Region {x, y, width, height})

Example:

float source_width = 0.0f;
float source_height = 0.0f;
Region region;

params.get("input-tensor-width", source_width);
params.get("input-tensor-height", source_height);
params.get("input-tensor-region", region);

Callback signatures

Callbacks (ClassificationPostprocessCallback, ObjectDetectionPostprocessCallback, PoseEstimationPostprocessCallback, DepthEstimationPostprocessCallback, SegmentationPostprocessCallback) fill the provided output metadata container (MLClassifications, MLDetections, MLPoses, MLDepthMaps, MLSegmentations).
Tensor callback (TensorsPostprocessCallback) fills MLFrame& output for downstream tensor stages.
Input MLFrameView is read-only and must not be modified.

Classification callback

using ClassificationPostprocessCallback =
 std::function<bool(const MLFrameView&, const MLParam&, MLClassifications&)>;

Output: fill MLClassifications with MLClassification items. Field meanings:

name: class label text
confidence: prediction confidence
color: optional UI/overlay color
extra: optional metadata map

Object detection callback

using ObjectDetectionPostprocessCallback =
 std::function<bool(const MLFrameView&, const MLParam&, MLDetections&)>;

Output: fill MLDetections with MLDetection items. Field meanings:

left, top, right, bottom: normalized box coordinates
name, confidence, color: class info
landmarks: optional keypoints
extra: optional metadata map

Implemented example includes confidence thresholding, region coordinate transform, and NMS. Segmentation callback

using SegmentationPostprocessCallback =
 std::function<bool(const MLFrameView&, const MLParam&, MLSegmentations&)>;

Input parameters

frame (const MLFrameView&): read-only model output tensors.
params (const MLParam&): runtime metadata (for example: input-tensor-width, input-tensor-height, input-tensor-region).
segmentations (MLSegmentations&): output container to fill.

Output contract (MLSegmentation)

labels: per-pixel semantic labels.
colors: per-pixel colors; index-aligned with labels.
n_rows, n_columns: segmentation map dimensions.
extra: optional metadata fields.

Pose estimation callback

using PoseEstimationPostprocessCallback =
 std::function<bool(const MLFrameView&, const MLParam&, MLPoses&)>;

Output: fill MLPoses with MLPose items. Field meanings:

keypoints: list of MLKeypoint (name, x, y, confidence, color)
links: list of MLKeypointLink describing skeleton connections
name, confidence, extra: pose-level metadata

Depth estimation callback

using DepthEstimationPostprocessCallback =
 std::function<bool(const MLFrameView&, const MLParam&, MLDepthMaps&)>;

Output: fill MLDepthMaps with one or more MLDepthMap items. Field meanings:

values: per-pixel depth values (double)
colors: per-pixel pseudo-color values (uint32_t)
n_rows, n_columns: depth map shape
extra: optional metadata map

Tensor-to-tensor callback

using TensorsPostprocessCallback =
 std::function<bool(const MLFrameView&, const MLParam&, MLFrame&)>;

output is owned and writable; callback typically:

validates input tensors from frame
copies/selects required tensors into output
updates output tensor dimensions/order if needed

Semantic callbacks (classification, detection, pose, depth, segmentation)

bool callback(const MLFrameView& frame,
                         const MLParam& params,
                         OutputType& output)

frame: input tensors from inference (read-only)
params: runtime metadata (input sizes, region, etc.)
output: semantic objects to be filled by callback
return true on valid output
return false on validation/decode failure

Tensor callback

bool callback(const MLFrameView& frame,
                         const MLParam& params,
                         MLFrame& output)

frame: input tensors from inference (read-only)
params: runtime metadata/context
output: writable output tensors for downstream tensor stage
return true on valid output
return false on validation/decode failure

Copy-ready callback skeletons Classification

ClassificationPostprocessCallback cb =
         [](const MLFrameView& frame, const MLParam& params,
              MLClassifications& out) {
              // decode tensors -> classes
              return true;
         };

Detection

ObjectDetectionPostprocessCallback cb =
         [](const MLFrameView& frame, const MLParam& params,
              MLDetections& out) {
              // decode tensors -> detections
              return true;
         };

Pose estimation

PoseEstimationPostprocessCallback cb =
         [](const MLFrameView& frame, const MLParam& params,
              MLPoses& out) {
              // decode tensors -> poses
              return true;
         };

Depth estimation

DepthEstimationPostprocessCallback cb =
         [](const MLFrameView& frame, const MLParam& params,
              MLDepthMaps& out) {
              // decode tensors -> depthmaps
              return true;
         };

Segmentation

SegmentationPostprocessCallback cb =
         [](const MLFrameView& frame, const MLParam& params,
              MLSegmentations& out) {
              // decode tensors -> segmentations
              return true;
         };

Tensors

TensorsPostprocessCallback cb =
         [](const MLFrameView& frame, const MLParam& params,
              MLFrame& output) {
              // map/copy selected tensors from frame -> output
              return true;
         };

Try me

Check application source code on GitHub: qimsdk_ref_external_postprocess_detection_yolov8Pre-built application on device: /usr/bin/qimsdk_ref_external_postprocess_detection_yolov8

Download Required Files

File	Save as
Model	yolov8_det_quantized.tflite
yolov8.json	yolov8.json
ai_demo_sample.mp4	ai_demo_sample.mp4

If the downloaded model file is a .zip archive, extract it on your host machine before copying: unzip filename.zipSome AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn’t always sufficient.

Copy Files to Device

# Replace <user> and <device-ip> with your device credentials.
# ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).

ssh <user>@<device-ip> "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
scp yolov8_det_quantized.tflite <user>@<device-ip>:~/Downloads/qimsdk_samples/models/
scp yolov8.json                 <user>@<device-ip>:~/Downloads/qimsdk_samples/labels/
scp ai_demo_sample.mp4          <user>@<device-ip>:~/Downloads/qimsdk_samples/media/

Run the application

/usr/bin/qimsdk_ref_external_postprocess_detection_yolov8

The application runs inference on the video file and uses the custom, application-defined postprocessing callback to decode detections, which are then overlaid and displayed fullscreen.To stop the application, press CTRL + C.

Getting Started

Examples

Design

Plugins

Pipeline SDK Design

APIs

Pipeline

Element

AppSrc

AppSink

CamSrc

MLVConverter

MLPostprocess

ML inference bins

Buffer

StreamFilter

VideoFilter

H264Filter

TensorFilter

TextFilter

ImageFilter

AudioFilter

Logging

QIM SDK Nodes

Common Application Design

Create pipeline with config file

Multi-Model Daisy Chain

Custom Postprocessing with YOLOv8 (Detection)

Download Required Files

​Pipeline SDK Design

​APIs

​Pipeline

​Element

​AppSrc

​AppSink

​CamSrc

​MLVConverter

​MLPostprocess

​ML inference bins

​Buffer

​StreamFilter

​VideoFilter

​H264Filter

​TensorFilter

​TextFilter

​ImageFilter

​AudioFilter

​Logging

​QIM SDK Nodes

​Common Application Design

​Create pipeline with config file

​Multi-Model Daisy Chain

​Custom Postprocessing with YOLOv8 (Detection)

​Download Required Files

Pipeline SDK Design

APIs

Pipeline

Element

AppSrc

AppSink

CamSrc

MLVConverter

MLPostprocess

ML inference bins

Buffer

StreamFilter

VideoFilter

H264Filter

TensorFilter

TextFilter

ImageFilter

AudioFilter

Logging

QIM SDK Nodes

Common Application Design

Create pipeline with config file

Multi-Model Daisy Chain

Custom Postprocessing with YOLOv8 (Detection)

Download Required Files