> ## Documentation Index
> Fetch the complete documentation index at: https://imsdkdocs.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.
# SDK Architecture and API Reference
> Design and API details of the QIM SDK Pipeline SDK (C++ and Python)
## Pipeline SDK Design
**Overview** This SDK delivers a streamlined, modern interfaces, engineered to facilitate the rapid development, execution, and management of sophisticated multimedia pipelines. By implementing a high-level abstraction layer, the SDK encapsulates complex underlying GStreamer based mechanisms — including element negotiation, signal management, event loops, and state transitions — allowing engineers to prioritize core application logic over boilerplate framework intricacies.
**Key Capabilities** The wrapper empowers developers to architect advanced multimedia and AI-enhanced workflows with precision and efficiency. It supports the seamless integration of diverse components, such as video capture sources, signal processing nodes, machine learning inference engines, graphical overlays, and rendering outputs. Uniquely designed to decouple implementation from the underlying framework, the SDK eliminates the need for deep GStreamer expertise or compile-time dependencies on GStreamer headers, ensuring a clean, robust, and accessible development experience.
## APIs
**Public APIs:**
* `Pipeline` — build & control pipelines fluently.
* `Element` — generic element wrapper with variadic property setting, and state control. \[it can be instantiated at create time as well as at runtime to update the pipeline.] Some element has special functions which can be controlled by dedicated elements types. Those elements derived from `Element` can be use as `Element` if special functions are not used.
* `AppSrc` — app-facing source for zero-copy buffer pull. \[Providing the buffer from application to pipeline using standard lambdas, push APIs, EOS etc]
* `AppSink` — app-facing sink for zero-copy buffer push \[Providing the buffer from pipeline to application with zero copy]
* `CamSrc` - Build-in camera element which expose API for capture images \[provides functionality to take snapshots at runtime]
* `MLVConverter` - ML video converter (`qtimlvconverter`). Expose API for custom pre-processing implementation at application utilizing standard lambdas.
* `MLPostprocess` - ML postprocess element (`qtimlpostprocess`). Expose API for custom post-processing implementation at application utilizing standard lambdas. The callback is typed per ML task (classification, object detection, pose estimation, depth estimation, segmentation, raw tensors).
* `MLVideoONNXBin` - ONNX inference bin (`qtimlvideoonnxbin`). Expose API for custom pre-processing and post-processing implementation at application utilizing standard lambdas.
* `MLVideoQNNBin` - QNN inference bin (`qtimlvideoqnnbin`). Expose API for custom pre-processing and post-processing implementation at application utilizing standard lambdas.
* `MLVideoSNPEBin` - SNPE inference bin (`qtimlvideosnpebin`). Expose API for custom pre-processing and post-processing implementation at application utilizing standard lambdas.
* `MLVideoTFLiteBin` - TFLite inference bin (`qtimlvideotflitebin`). Expose API for custom pre-processing and post-processing implementation at application utilizing standard lambdas.
* `Buffer` — opaque buffer that bridges Native Buffer with timestamp helpers and zero-copy paths.
* `StreamFilter` and derived concrete stream helpers: `VideoFilter` (all RAW formats - YUV, RGB, bayer), `H264Filter`, `TensorFilter`, `TextFilter`, `AudioFilter`, `ImageFilter`. these filters are used to set characteristics of the streams.
* Logging controls — `SetImsdkLogLevel()` and `SetImsdkGstLogMode()` select the SDK's log verbosity and whether GStreamer's output is simplified or passed through raw. See [Logging and Diagnostics](/app-builder/cpp-logging).
**Reference flows**
### Pipeline
**Purpose**
A convenient, exception-safe wrapper around a `GstPipeline` that:
* Builds pipelines using element factory names or pre-constructed wrapper elements.
* Manages pipeline lifecycle and execution state.
* Performs delayed linking for dynamic/request-pad elements automatically via `pad-added`.
**Key Responsibilities**
* Create and own the underlying `GstPipeline` instance.
* Add elements, assign properties, and insert caps filters.
* Link elements (immediately when possible, deferred otherwise).
* Manage the full pipeline lifecycle (`prepare`, `start`, `wait`, `stop`).
* Install bus watches to handle `ERROR`, `EOS`, and state transitions.
* Integrate with runtime shutdown notifications.
**A minimal pipeline**
Add the elements in the order data flows through them, then run:
```cpp theme={null}
Pipeline pipeline("my-pipeline");
pipeline
.add("filesrc", "src", "location", "/home/user/media/video.mp4")
.add("qtdemux", "demux")
.add("h264parse", "parse")
.add("v4l2h264dec", "decoder", "output-io-mode", 4, "capture-io-mode", 4)
.add_stream_filter("vf", VideoFilter().format("NV12"))
.add("waylandsink", "display", "fullscreen", true)
.execute();
```
`execute()` starts the pipeline, blocks until EOS/error/termination, and tears it down.
**Adjacent elements are linked in add order.** In the example above `src` → `demux` → `parse` → `decoder` → `vf` → `display` are linked simply because that is the order of the `add()` calls, so a linear graph needs no explicit linking. Use [`link()`](#branching-with-explicit-links) when the graph is not linear — a `tee` splitting into a display branch and an inference branch, or a `qtimetamux` rejoining them.
`qtdemux` is an example of the delayed linking described above: its src pads appear only once the container has been parsed, so the `demux` → `parse` link is completed automatically when the pad shows up. No `pad-added` handling is needed in application code.
**Building the pipeline**
| Method | Description |
| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Pipeline(const std::string &name)` | Creates an empty pipeline. `name` is the `GstPipeline` name, used in log messages. |
| `Pipeline(const std::string &name, const std::string &config)` | Builds a complete pipeline from a YAML description. `config` is the YAML **content**, not a path. See [Describing a pipeline in YAML](#describing-a-pipeline-in-yaml). |
| `Pipeline &add(String factory, String name, ...)` | Creates an element from the factory name and adds it under a unique instance name. Trailing arguments are property key/value pairs, applied immediately. |
| `Pipeline &add(const Element &element)` | Adds an externally constructed element. Required for the specialized wrappers (`AppSrc`, `AppSink`, `CamSrc`, `MLVConverter`, `MLPostprocess`, the ML bins) so callbacks can be attached before the element joins the pipeline. |
| `Pipeline &add_stream_filter(String name, const StreamFilter &filter)` | Inserts a caps filter between the previous and next element. When neighbouring elements have several formats in common, this fixates the one you want instead of leaving the choice to caps negotiation. See [StreamFilter](#streamfilter). |
| `Pipeline &link(names...)` | Links the named elements in the given order. Needed only for non-linear graphs. |
The optional key/value pairs passed to `add()` are the element's properties, and they are validated as soon as the element is added. Setting a property the element does not have throws `std::invalid_argument` naming the property, the element instance, and the element type — so a typo surfaces immediately at that `add()` call instead of being silently ignored:
```cpp theme={null}
// "fulscreen" is not a waylandsink property -> throws std::invalid_argument
pipeline.add("waylandsink", "display", "fulscreen", true);
```
**Branching with explicit links**
Add order cannot express a branch, so name the chains explicitly. Element instance names are the identifiers.
```cpp theme={null}
pipeline
.add("filesrc", "src", "location", path)
.add("qtdemux", "demux")
.add("h264parse", "parse")
.add("v4l2h264dec", "decoder", "output-io-mode", 4, "capture-io-mode", 4)
.add_stream_filter("vf", VideoFilter().format("NV12"))
.add("tee", "split")
.add("queue", "q1")
.add("qtimlvconverter", "preprocessing")
.add("qtimltflite", "inferencing", "model", model_path)
.add("qtimlpostprocess", "postprocessing", "module", "yolov8")
.add_stream_filter("mlf", TextFilter())
.add("qtimetamux", "mlmuxer")
.add("qtivoverlay", "overlay")
.add("waylandsink", "display", "fullscreen", true)
// trunk, up to the tee
.link("src", "demux", "parse", "decoder", "vf", "split")
// branch 1: raw video to the metadata muxer
.link("split", "mlmuxer")
// branch 2: inference path, rejoining at the muxer
.link("split", "q1", "preprocessing", "inferencing", "postprocessing",
"mlf", "mlmuxer", "overlay", "display");
```
Elements whose src pads are `SOMETIMES` or `REQUEST` — `qtdemux`, `tsdemux`, `tee`, `qtivsplit` — cannot be linked at construction time, because the pads do not exist yet. The pipeline detects this from the pad templates, defers the link, and completes it from a `pad-added` handler it installs itself. Request pads on the sink side (for example `qtimetamux`, `qtivcomposer`) are requested and released automatically. No `pad-added` callback is needed in application code.
A link between two elements that both have only static pads and no common caps fails immediately with `std::runtime_error` naming both elements. A deferred link that never resolves surfaces at run time instead; on error the pipeline logs which pads stayed unlinked.
**Running the pipeline**
A pipeline is always in one of three states, and the lifecycle methods are the transitions between them:
| State | Meaning |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `NULL` | No resources allocated. Initial state, and the state after `stop()`. Devices are closed, buffer pools released. |
| `PAUSED` | Elements are allocated and caps are negotiated between linked pads, but the clock is not running and no data flows. Sinks have pre-rolled. This is the state in which elements can be added, removed, or relinked. |
| `PLAYING` | The clock is running and buffers flow through the graph. |
`execute()` performs the whole `NULL → PLAYING → NULL` cycle, so most applications never call the individual transitions. `prepare()`, `activate()`, and `deactivate()` — the transitions through `PAUSED` shown above — are covered in [Advanced state control](#advanced-state-control).
| Method | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `void execute()` | `start()`, `wait()`, `stop()` in sequence. The normal entry point. |
| `Pipeline &start()` | Sets the pipeline to `PLAYING`. Non-blocking: the state change is asynchronous and this returns without waiting for it. Throws `std::runtime_error` if the transition fails. |
| `Pipeline &wait()` | Blocks until the pipeline is done: `EOS` on the bus, `ERROR` on the bus, `stop()` from another thread, or `SIGINT`/`SIGTERM`. |
| `Pipeline &stop()` | Sets the pipeline to `NULL` and releases resources. Blocks until teardown completes. |
| `Pipeline &eos(bool enabled)` | When enabled, `stop()` first sends an end-of-stream event through the pipeline and waits for it to come back from the sinks, so that data already inside the pipeline is processed before teardown. Off by default. See [Draining the pipeline on shutdown](#draining-the-pipeline-on-shutdown). |
`start()` is non-blocking, so it must be paired with `wait()` (or your own blocking logic). A `main()` that calls `start()` and returns destroys the pipeline before buffers flow.
To run the pipeline while the application does other work, split `execute()` apart:
```cpp theme={null}
pipeline.start(); // returns immediately
// ... application work, runtime property changes, snapshots ...
pipeline.wait(); // block until EOS / error / Ctrl+C
pipeline.stop();
```
**Draining the pipeline on shutdown**
By default `stop()` tears the pipeline down as fast as it can. Every element goes to `NULL` immediately and whatever buffers are still in flight — queued between elements, held inside an encoder, not yet written by a sink — are discarded. This is usually what you want: the pipeline stops without delay.
Some pipelines must not lose those buffers, and for them `eos(true)` changes the shutdown sequence. Instead of stopping straight away, `stop()` sends an end-of-stream event into the pipeline. The event travels downstream element by element; each element finishes processing the data it has already accepted, passes the event on, and the sinks report it back on the bus. Only once that end-of-stream notification comes back does `stop()` tear the pipeline down. Nothing that had already entered the pipeline is dropped.
The trade-off is shutdown latency: `stop()` now blocks until the last buffer has made it through the whole graph, which takes as long as the remaining data needs.
Use `eos(true)` when the tail of the stream is part of the result:
* **Muxed recordings.** Containers such as MP4 and MKV are written by a muxer (`mp4mux`, `matroskamux`) that writes its index and header metadata — the `moov` atom, in MP4's case — only on end-of-stream. Cut it off early and the file is left incomplete, often unplayable rather than merely short.
* **Encoders with buffered frames.** Encoders keep several frames in flight for B-frames, lookahead, and similar optimizations. End-of-stream is what makes the encoder flush those frames downstream; without it the tail of the recording is simply missing.
* **Any sink where losing the final buffered data is unacceptable** — file sinks, muxed outputs, and applications that expect to receive every frame through `AppSink`.
Not every sink needs draining, though. A live network stream can usually just be cut off, and a receiver will not care that the last few buffers never arrived. The question is not whether a sink writes somewhere, but whether anything downstream depends on the data still sitting in the pipeline.
Leave `eos()` off (the default) when the newest data is all that matters and stopping promptly is preferable — live preview to a display, a camera feed on screen, or writing self-contained files where each buffer is already complete on its own (JPEG frames through `multifilesink`, for example, which needs no finalization).
```cpp theme={null}
Pipeline pipeline("cam-encoder-pipeline");
pipeline
.add("qtiqmmfsrc", "source")
.add_stream_filter("vf", VideoFilter().format("NV12")
.resolution(1920, 1080)
.framerate(30))
.add("v4l2h264enc", "encoder",
"output-io-mode", "dmabuf-import", "capture-io-mode", "dmabuf")
.add("h264parse", "parser")
.add("mp4mux", "muxer")
.add("filesink", "sink", "location", HOME_PATH + "/media/output.mp4")
.eos(true) // drain, so mp4mux can finalize the container
.execute();
```
If the pipeline has already reached end-of-stream on its own — a file source that ran out of data, for instance — `stop()` detects it and tears down immediately instead of sending a second event and waiting again. Enabling `eos(true)` therefore costs nothing for pipelines that end naturally; it only matters when you stop a pipeline that is still running.
**Advanced state control**
For reconfiguring a loaded pipeline — adding, removing, or relinking elements without returning to `NULL`.
| Method | Description |
| ------------------------ | ----------------------------------------------------------------------------------------------- |
| `Pipeline &prepare()` | `NULL` → `PAUSED`. Allocates resources and negotiates caps without running the clock. Blocking. |
| `Pipeline &activate()` | `PAUSED` → `PLAYING`. |
| `Pipeline &deactivate()` | `PLAYING` → `PAUSED`. Resources stay allocated. |
**Accessing elements at run time**
| Method | Description |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Element get(String name)` | Returns a wrapper for the named element, for runtime property changes and per-pad configuration. Throws `std::runtime_error` if the name is not in the pipeline. |
| `T get(String name)` | Returns a typed wrapper, exposing that type's element-specific API — for example `get("source")` for the image-capture calls. Use it when the element was added by factory name but its specialized interface is needed. |
**Runtime property and per-pad configuration** — `get()` reaches any element by name while the pipeline is loaded or playing:
```cpp theme={null}
// Change a property at run time.
pipeline.get("display").set("fullscreen", false);
// Per-pad properties on a request pad of qtivcomposer.
pipeline.get("composer").input(1).set("alpha", 0.5);
```
**Typed access to an element's own API** — `get()` returns the specialized wrapper, so element-specific calls become available on an element that was added by factory name:
```cpp theme={null}
// qtiqmmfsrc was added as "source"; CamSrc exposes its capture API.
auto cam = pipeline.get("source");
cam.image_capture();
```
See the sample applications for complete image-capture pipelines.
`get()` returns an element wrapper **by value**; each call yields a new wrapper referencing the same `GstElement`. Bind it with `auto`, not `auto&`.
**Describing a pipeline in YAML**
The graph — elements, properties, stream filters, and links — can be described in YAML instead of in code. This separates *what the pipeline is* from *what the application does with it*: the same binary can run a different topology, a different model, or a different source by pointing it at another configuration file.
That makes it useful where the pipeline is not fixed at build time:
* **Deploying one binary across variants** — the same application against a built-in camera, a USB camera, or an RTSP stream, switching source and format per device.
* **Tuning without rebuilding** — resolution, framerate, model path, or delegate changed on the target, which matters when a rebuild means a full cross-compile and reflash.
* **Keeping topology under review** — the pipeline shape lives in a file that can be diffed and versioned on its own.
Loading from YAML does not give up application logic: the elements are ordinary pipeline members afterwards, so `get()` and `get()` reach them as usual.
```yaml theme={null}
pipeline:
elements:
- type: filesrc
name: src
location: ~/media/video.mp4
- type: qtdemux
name: demux
- type: h264parse
name: parse
- type: v4l2h264dec
name: decoder
output-io-mode: 4
capture-io-mode: 4
- type: filter
name: videofilter
video:
format: NV12
- type: waylandsink
name: display
fullscreen: true
```
Everything lives under a single root key, `pipeline:`, which takes three independent child keys:
```yaml theme={null}
pipeline:
eos: true # optional — EOS-on-shutdown, same as eos(true) in code
elements: # required — the elements, in add order
- type: ...
links: # optional — explicit links, for non-linear graphs
- [ ... ]
```
| Key | Required | Description |
| ---------- | -------- | -------------------------------------------------------------------------------------------------------------------------- |
| `eos` | No | Boolean. Equivalent to calling [`eos(bool)`](#draining-the-pipeline-on-shutdown). Defaults to false when omitted. |
| `elements` | Yes | Sequence of elements. Add order is link order, exactly as with `add()`. |
| `links` | No | Sequence of chains, each a list of element names. Equivalent to one `link()` call per entry. Omit it for linear pipelines. |
Within `elements:`, each entry needs `type` (the factory name) and `name` (the instance name); every remaining key is applied as a property of that element. An entry with `type: filter` inserts a stream filter instead, described by a `video:`, `image:`, `h264:`, `tensor:`, `text:`, `audio:`, or raw `caps:` block. Any of those blocks may carry an `add:` sequence of `key=value` strings, appended to the filter's caps through [`add()`](#videofilter) — the same escape-hatch the typed filters expose in code. It is accepted both inside the filter block and alongside it:
```yaml theme={null}
- type: filter
name: videofilter
video:
format: NV12
add:
- colorimetry=bt709
```
`links:` mirrors [`link()`](#branching-with-explicit-links) — one sequence per chain, so the branching pipeline shown earlier becomes:
```yaml theme={null}
links:
- [split, mlmuxer]
- [source, videostream, split, q1, preprocessing, inferencing,
postprocessing, mlf, mlmuxer, overlay, display]
```
The constructor takes the YAML content, so read the file first:
```cpp theme={null}
std::ifstream input(config_path);
std::ostringstream buffer;
buffer << input.rdbuf();
Pipeline pipeline("demo-pipeline", buffer.str());
pipeline.execute();
```
The names in the configuration are the handles the application uses afterwards. Given a config that declares a `qtimlpostprocess` element named `postprocessing`, the application can claim its post-processing without the topology being hard-coded:
```cpp theme={null}
Pipeline pipeline("demo-pipeline", buffer.str());
auto postprocessing = pipeline.get("postprocessing");
postprocessing.set_handler([](const MLFrame& frame, const MLParam& params,
MLDetections& detections) {
return decode_detection(frame, params, detections);
});
pipeline.execute();
```
### Element
**Purpose**
Generic wrapper around a `GstElement` with fluent, variadic property setting. Can be instantiated at create time or used at runtime to update a live pipeline. Serves as the base class for all specialized element types.
**Key Responsibilities**
* Create GStreamer elements from a factory name.
* Provide fluent, variadic property setting for all common types and `StreamFilter` (caps).
* Implement pad linking/unlinking with error checking.
* Support state publication to pipeline (`sync`, `deactivate`, `stop`).
* Expose per-pad configuration through `Port`.
* Serve as the base class for specialized wrappers.
```cpp theme={null}
Element display("waylandsink", "display");
display.set("fullscreen", true);
display.set("sync", false);
```
Any element can be constructed directly and handed to `Pipeline::add()`, or reached later by name with `Pipeline::get()`. Elements cannot be copied — only moved — so pass them by reference or move them where ownership has to change.
| `qti::Element` | Description |
| ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Element(const std::string &factory, const std::string &name = {})` | Creates the element from a GStreamer factory name. The instance name is optional but recommended, since it is how the element is addressed in `link()`, `get()`, and log messages. |
| `Element &set(String prop, Value value, ...)` | Sets one or more properties. Accepts any number of key/value pairs in a single call and returns `*this`, so calls can be chained. |
| `Element &link(Element &downstream, String src_pad = "src", String sink_pad = "sink")` | Links this element's src pad to `downstream`'s sink pad. Throws `std::runtime_error` naming both pads if the link fails. |
| `Element &unlink(Element &downstream, String src_pad = "src", String sink_pad = "sink")` | Unlinks the named pad pair between this element and `downstream`. |
| `Element &unlink(String src_pad = "src")` | Unlinks whatever is connected to the given src pad, without naming the peer. |
| `Element &sync()` | Synchronizes the element's state with its parent pipeline's state. Call this after adding an element to a pipeline that is already running. |
| `Element &deactivate()` | Moves the element to `PAUSED`. |
| `Element &stop()` | Moves the element to `NULL`. |
| `Port input(unsigned int id)` | Returns the sink pad at the given index, for per-pad configuration. |
| `Port input(String name_or_type, unsigned int id)` | Returns the sink pad matching a pad-template name and index. |
| `Port output(unsigned int id)` | Returns the src pad at the given index. |
| `Port output(String name_or_type, unsigned int id)` | Returns the src pad matching a pad-template name and index. |
**Setting properties**
`set()` is variadic, so a whole configuration can go in one call, and property names are validated immediately — an unknown property throws `std::invalid_argument` naming the property and the element type rather than being silently ignored:
```cpp theme={null}
Element decoder("v4l2h264dec", "decoder");
decoder.set("output-io-mode", 4, "capture-io-mode", 4);
```
Besides the usual scalar types and strings, `set()` accepts a `StreamFilter` wherever an element takes a caps property, and plain enums (converted to their underlying integer):
```cpp theme={null}
AppSrc appsrc("appsrc");
appsrc.set("caps", VideoFilter().format("NV12")
.resolution(1920, 1080)
.framerate(30));
```
**Per-pad configuration with Port**
Some elements are configured per pad rather than per element — a compositor's inputs each have their own position, size, and alpha. `input()` and `output()` return a `Port` for one pad, which carries the same fluent, validated `set()`:
```cpp theme={null}
// Second input of qtivcomposer: blend at 50% alpha.
pipeline.get("composer").input(1).set("alpha", 0.5);
```
| `qti::Port` | Description |
| ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Port &set(String prop, Value value, ...)` | Sets one or more pad properties. Validated like `Element::set()`, and additionally accepts `std::vector` and `{a, b}` initializer lists for array-valued pad properties such as positions and dimensions. |
The `id` selects which pad, and `name_or_type` narrows it to a pad template when an element has several families of pads. A template containing `%u` is formatted with the index (`"image_%u"` with `id` 0 gives `image_0`); otherwise the name and index are joined with an underscore (`"sink"` with `id` 1 gives `sink_1`). Without a name, the pad is matched by numeric suffix first and by position among same-direction pads second. An unresolvable pad throws `std::runtime_error`.
A `Port` refers to a pad of a live element and cannot be copied, only moved. Get it, set the properties, and let it go — as in the one-liner above — rather than storing it beyond the lifetime of the element.
**Runtime changes**
Because `Pipeline::get()` returns an `Element` for any element in the pipeline, the same `set()` works while the pipeline is playing:
```cpp theme={null}
pipeline.get("display").set("fullscreen", false);
```
`sync()`, `deactivate()`, and `stop()` exist for reconfiguring a live pipeline. An element added to an already-running pipeline starts in `NULL` while the pipeline is `PLAYING`, so it will not process data until its state is brought up to match — that is what `sync()` does. `deactivate()` and `stop()` take a single element down without touching the rest of the graph.
Runtime relinking is an advanced use of the API. Pad linking, unlinking, and per-element state changes on a live pipeline must respect GStreamer's rules about when pads may be changed; the wrappers report failures as exceptions but do not remove the need to get the sequence right.
### AppSrc
**Purpose**
`AppSrc` wraps the `appsrc` element, the entry point for feeding application-produced buffers into a pipeline. It manages the `need-data`/`enough-data` signal plumbing so the application only has to supply buffers, either from a callback or by pushing them directly.
**Key Responsibilities**
* Validate that the wrapped element is indeed an `appsrc`.
* Register and dispatch the underlying `need-data` and `enough-data` signals.
* Let the application supply buffers on demand through `set_buffer_producer()`, or push them directly with `push_buffer()`.
* Signal end-of-stream on this element's output.
* Manage lifetime of the connected signal handlers.
```cpp theme={null}
AppSrc appsrc("src");
appsrc.set("is-live", true)
.set("block", true)
.set("format", AppSrc::Format::TIME)
.set("do-timestamp", true)
.set("caps", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30));
```
| `qti::AppSrc` | Description |
| ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `AppSrc(const std::string &name = {})` | Creates an `appsrc` element with an optional instance name. |
| `AppSrc &set(String prop, Value value, ...)` | Sets one or more properties, same as `Element::set()`. |
| `AppSrc &set_buffer_producer(std::function producer)` | Registers a callback invoked whenever appsrc needs more data. Fill the buffer and return `true` to push it, `false` to skip this round. |
| `AppSrc &set_enough_handler(std::function enough)` | Registers a callback invoked when appsrc reports it is holding enough queued data. |
| `bool push_buffer(qti::Buffer &buffer)` / `bool push_buffer(qti::Buffer &&buffer)` | Pushes a buffer into the stream directly, outside the `need-data` callback. Returns `true` once it is accepted downstream. |
| `void end_of_stream()` | Sends an end-of-stream event on this element's output. |
`AppSrc::Format` mirrors appsrc's own `format` property (`DEFAULT`, `BYTES`, `TIME`, `BUFFERS`, `PERCENT`); `TIME`, used above, is the usual choice for timestamped media.
`push_buffer()` transfers a buffer's underlying `GstBuffer` without copying when the `Buffer` already owns one — which is exactly the case for a buffer just pulled from an `AppSink`. A `Buffer` filled with raw data instead (as inside a `set_buffer_producer()` callback) is copied into a new `GstBuffer` on push.
### AppSink
**Purpose**
`AppSink` wraps the `appsink` element, the exit point that delivers pipeline output back to application code. It pulls samples through appsink's action signals and hands them to the application as `Buffer`.
**Key Responsibilities**
* Validate that the wrapped element is indeed an `appsink`.
* Pull samples via appsink's action signals and convert them into `Buffer`.
* Deliver each buffer to the application through `set_buffer_consumer()`.
* Deliver the first (preroll) sample separately through `set_preroll_handler()`.
* Notify the application of end-of-stream through `set_eos_handler()`.
* Manage lifetime of the connected signal handlers.
```cpp theme={null}
AppSink appsink("sink");
appsink.set("emit-signals", true)
.set("max-buffers", 5)
.set("drop", true)
.set_buffer_consumer([](Buffer buf) {
// consume buf.data(), buf.size(), buf.pts()
});
```
| `qti::AppSink` | Description |
| --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `AppSink(const std::string &name = {})` | Creates an `appsink` element with an optional instance name. |
| `AppSink &set(String prop, Value value, ...)` | Sets one or more properties, same as `Element::set()`. |
| `AppSink &set_buffer_consumer(std::function consumer)` | Registers a callback invoked for every buffer pulled from the pipeline. |
| `AppSink &set_preroll_handler(std::function preroll)` | Registers a callback invoked for the first (preroll) sample only; return `false` to report an error instead of accepting it. |
| `AppSink &set_eos_handler(std::function eos)` | Registers a callback invoked when appsink receives end-of-stream. |
**Bridging two pipelines**
Because `AppSink` and `AppSrc` are ordinary elements, they can sit in two independent `Pipeline` objects and be wired together in application code — useful when one graph produces frames (say, from a camera) and another consumes them (say, encoding or display) on its own clock:
```cpp theme={null}
AppSrc appsrc("src");
appsrc.set("is-live", true)
.set("block", true)
.set("format", AppSrc::Format::TIME)
.set("do-timestamp", true)
.set("caps", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30));
AppSink appsink("sink");
appsink.set("emit-signals", true)
.set("max-buffers", 5)
.set("drop", true)
.set_buffer_consumer([&](Buffer b) {
appsrc.push_buffer(std::move(b));
});
Pipeline pipeline1("p1");
pipeline1.add("videotestsrc", "testsrc", "is-live", true, "pattern", "ball")
.add_stream_filter("videostream", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
.add(appsink);
Pipeline pipeline2("p2");
pipeline2.add(appsrc)
.add("waylandsink", "display", "fullscreen", true);
```
Each buffer handed to `set_buffer_consumer()` on `pipeline1` is pushed straight into `pipeline2` through `appsrc.push_buffer()`, without a copy.
### CamSrc
**Purpose**
`CamSrc` wraps the `qtiqmmfsrc` camera source element and exposes its still/burst image-capture action signals as ordinary method calls, so an application can trigger a snapshot without touching GStreamer signals directly.
**Key Responsibilities**
* Validate that the wrapped element is indeed a `qtiqmmfsrc`.
* Trigger still or burst image capture through `image_capture()`.
* Cancel any pending capture request through `cancel_capture()`.
| `qti::CamSrc` | Description |
| --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| `CamSrc(const std::string &name = {})` | Creates a `qtiqmmfsrc` element with an optional instance name. |
| `bool image_capture(unsigned int count = 1)` | Captures `count` still images. |
| `bool image_capture(CaptureMode mode, unsigned int count = 1)` | Captures `count` images using `CaptureMode::kStill` or `CaptureMode::kBurst`. |
| `bool image_capture(CaptureMode mode, unsigned int count, const std::vector &metadata_ptrs)` | Same as above, attaching a per-request metadata payload to each capture. |
| `bool cancel_capture()` | Cancels any capture requests still pending. |
A running pipeline built from a plain factory name still has a `qtiqmmfsrc` element in it; use `Pipeline::get()` to reach it as a `CamSrc` and call its capture API:
```cpp theme={null}
auto cam = pipeline.get("source");
cam.image_capture();
```
See `examples/test_camera_and_capture` for a complete pipeline that branches a `qtiqmmfsrc` into a live display and a `multifilesink` for captured images.
### MLVConverter
**Purpose**
`MLVConverter` wraps the `qtimlvconverter` element, which converts raw video frames into the input tensor format expected by a model. Beyond the properties available on the generic `Element`, it lets the application replace the built-in conversion with its own implementation through a lambda.
**Key Responsibilities**
* Validate that the wrapped element is a `qtimlvconverter`.
* Convert incoming video frames into model input tensors.
* Optionally delegate the conversion to an application-supplied pre-process callback.
* Provide fluent, variadic property setting that returns `MLVConverter&` for chaining.
**Pre-process callback**
The callback receives the source frames as `MLVideoBlits` (each entry carries the image planes plus the source quadrilateral, destination rectangle, alpha and rotation) and must fill the tensors of the `MLFrame` output. Returning `false` marks the conversion as failed.
```cpp theme={null}
using TensorsPreprocessCallback =
std::function;
```
Set `engine` to `none` to disable the internal pre-processing path when you supply your own callback — otherwise the built-in conversion runs instead.
**Usage**
```cpp theme={null}
MLVConverter preprocessing("preprocessing");
// engine=none disables internal preprocessing path in qtimlvconverter.
preprocessing.set("engine", "none");
preprocessing.set_handler(
[](const MLVideoBlits& blits, MLFrame& output) {
if (blits.entries.empty() || output.tensors.empty())
return false;
const MLVideoBlit& blit = blits.entries.front();
return convert_nv12_to_nhwc_i8(blit.image, &blit,
output.tensors.front());
});
pipeline.add(preprocessing);
```
| `qti::MLVConverter` | Description |
| -------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| `MLVConverter(const std::string &name = {})` | Creates a `qtimlvconverter` element with an optional instance name. |
| `MLVConverter &set(String prop, Value value, ...)` | Sets one or more properties, same as `Element::set()`. |
| `MLVConverter &set_handler(TensorsPreprocessCallback handler)` | Registers the external pre-process callback. Requires `engine=none` to take effect. |
### MLPostprocess
**Purpose**
`MLPostprocess` wraps the `qtimlpostprocess` element, which decodes model output tensors into ML metadata attached to the frame. It lets the application replace the built-in decoding module with its own implementation through a lambda.
**Key Responsibilities**
* Validate that the wrapped element is a `qtimlpostprocess`.
* Decode output tensors into ML metadata (classifications, detections, poses, depth maps, segmentations), or pass the raw tensors through for a further stage.
* Optionally delegate the decoding to an application-supplied post-process callback.
* Select the callback type automatically from the lambda's third parameter.
* Provide fluent, variadic property setting that returns `MLPostprocess&` for chaining.
**Post-process callbacks**
`set_handler()` is overloaded once per ML task. The overload is selected by the type of the third parameter of the callback, so the same method name registers any of the supported result types. Each callback receives the model output tensors as `MLFrame`, the element parameters as `MLParam`, and an output container to fill. Returning `false` marks the frame as failed.
| Callback type | Signature | Output |
| ------------------------------------ | ---------------------------------------------------------- | -------------------------------------------------------- |
| `ClassificationPostprocessCallback` | `bool(const MLFrame&, const MLParam&, MLClassifications&)` | Class name, confidence, color |
| `ObjectDetectionPostprocessCallback` | `bool(const MLFrame&, const MLParam&, MLDetections&)` | Bounding boxes, confidence, landmarks |
| `PoseEstimationPostprocessCallback` | `bool(const MLFrame&, const MLParam&, MLPoses&)` | Keypoints and keypoint links |
| `DepthEstimationPostprocessCallback` | `bool(const MLFrame&, const MLParam&, MLDepthMaps&)` | Per-cell depth values and colors |
| `SegmentationPostprocessCallback` | `bool(const MLFrame&, const MLParam&, MLSegmentations&)` | Per-cell labels and colors |
| `TensorsPostprocessCallback` | `bool(const MLFrame&, const MLParam&, MLFrame&)` | Raw output tensors, for daisy-chaining into a next stage |
Use `MLParam::get()` to read element parameters inside the callback. For example `params.get("input-tensor-region", region)` returns the `Region` the frame was scaled into, which is needed to map model coordinates back to frame coordinates.
**Usage**
```cpp theme={null}
static const std::vector labels =
load_labels(HOME_PATH + "/labels/resnet101.json");
MLPostprocess postprocessing("postprocessing");
postprocessing
.set("results", 1)
.set_handler([](const MLFrame& frame, const MLParam& params,
MLClassifications& classifications) {
return decode_top1_classification(frame, params, labels,
classifications,
/*confidence_threshold=*/51.0f, "");
});
pipeline.add(postprocessing);
```
| `qti::MLPostprocess` | Description |
| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------ |
| `MLPostprocess(const std::string &name = {})` | Creates a `qtimlpostprocess` element with an optional instance name. |
| `MLPostprocess &set(String prop, Value value, ...)` | Sets one or more properties, same as `Element::set()`. |
| `MLPostprocess &set_handler(ClassificationPostprocessCallback handler)` | Registers a callback that decodes tensors into classifications. |
| `MLPostprocess &set_handler(ObjectDetectionPostprocessCallback handler)` | Registers a callback that decodes tensors into detections. |
| `MLPostprocess &set_handler(PoseEstimationPostprocessCallback handler)` | Registers a callback that decodes tensors into poses. |
| `MLPostprocess &set_handler(DepthEstimationPostprocessCallback handler)` | Registers a callback that decodes tensors into depth maps. |
| `MLPostprocess &set_handler(SegmentationPostprocessCallback handler)` | Registers a callback that decodes tensors into segmentations. |
| `MLPostprocess &set_handler(TensorsPostprocessCallback handler)` | Registers a callback that produces raw output tensors, for chaining into another ML stage. |
### ML inference bins
**Purpose**
The ML video bins wrap a full pre-process → inference → post-process chain into a single element, so a whole AI branch can be added to the pipeline as one node instead of being assembled from separate elements and queues. Four bins are available, one per inference runtime:
| QIM SDK C++ class | GStreamer element | Runtime |
| ------------------ | --------------------- | ------------------------------------------- |
| `MLVideoONNXBin` | `qtimlvideoonnxbin` | ONNX Runtime |
| `MLVideoQNNBin` | `qtimlvideoqnnbin` | Qualcomm® AI Engine Direct (QNN) |
| `MLVideoSNPEBin` | `qtimlvideosnpebin` | Snapdragon® Neural Processing Engine (SNPE) |
| `MLVideoTFLiteBin` | `qtimlvideotflitebin` | LiteRT (TensorFlow Lite) |
All four expose an identical API and differ only in the wrapped element and the runtime-specific properties. Each inherits from both the pre-process and the post-process base, so it accepts a custom pre-process callback, a custom post-process callback, or both.
**Key Responsibilities**
* Validate that the wrapped element matches the expected bin factory.
* Run pre-processing, inference, and post-processing inside a single element.
* Optionally delegate pre-processing to an application-supplied callback via `set_preprocess_handler()`.
* Optionally delegate post-processing to an application-supplied callback via `set_postprocess_handler()`.
* Provide fluent, variadic property setting that returns the concrete bin type for chaining.
**Properties**
Because the bin embeds several stages, its properties are prefixed by the stage they configure — `preprocess-*`, `inference-*`, and `postprocess-*`. For example `inference-model` on the bin corresponds to `model` on a standalone inference element, and `postprocess-labels` corresponds to `labels` on a standalone `qtimlpostprocess`.
Set `preprocess-engine` to `none` to disable the internal pre-processing path when you supply your own pre-process callback.
**Usage**
**Custom post-processing** — the bin handles pre-processing and inference internally, the application decodes the output tensors.
```cpp theme={null}
static const std::vector labels =
load_labels(HOME_PATH + "/labels/yolov8.json");
MLVideoTFLiteBin mlbin("mlbin");
mlbin
.set("inference-delegate", "external",
"inference-external-delegate-path", "libQnnTFLiteDelegate.so",
"inference-external-delegate-options",
"QNNExternalDelegate,backend_type=htp;",
"inference-model", HOME_PATH + "/models/yolov8_det_quantized.tflite")
.set_postprocess_handler(
[&](const MLFrame& frame, const MLParam& params,
MLDetections& detections) {
return decode_detection(frame, detections, params, labels,
/*confidence_threshold=*/0.70f);
});
Pipeline pipeline("mlbin-external-detection");
pipeline
.add("filesrc", "src", "location", HOME_PATH + "/media/video.mp4")
.add("qtdemux", "demux")
.add("h264parse", "parse")
.add("v4l2h264dec", "decoder", "output-io-mode", 4, "capture-io-mode", 4)
.add_stream_filter("vf", VideoFilter().format("NV12"))
.add(mlbin)
.add("qtivoverlay", "overlay")
.add("waylandsink", "display", "fullscreen", true);
pipeline.execute();
```
**Custom pre-processing** — the application converts frames into input tensors, the bin runs inference and its built-in post-process module.
```cpp theme={null}
MLVideoTFLiteBin mlbin("mlbin");
// preprocess-engine=none disables internal preprocessing path in mlbin.
mlbin.set("preprocess-engine", "none");
mlbin.set("inference-delegate", "external");
mlbin.set("inference-external-delegate-path", "libQnnTFLiteDelegate.so");
mlbin.set("inference-external-delegate-options",
"QNNExternalDelegate,backend_type=htp;");
mlbin.set("inference-model",
HOME_PATH + "/models/yolov8_det_quantized.tflite");
mlbin.set("postprocess-module", "yolov8");
mlbin.set("postprocess-labels", HOME_PATH + "/labels/yolov8.json");
mlbin.set_preprocess_handler(
[](const MLVideoBlits& blits, MLFrame& output) {
if (blits.entries.empty() || output.tensors.empty())
return false;
const MLVideoBlit& blit = blits.entries.front();
return convert_nv12_to_nhwc_i8(blit.image, &blit,
output.tensors.front());
});
```
Swapping runtimes is a matter of changing the class and the runtime-specific properties; the callback registration is unchanged.
```cpp theme={null}
MLVideoQNNBin qnn_bin("mlbin"); // qtimlvideoqnnbin
MLVideoSNPEBin snpe_bin("mlbin"); // qtimlvideosnpebin
MLVideoONNXBin onnx_bin("mlbin"); // qtimlvideoonnxbin
```
**APIs**
The table below applies to each of `MLVideoONNXBin`, `MLVideoQNNBin`, `MLVideoSNPEBin`, and `MLVideoTFLiteBin`; substitute the concrete class name for ``.
| `qti::` | Description |
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
| `(const std::string &name = {})` | Creates the corresponding `qtimlvideo*bin` element with an optional instance name. |
| ` &set(String prop, Value value, ...)` | Sets one or more properties, same as `Element::set()`. Properties are stage-prefixed: `preprocess-*`, `inference-*`, `postprocess-*`. |
| ` &set_preprocess_handler(TensorsPreprocessCallback handler)` | Registers the external pre-process callback. Requires `preprocess-engine=none` to take effect. |
| ` &set_postprocess_handler(Callback handler)` | Registers the external post-process callback. Accepts the same task-specific callback types as `MLPostprocess::set_handler()`. |
The bins cannot be copied, only moved — the same as the other element wrappers.
### Buffer
**Purpose**
Unified buffer abstraction over native GStreamer memory. Provides read/write access, timestamp metadata (pts, dts, duration), resize support, and zero-copy transfer between AppSink and AppSrc.
**Key Responsibilities**
* Own and manage mapping/unmapping of underlying `GstBuffer` or `GstSample`.
* Provide read/write access to buffer memory.
* Support resizing and writable allocation for AppSrc.
* Carry timestamps (PTS/DTS/duration) and apply them back to `GstBuffer`.
* Enable zero-copy extraction using `take_gst_buffer()`.
```cpp theme={null}
appsink.set_buffer_consumer([](Buffer buf) {
auto* data = buf.data();
auto pts = buf.pts();
});
```
A `Buffer` arrives either from an `AppSink` — wrapping the sample the pipeline just produced — or is allocated by the application to be filled and pushed through an `AppSrc`. Which of the two it is determines whether the payload is writable: samples coming out of the pipeline are read-only.
**APIs**
| `qti::Buffer` | Description |
| -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Buffer()` | Creates an empty wrapper, referencing no data. `valid()` returns `false` until it is filled. |
| `Buffer(size_t size)` | Allocates a writable buffer with a payload of `size` bytes, for pushing into an `AppSrc`. |
| `static Buffer from_readable_sample(void *gst_sample_opaque)` | Wraps an existing `GstSample` and exposes its payload and metadata as a `Buffer`. |
| `uint8_t *data()` / `const uint8_t *data() const` | Returns a pointer to the payload. The non-const overload requires a writable buffer. |
| `size_t size() const` | Returns the payload size in bytes. |
| `void resize(size_t n)` | Resizes the payload storage. |
| `void set_pts(uint64_t ns)` / `void set_dts(uint64_t ns)` / `void set_duration(uint64_t ns)` | Sets the presentation timestamp, decode timestamp, or duration, in nanoseconds. |
| `uint64_t pts() const` / `uint64_t dts() const` / `uint64_t duration() const` | Reads the corresponding timestamp field, in nanoseconds. |
| `bool is_writable() const` | Returns `true` when the underlying memory can be modified in place. |
| `bool is_readonly() const` | Returns `true` when the buffer came from an `AppSink` and its memory cannot be modified. |
| `bool valid() const` | Returns `true` when the wrapper currently references valid data. |
| `void *take_gst_buffer()` | Transfers ownership of the underlying `GstBuffer` to the caller and clears this wrapper. This is the zero-copy hand-off path used by `AppSrc::push_buffer()`. |
| `void refill_for_appsrc(size_t n)` | Prepares the buffer's storage for reuse inside an `AppSrc` producer callback, resizing it to `n` bytes. |
Like the element wrappers, a `Buffer` cannot be copied, only moved — ownership of the underlying memory is unique. Pass it by reference, or `std::move()` it where ownership has to change, as in the `AppSink` → `AppSrc` bridge above.
### StreamFilter
**Purpose**
StreamFilter describes the characteristics of a media stream between two pipeline stages so they can negotiate concrete parameters rather than relying on defaults. Without explicit parameters, adjacent components may fall back to suboptimal formats, trigger implicit conversions, or produce mismatched configurations.
Specialized subtypes provide fluent, type-safe setters for each domain. Raw caps strings are also accepted for cases not covered by typed setters or when copying directly from gst-launch output.
**When to Use StreamFilter**
* When two adjacent stages can interoperate in multiple ways, and you want to select explicit parameters instead of relying on defaults.
* When you need to optimize latency or performance by avoiding implicit conversions (e.g., choose a hardware-friendly pixel format).
* When you want predictable behavior across platforms (e.g., lock color range/primaries/transfer/matrix).
* When you integrate tensors for AI/ML and must precisely specify dimensions and element types.
* When you need a human-readable summary of the stream parameters for logs and debugging.
A `StreamFilter` is not an `Element` and is never added to a pipeline with `add()`. It is always attached between stages with `Pipeline::add_stream_filter(unique_name, filter)`, which internally creates a `capsfilter` element and applies the filter's caps to it. The base `StreamFilter` class itself only exposes a raw-caps-string constructor and `to_string()`; the fluent per-domain setters (`.format()`, `.resolution()`, `.framerate()`, ...) live on the typed subclasses such as `VideoFilter`.
**Usage**
**Typed subclass**
```cpp theme={null}
Pipeline pipeline("video-pipeline");
pipeline.add("videotestsrc", "src", "pattern", "ball")
.add_stream_filter("filter", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
.add("waylandsink", "sink")
.execute();
```
**Raw caps string**
```cpp theme={null}
Pipeline pipeline("video-pipeline-streamfilter-string");
pipeline.add("videotestsrc", "src", "pattern", "ball")
.add_stream_filter("filter", qti::StreamFilter(
"video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1"))
.add("waylandsink", "sink")
.execute();
```
**APIs**
| `qti::StreamFilter` | Description |
| ---------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `StreamFilter()` | Creates an empty filter expression. |
| `StreamFilter(const std::string &caps_or_mediatype)` | Creates a filter from a raw caps or media-type string, e.g. `"video/x-raw,format=NV12"`. |
| `std::string to_string() const` | Serializes the current filter expression to a caps string, for logging/debugging. |
### VideoFilter
**Purpose**
Configures a raw video stream (YUV, RGB, Bayer) between a camera source and any downstream processing or display element. Supports format, resolution, framerate, colorimetry, interlace mode, and pixel aspect ratio.
**Key Responsibilities**
* Produce caps with standard raw video attributes: format, resolution, framerate, colorimetry, interlace mode, pixel aspect ratio, etc.
**APIs**
| Member | Description |
| ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `format(const std::string &fmt)` / `format(Format fmt)` | Sets the stream's pixel format, for example `NV12`, `YUY2`, `RGB`, `RGBA`. Determines how pixel data is laid out in memory; essential for hardware compatibility and color-conversion paths. Accepts a string or a `VideoFilter::Format` enumerator. |
| `resolution(int width, int height)` | Specifies the frame dimensions in pixels. |
| `framerate(int num, int den = 1)` / `framerate(float num)` | Sets the frame rate as a fraction, for example `30/1` or `30000/1001` frames-per-second. Pass a single value when you do not need a fractional frame rate, like `30.0` fps. |
| `colorimetry(const std::string &value)` | Defines the colorimetry standard, for example `bt601`, `bt709`, `bt2020`. |
| `range(const std::string &value)` | Selects the quantization range, `full` or `limited`. |
| `interlace(const std::string &mode)` | Sets the interlacing mode, for example `progressive` or `interleaved`. Progressive is standard for modern pipelines; interlaced is still used in some camera/legacy systems. |
| `pixel_aspect_ratio(int num, int den = 1)` | Specifies the pixel aspect ratio, also called PAR. Normally `1/1`; used for anamorphic or non-square-pixel formats. |
| `add(const std::string &expr)` | Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API. |
`VideoFilter::Format` enumerates the raw video formats (`video/x-raw`), `JPEG` (`image/jpeg`), and the Bayer patterns (`video/x-bayer`: `BGGR`, `RGGB`, `GBRG`, `GRBG`, `MONO`).
**Usage**
We can achieve the same results in three different ways:
**Typed (enum) API** - if you want compile-time safety, IDE autocompletion, and guardrails against typos. Ideal for SDK consumers and most production code.
```cpp theme={null}
auto v1 = VideoFilter()
.format(VideoFilter::Format::NV12)
.resolution(1920, 1080)
.framerate(30)
.colorimetry("bt709")
.range("limited")
.interlace("progressive");
```
**String API** - if you need quick iteration or to pass values not (yet) covered by enums — e.g., enabling a freshly added GStreamer value on a target platform without immediately updating headers.
```cpp theme={null}
auto v2 = VideoFilter()
.format("NV12")
.resolution(1920, 1080)
.framerate(30)
.colorimetry("bt709")
.range("limited")
.interlace("progressive");
```
**Raw caps string** - very useful for setting arbitrary stream (caps) filter or you want to lift an existing caps line from logs, gst-launch, or a pad template and drop it as-is — great for debugging, prototyping, or reproducing pipelines exactly.
```cpp theme={null}
auto v3 = StreamFilter("video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1,colorimetry=bt709,range=limited");
```
### H264Filter
**Purpose**
Configures an encoded H.264 stream between an encoder and a downstream parser or muxer. Supports resolution, framerate, stream-format, alignment, profile, and level.
**Key Responsibilities**
* Specify resolution, framerate, stream-format, alignment, profile, level.
* Insert `codec_data` buffers.
* Attach arbitrary name/value string properties.
**APIs**
| Member | Description |
| ---------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| `resolution(int width, int height)` | Specifies the frame dimensions in pixels. |
| `framerate(int num, int den = 1)` / `framerate(float num)` | Sets the frame rate as a fraction, or as a single value when a fractional rate is not needed. |
| `profile(const std::string &p)` | Sets the H.264 profile caps field, for example `baseline`, `main`, `high`. |
| `level(const std::string &lvl)` | Sets the H.264 level caps field. |
| `stream_format(const std::string &f)` | Sets the H.264 stream-format caps field, for example `byte-stream`, `avc`. |
| `alignment(const std::string &a)` | Sets the H.264 alignment caps field, for example `au`, `nal`. |
| `codec_data(const std::string &bytes)` | Sets the H.264 `codec_data` caps field from a byte string. |
| `set(const std::string &key, const std::string &val)` | Shorthand for `add(key + "=" + val)`. |
| `add(const std::string &expr)` | Appends an arbitrary custom caps field using a `key=value` expression. |
**Usage**
```cpp theme={null}
auto h264f = H264Filter().resolution(1920, 1080)
.framerate(30)
.profile("baseline")
.stream_format("byte-stream")
.alignment("au");
Pipeline pipeline("h264-pipeline");
pipeline.add_stream_filter("h264f", h264f);
```
### TensorFilter
**Purpose**
Configures an ML tensor stream with explicit data type and shape (e.g., UINT8, NHWC). Not needed in a full inference pipeline — when preprocess, inference, and postprocess elements are all present, tensor parameters are negotiated automatically. Use TensorFilter when:
* The preprocessing stage does not have model dimension information
* You need exact tensor data output from the inference plugin
**Key Responsibilities**
* Declare element type (e.g., `UINT8`, `FLOAT32`) for tensors.
* Specify shape (dimensions). Supports:
* Single tensor: a 1D list (e.g., `1, 520, 520, 3`).
* Multiple tensors: array-of-arrays (e.g., `{ {1,3,H,W}, {1,32} }`) when needed.
**APIs**
| Member | Description |
| -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `type(const std::string &t)` / `type(Type t)` | Specifies the tensor data type, for example `UINT8` or `FLOAT32`. Accepts a string or a `TensorFilter::Type` enumerator (`UINT8`, `UINT16`, `UINT32`, `INT8`, `INT16`, `INT32`, `FLOAT16`, `FLOAT32`). |
| `dimensions(UInts... dims)` | Specifies a single tensor's dimensions as a variadic list of integers, for example `dimensions(1, 520, 520, 3)`. |
| `dimensions(const std::vector &one_tensor_dims)` | Specifies a single tensor's dimensions as a vector. |
| `dimensions(const std::vector> &many_tensors_dims)` | Specifies several tensors at once, one inner vector per tensor. |
| `add(const std::string &expr)` | Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API. |
**Usage**
We can achieve the same results in three different ways:
**Typed (enum) API** - if you want compile-time safety, IDE autocompletion, and guardrails against typos. Ideal for SDK consumers and most production code.
```cpp theme={null}
auto t1 = TensorFilter()
.type(TensorFilter::Type::UINT8)
.dimensions(1, 520, 520, 3); // NHWC, for example
```
**String API** - if you need quick iteration or to pass values not (yet) covered by enums — e.g., enabling a freshly added GStreamer value on a target platform without immediately updating headers.
```cpp theme={null}
auto t2 = TensorFilter()
.type("FLOAT32")
.dimensions(1, 3, 640, 640); // NCHW, for example
```
**Raw caps string** - very useful for setting arbitrary stream (caps) filter or you want to lift an existing caps line from logs, gst-launch, or a pad template and drop it as-is — great for debugging, prototyping, or reproducing pipelines exactly.
```cpp theme={null}
auto t3 = StreamFilter("neural-network/tensors,type=UINT8,dimensions=<<1,520,520,3>>");
```
### TextFilter
**Purpose**
Configures a raw text stream (text/x-raw) for pipelines that carry textual data such as metadata, subtitles, or logs. Minimal by design — attach only the fields you need using .add("key=value").
**Key Responsibilities**
* Provide a lightweight caps descriptor for text (`text/x-raw`) without enforcing a schema.
* Let callers attach arbitrary properties (e.g., encoding/charset, language, packetization hints) using `.add("key=value")`.
**APIs**
| Member | Description |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `add(const std::string &expr)` | Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API. |
**Usage**
```cpp theme={null}
auto txt = TextFilter().add("charset=utf8");
```
### ImageFilter
**Overview**
Used exclusively with CamSrc (qtiqmmfsrc) to enable and link its dedicated image capture pad. Without ImageFilter, the image pad is not requested, not activated, and produces no buffers. It is a structural enabler, not a processing stage — it is not intended for general image processing or frame extraction from video sources.
`ImageFilter` is used to:
* Enable the image output pad exposed by `CamSrc` (`qtiqmmfsrc`)
* Attach that pad to the pipeline so the source can activate image capture
* Provide a concrete sink target required by GStreamer for pad activation
It is not intended for:
* Generic image processing
* Frame extraction from arbitrary video sources
* Replacement of `VideoFilter` or `AppSink`
**Relationship to CamSrc (qtiqmmfsrc)**
* `CamSrc` (`qtiqmmfsrc`) exposes a dedicated image pad that remains inactive unless linked.
* `ImageFilter` explicitly exists to satisfy this requirement.
* Without `ImageFilter`, the image pad:
* Will not be requested
* Will not be activated
* Will not produce buffers
This makes `ImageFilter` a structural enabler, not a processing stage.
**APIs**
| Member | Description |
| ---------------------------------------------------------- | ---------------------------------------------------------------------- |
| `format(const std::string &fmt)` | Sets the image format for the capture pad, for example `JPEG`. |
| `resolution(int width, int height)` | Specifies the captured image dimensions in pixels. |
| `framerate(int num, int den = 1)` / `framerate(float num)` | Sets the capture pad's frame rate. |
| `add(const std::string &expr)` | Appends an arbitrary custom caps field using a `key=value` expression. |
**Usage**
```cpp theme={null}
auto imagefilter1 = ImageFilter()
.format("JPEG")
.resolution(3840, 2160);
```
### AudioFilter
**Purpose**
Configures a raw PCM audio stream with sample format, channel count, sample rate, and layout. Use this to ensure audio converters and sinks negotiate exactly the format your hardware supports rather than inserting an implicit resampler.
**Key Responsibilities**
* Set audio sample format (e.g., `S16LE`, `F32LE`) to match HW/SW capabilities.
* Define channels and rate for correct resampling/mixing.
* Choose layout (`interleaved` vs `non-interleaved`) to align with downstream expectations.
**APIs**
| Member | Description |
| ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `format(const std::string &fmt)` / `format(Format f)` | Sets the audio sample format, for example `S16LE`, `F32LE`. Accepts a string or an `AudioFilter::Format` enumerator. |
| `channels(int n)` | Specifies the number of audio channels. |
| `rate(int hz)` | Specifies the audio sample rate in Hz. |
| `layout(const std::string &l)` / `layout(Layout l)` | Specifies the audio layout. Accepts a string or `AudioFilter::Layout::INTERLEAVED` / `NON_INTERLEAVED`. |
| `add(const std::string &expr)` | Appends an arbitrary custom caps field using a `key=value` expression. Escape-hatch for advanced/experimental fields not covered by the typed API. |
`AudioFilter::Format` covers the signed/unsigned integer formats (`S8` through `U32BE`, including the packed 18/20/24-bit variants) and the float formats `F32LE`, `F32BE`, `F64LE`, `F64BE`.
**Usage**
We can achieve the same results in three different ways:
**Typed (enum) API** - if you want compile-time safety, IDE autocompletion, and guardrails against typos. Ideal for SDK consumers and most production code.
```cpp theme={null}
auto a1 = AudioFilter()
.format(AudioFilter::Format::S16LE)
.rate(48000)
.channels(2)
.layout(AudioFilter::Layout::INTERLEAVED);
```
**String API** - if you need quick iteration or to pass values not (yet) covered by enums — e.g., enabling a freshly added GStreamer value on a target platform without immediately updating headers.
```cpp theme={null}
auto a2 = AudioFilter()
.format("F32LE")
.rate(44100)
.channels(1)
.layout("non-interleaved");
```
**Raw caps string** - very useful for setting arbitrary stream (caps) filter or you want to lift an existing caps line from logs, gst-launch, or a pad template and drop it as-is — great for debugging, prototyping, or reproducing pipelines exactly.
```cpp theme={null}
auto a3 = StreamFilter("audio/x-raw,format=S16LE,rate=48000,channels=2,layout=interleaved");
```
### Logging
The SDK can either simplify GStreamer's log stream into a compact, uniform format or step aside and let GStreamer log raw, with `GST_DEBUG` fully in control. Both modes, the verbosity levels, and the ordering constraint on configuring them are covered in [Logging and Diagnostics](/app-builder/cpp-logging).
## QIM SDK Nodes
All nodes are executed by GStreamer elements under the hood. Most elements are handled through the `Element` base class. However, elements that provide additional functionality — such as application buffer interfaces in appsink and appsrc, or image-capture signals in camera sources like `qtiqmmfsrc` — require extended interfaces.
If you do not need these extra capabilities (for example, image capture), you can still use these elements via the generic `Element` base class without any limitations for standard data processing.
Below is a table listing all supported elements and the corresponding QIM SDK wrapper classes used to work with them.
**Sources**
| GStreamer | QIM SDK C++ Class |
| -------------- | --------------------------------------------- |
| `qtiqmmfsrc` | `qti::CamSrc` (also usable as `qti::Element`) |
| `v4l2src` | `qti::Element` |
| `filesrc` | `qti::Element` |
| `multifilesrc` | `qti::Element` |
| `appsrc` | `qti::AppSrc` |
| `appsink` | `qti::AppSink` |
| `qtisocketsrc` | `qti::Element` |
**AI**
| GStreamer | QIM SDK C++ Class |
| --------------------- | ------------------------------------------------------- |
| `qtimlaconverter` | `qti::Element` |
| `qtimlvconverter` | `qti::MLVConverter` (also usable as `qti::Element`) |
| `qtimlqnn` | `qti::Element` |
| `qtimlsnpe` | `qti::Element` |
| `qtimltflite` | `qti::Element` |
| `qtimlpostprocess` | `qti::MLPostprocess` (also usable as `qti::Element`) |
| `qtimlvideoonnxbin` | `qti::MLVideoONNXBin` (also usable as `qti::Element`) |
| `qtimlvideoqnnbin` | `qti::MLVideoQNNBin` (also usable as `qti::Element`) |
| `qtimlvideosnpebin` | `qti::MLVideoSNPEBin` (also usable as `qti::Element`) |
| `qtimlvideotflitebin` | `qti::MLVideoTFLiteBin` (also usable as `qti::Element`) |
| `qtimlaic` | `qti::Element` |
| `qtimetamux` | `qti::Element` |
| `qtimlmetaextractor` | `qti::Element` |
| `qtimlmetaparser` | `qti::Element` |
| `qtimldemux` | `qti::Element` |
| `qtimetatransform` | `qti::Element` |
| `mlonnx` | `qti::Element` |
**Multimedia**
| GStreamer | QIM SDK C++ Class |
| ----------------- | ----------------- |
| `qtiobjtracker` | `qti::Element` |
| `qtismartvencbin` | `qti::Element` |
| `qtivoverlay` | `qti::Element` |
| `qtivsplit` | `qti::Element` |
| `qtivcomposer` | `qti::Element` |
| `qtibatch` | `qti::Element` |
| `qticamreproc` | `qti::Element` |
| `qtic2adec` | `qti::Element` |
| `qtic2aenc` | `qti::Element` |
| `qtic2vdec` | `qti::Element` |
| `qtic2venc` | `qti::Element` |
| `qtijpegenc` | `qti::Element` |
| `qtivtransform` | `qti::Element` |
| `qticvimgpyramid` | `qti::Element` |
| `qticvoptclflow` | `qti::Element` |
| `qtidfs` | `qti::Element` |
| `v4l2h264dec` | `qti::Element` |
| `v4l2h264enc` | `qti::Element` |
| `v4l2h265dec` | `qti::Element` |
| `v4l2h265enc` | `qti::Element` |
| `tee` | `qti::Element` |
| `h264parse` | `qti::Element` |
| `rtpptdemux` | `qti::Element` |
| `rtph264depay` | `qti::Element` |
| `queue` | `qti::Element` |
| `qtdemux` | `qti::Element` |
| `tsdemux` | `qti::Element` |
## Common Application Design
### Create pipeline with config file
The SDK supports fully declarative pipeline construction using YAML. A YAML pipeline describes elements and links between them, which are translated at runtime into a GStreamer pipeline.
**TopLevel Structure**
A YAML pipeline file has a single top-level key:
```yaml theme={null}
pipeline:
elements: [...]
links: [...]
```
**Elements Section**
`elements` is an ordered list of pipeline elements. Each element entry defines:
* The GStreamer element type
* A unique name used for linking
* Optional properties
* Optional filter configuration
**Common Fields**
```yaml theme={null}
- type:
name:
:
```
* `type`: GStreamer element factory name, or `filter`
* `name`: Unique identifier within the pipeline
* properties: Any remaining key/value pairs are passed directly to the element as GStreamer properties
**Filter Elements**
Filters are declared using `type: filter` and are further specialized by their configuration block.
**Stream Filter**
```yaml theme={null}
- type: capsfilter
name: raw_caps
caps: "video/x-raw,format=NV12,width=1280,height=720,framerate=30/1"
```
**Video Filter**
```yaml theme={null}
- type: filter
name: vf
video:
format: NV12
width: 1920
height: 1080
framerate: 30
```
**Links Section**
`links` defines explicit connection paths between elements.
Each entry is an ordered list of element names. Elements are linked sequentially from left to right.
```yaml theme={null}
links:
- [src, demux, parse, decoder, vf, split]
```
**Example yaml files**
**YAML config file**
```yaml theme={null}
pipeline:
elements:
- type: qtiqmmfsrc
name: source
camera: "0"
- type: filter
name: filter
video:
format: NV12
width: 1920
height: 1080
framerate: 30.0
- type: tee
name: split
- type: qtimlvconverter
name: preprocessing
- type: qtimltflite
name: inferencing
delegate: gpu
model: /etc/models/model.tflite
- type: qtimlpostprocess
name: postprocessing
- type: qtimetamux
name: mlmuxer
- type: qtivoverlay
name: overlay
- type: waylandsink
name: display
sync: true
async: true
fullscreen: true
links:
- [source, filter, split, preprocessing, inferencing, postprocessing, mlmuxer, overlay, display]
- [split, mlmuxer]
```
### Multi-Model Daisy Chain
Below is shown a logical diagram of an AI daisy chain pipeline with sequential and parallel inferencing.
The same could be handled by a straight sequential pipeline of inference bins. But we must set manually the source for every inference bin.
Several `.add(...)` calls in this example had missing/mismatched quotes in the source PDF (e.g. a stray closing paren instead of a closing quote, or a missing leading quote). These have been corrected below where unambiguous. One inline comment truncated at the page margin was completed using the identical sentence found in the prose above.
```cpp theme={null}
Element ml_car("qtimlvideotflitebin", "car")
.add("performance-profile", "high-performance")
.add("inference-delegate", "dsp")
.add("inference-model", "/opt/car_detect.dlc")
.add("inference-layers", "< /model.22/Mul_2, /model.22/Sigmoid >")
.add("postprocess-threshold", 60.0)
.add("postprocess-results", 4)
.add("postprocess-module", "yolov8")
.add("postprocess-labels", "/opt/car_detect_model.labels");
// The same could be handle by straight sequential pipeline of inference bins. But we must set manually source to every inference bin.
Element ml_plate("qtimlvideotflitebin", "plated")
.add("source", 1)
.add("image-disposition", "centre")
.add("performance-profile", "high-performance")
.add("inference-delegate", "dsp")
.add("inference-model", "/opt/plate_detect.dlc")
.add("inference-layers", "< /model.22/Mul_2, /model.22/Sigmoid >")
.add("postprocess-threshold", 40.0)
.add("postprocess-results", 1)
.add("postprocess-module", "yolov8")
.add("postprocess-labels", "/opt/plate_detect_model.labels");
Element ml_anpr("qtimlvideotflitebin", "anpr")
.add("source", 2)
.add("image-disposition", "centre")
.add("inference-delegate", "dsp")
.add("inference-model", "/opt/anpr.dlc")
.add("inference-layers", "< /decode_head2/classifier/Conv, /decode_head/classifier/Conv >")
.add("postprocess-threshold", 15.0)
.add("postprocess-results", 1)
.add("postprocess-module", "anpr")
.add("postprocess-labels", "/opt/anpr_model.labels");
Element ml_siglip("qtimlvideotflitebin", "siglip")
.add("source", 1)
.add("image-disposition", "centre");
.add("inference-delegate", "dsp");
.add("inference-model", "/opt/car_make_model.dlc");
.add("inference-layers", "< /classifier/Gemm >");
.add("postprocess-threshold", 15.0);
.add("postprocess-results", 1);
.add("postprocess-module", "mobilenet");
.add("postprocess-labels", "/opt/car_make_model.labels");
Pipeline pipeline("SmartVehicleIdentification");
pipeline.add("camsrc", "source", "camera", "0")
.add_stream_filter("filter", VideoFilter().format("NV12").resolution(1920, 1080).framerate(30))
.add(ml_car)
.add(ml_plate)
.add(ml_anpr)
.add(ml_siglip)
.add("qtivoverlay", "overlay")
.add("waylandsink", "display", "sync", "true", "async", "true", "fullscreen")
.execute();
```
### Custom Postprocessing with YOLOv8 (Detection)
QIM SDK implements AI model post-processing through dedicated modules that take tensors as input and produce predictions. All remaining complexity — such as batching, daisy-chaining, image mask support — is encapsulated within the post-processing plugin. This makes the post-processing module API an ideal integration point for application-specific post-processing logic. As a result, applications do not need to handle batching, chaining, image masks, or ML metadata themselves. We will implement a specialized post-processing module to manage bindings with the application.
Custom postprocessing callback is attached to a [`qtimlpostprocess`](/plugin-reference/qtimlpostprocess) element via [`MLPostprocess::set_handler(...)`](/plugin-reference/qtimlpostprocess).
Typical pipeline segment:
1. Preprocess frame/tensors ([`qtimlvconverter`](/plugin-reference/qtimlvconverter))
2. Run ML inference ([`qtimltflite`](/plugin-reference/qtimltflite))
3. Convert raw tensors in custom callback ([`qtimlpostprocess`](/plugin-reference/qtimlpostprocess))
**Supported callback types**
* `ClassificationPostprocessCallback`
* `ObjectDetectionPostprocessCallback`
* `PoseEstimationPostprocessCallback`
* `DepthEstimationPostprocessCallback`
* `SegmentationPostprocessCallback`
* `TensorsPostprocessCallback`
**Minimal usage**
```cpp theme={null}
MLPostprocess postprocessing("postprocessing");
postprocessing.set_handler(my_callback);
pipeline.add("qtimltflite", "inferencing", ...)
.add(postprocessing)
.add_stream_filter("mlf", TextFilter())
...;
```
**Input and output explained**
`MLFrameView` (read-only model tensors)
Used by classification/detection/pose/depth/segmentation callbacks. Each `TensorView` contains:
* `type`: tensor element type (`MLTensorType`)
* `dimensions`: tensor shape
* `data`: raw tensor memory (`const uint8_t*`)
* `size`: tensor bytes
Common access pattern:
```cpp theme={null}
const auto& t = frame.tensors[0];
const float* data = static_cast(t.data);
```
`MLParam` (runtime metadata and context)
`MLParam` carries key/value fields provided by pipeline and preprocessing/postprocess context. Commonly used keys:
* `input-tensor-width`
* `input-tensor-height`
* `input-tensor-region` (mapped to `Region {x, y, width, height}`)
Example:
```cpp theme={null}
float source_width = 0.0f;
float source_height = 0.0f;
Region region;
params.get("input-tensor-width", source_width);
params.get("input-tensor-height", source_height);
params.get("input-tensor-region", region);
```
**Callback signatures**
* Callbacks (`ClassificationPostprocessCallback`, `ObjectDetectionPostprocessCallback`, `PoseEstimationPostprocessCallback`, `DepthEstimationPostprocessCallback`, `SegmentationPostprocessCallback`) fill the provided output metadata container (`MLClassifications`, `MLDetections`, `MLPoses`, `MLDepthMaps`, `MLSegmentations`).
* Tensor callback (`TensorsPostprocessCallback`) fills `MLFrame& output` for downstream tensor stages.
* Input `MLFrameView` is read-only and must not be modified.
**Classification callback**
```cpp theme={null}
using ClassificationPostprocessCallback =
std::function;
```
Output: fill `MLClassifications` with `MLClassification` items. Field meanings:
* `name`: class label text
* `confidence`: prediction confidence
* `color`: optional UI/overlay color
* `extra`: optional metadata map
**Object detection callback**
```cpp theme={null}
using ObjectDetectionPostprocessCallback =
std::function;
```
Output: fill `MLDetections` with `MLDetection` items. Field meanings:
* `left`, `top`, `right`, `bottom`: normalized box coordinates
* `name`, `confidence`, `color`: class info
* `landmarks`: optional keypoints
* `extra`: optional metadata map
Implemented example includes confidence thresholding, region coordinate transform, and NMS.
**Segmentation callback**
```cpp theme={null}
using SegmentationPostprocessCallback =
std::function;
```
Input parameters
* `frame` (`const MLFrameView&`): read-only model output tensors.
* `params` (`const MLParam&`): runtime metadata (for example: `input-tensor-width`, `input-tensor-height`, `input-tensor-region`).
* `segmentations` (`MLSegmentations&`): output container to fill.
Output contract (`MLSegmentation`)
* `labels`: per-pixel semantic labels.
* `colors`: per-pixel colors; index-aligned with `labels`.
* `n_rows`, `n_columns`: segmentation map dimensions.
* `extra`: optional metadata fields.
**Pose estimation callback**
```cpp theme={null}
using PoseEstimationPostprocessCallback =
std::function;
```
Output: fill `MLPoses` with `MLPose` items. Field meanings:
* `keypoints`: list of `MLKeypoint` (`name`, `x`, `y`, `confidence`, `color`)
* `links`: list of `MLKeypointLink` describing skeleton connections
* `name`, `confidence`, `extra`: pose-level metadata
**Depth estimation callback**
```cpp theme={null}
using DepthEstimationPostprocessCallback =
std::function;
```
Output: fill `MLDepthMaps` with one or more `MLDepthMap` items. Field meanings:
* `values`: per-pixel depth values (`double`)
* `colors`: per-pixel pseudo-color values (`uint32_t`)
* `n_rows`, `n_columns`: depth map shape
* `extra`: optional metadata map
**Tensor-to-tensor callback**
```cpp theme={null}
using TensorsPostprocessCallback =
std::function;
```
output is owned and writable; callback typically:
1. validates input tensors from `frame`
2. copies/selects required tensors into output
3. updates output tensor dimensions/order if needed
**Semantic callbacks** (`classification`, `detection`, `pose`, `depth`, `segmentation`)
```cpp theme={null}
bool callback(const MLFrameView& frame,
const MLParam& params,
OutputType& output)
```
* `frame`: input tensors from inference (read-only)
* `params`: runtime metadata (input sizes, region, etc.)
* `output`: semantic objects to be filled by callback
* return `true` on valid output
* return `false` on validation/decode failure
**Tensor callback**
```cpp theme={null}
bool callback(const MLFrameView& frame,
const MLParam& params,
MLFrame& output)
```
* `frame`: input tensors from inference (read-only)
* `params`: runtime metadata/context
* `output`: writable output tensors for downstream tensor stage
* return `true` on valid output
* return `false` on validation/decode failure
**Copy-ready callback skeletons**
Classification
```cpp theme={null}
ClassificationPostprocessCallback cb =
[](const MLFrameView& frame, const MLParam& params,
MLClassifications& out) {
// decode tensors -> classes
return true;
};
```
Detection
```cpp theme={null}
ObjectDetectionPostprocessCallback cb =
[](const MLFrameView& frame, const MLParam& params,
MLDetections& out) {
// decode tensors -> detections
return true;
};
```
Pose estimation
```cpp theme={null}
PoseEstimationPostprocessCallback cb =
[](const MLFrameView& frame, const MLParam& params,
MLPoses& out) {
// decode tensors -> poses
return true;
};
```
Depth estimation
```cpp theme={null}
DepthEstimationPostprocessCallback cb =
[](const MLFrameView& frame, const MLParam& params,
MLDepthMaps& out) {
// decode tensors -> depthmaps
return true;
};
```
Segmentation
```cpp theme={null}
SegmentationPostprocessCallback cb =
[](const MLFrameView& frame, const MLParam& params,
MLSegmentations& out) {
// decode tensors -> segmentations
return true;
};
```
Tensors
```cpp theme={null}
TensorsPostprocessCallback cb =
[](const MLFrameView& frame, const MLParam& params,
MLFrame& output) {
// map/copy selected tensors from frame -> output
return true;
};
```
Check application source code on GitHub: [`qimsdk_ref_external_postprocess_detection_yolov8`](https://github.com/qualcomm/qimsdk/tree/main/cpp/examples/reference-apps/qimsdk_ref_external_postprocess_detection_yolov8)
Pre-built application on device: `/usr/bin/qimsdk_ref_external_postprocess_detection_yolov8`
#### Download Required Files
| File | Save as |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------- |
| [Model](https://aihub.qualcomm.com/iot/models/yolov8_det) | yolov8\_det\_quantized.tflite |
| yolov8.json | yolov8.json |
| [ai\_demo\_sample.mp4](https://github.com/qualcomm/sample-apps-for-qualcomm-linux/blob/main/qualcomm-linux/artifacts/videos/demo_samples/ai_demo_sample.mp4) | ai\_demo\_sample.mp4 |
If the downloaded model file is a `.zip` archive, extract it on your host machine before copying: `unzip filename.zip`
Some AI Hub models require running the AI Hub export/optimize step for your target runtime before the downloaded file is usable — download alone isn't always sufficient.
```bash SCP (SSH) theme={null}
# Replace and with your device credentials.
# ~ resolves automatically to the correct home directory on the device (/root on QLI, /home/ubuntu on Ubuntu).
ssh @ "mkdir -p ~/Downloads/qimsdk_samples/{models,labels,media}"
scp yolov8_det_quantized.tflite @:~/Downloads/qimsdk_samples/models/
scp yolov8.json @:~/Downloads/qimsdk_samples/labels/
scp ai_demo_sample.mp4 @:~/Downloads/qimsdk_samples/media/
```
```bash theme={null}
/usr/bin/qimsdk_ref_external_postprocess_detection_yolov8
```
The application runs inference on the video file and uses the custom, application-defined postprocessing callback to decode detections, which are then overlaid and displayed fullscreen.
To stop the application, press **CTRL + C**.