Building Production Quality Pipelines - Qualcomm Intelligent Multimedia SDK

Overview

This guide walks through the complete implementation of a GStreamer-based object detection application using Qualcomm hardware-accelerated plugins. The application:

Reads an H.264-encoded MP4 file from disk
Performs YOLOv8 object detection using a TFLite inference pipeline
Overlays detection bounding boxes and labels onto video frames
Renders the annotated output to a Wayland display

The complete application source code is available in the repository. For foundational concepts — including pipeline construction, state management, caps negotiation, bus handling, and dynamic pads — refer to Building a Native GStreamer Application.

Pipeline Diagram

Implementation Steps

Step 1 — Define the Application Context

The application consolidates all runtime state into a single context structure:

struct _GstAppContext
{
  GstElement *pipeline;
  GMainLoop  *mloop;
  gchar      *file;
  gchar      *model;
  gchar      *labels;
  GstElement *parse;
};

Field	Description
`pipeline`	Top-level GStreamer pipeline object for all pipeline operations
`mloop`	GLib main loop — keeps the application alive and dispatches events
`file`	Path to the input media file
`model`	Path to the TFLite inference model
`labels`	Path to the class label file
`parse`	Reference to `h264parse`, stored explicitly for deferred dynamic pad linking

This pattern avoids global variables, keeps application state self-contained, and simplifies callback implementations that require access to pipeline objects.

Step 2 — Create the Pipeline Object

pipeline = gst_pipeline_new ("gst-detection-display-example");
appctx.pipeline = pipeline;

A GstPipeline object is created using gst_pipeline_new(), establishing the top-level container that owns and manages all elements added to it.As a GstBin subclass, it is responsible for:

Element lifecycle management
State propagation across all contained elements
Synchronization of clocks and timing

All subsequent pipeline operations — element management, state transitions, and bus access — are performed through this object.

Step 3 — Instantiate Pipeline Elements

source  = gst_element_factory_make ("filesrc",               "file_src");
demux   = gst_element_factory_make ("qtdemux",               "file_qtdemux");
parse   = gst_element_factory_make ("h264parse",             "file_h264_parse");
decoder = gst_element_factory_make ("v4l2h264dec",           "v4l2h264dec");
mlbin   = gst_element_factory_make ("qtimlvideotflitebin",   "qtimlvideotflitebin");
overlay = gst_element_factory_make ("qtivoverlay",           "overlay");
sink    = gst_element_factory_make ("waylandsink",           "waylandsink");

Each element is instantiated using gst_element_factory_make(). The first argument identifies the element by its factory name; the second assigns a unique instance name (retrievable later via gst_bin_get_by_name()).

Element	Role
`filesrc`	Reads the input MP4 file from disk
`qtdemux`	Demuxes the MP4 container into elementary streams
`h264parse`	Parses the H.264 bitstream for downstream decoding
`v4l2h264dec`	Hardware-accelerated H.264 video decoder
`qtimlvideotflitebin`	Runs TFLite-based YOLOv8 object detection inference
`qtivoverlay`	Renders detection bounding boxes and labels onto video frames
`waylandsink`	Displays annotated video output on a Wayland surface

Step 4 — Configure Elements

Before the pipeline starts, element properties are set to define runtime behavior.Input source — configure filesrc with the path to the input file:

g_object_set (source, "location", appctx->file, NULL);

AI inference — configure qtimlvideotflitebin with delegate, post-processing module, model, and labels:

gst_element_set_enum_property (mlbin, "inference-delegate",  "external");
gst_element_set_enum_property (mlbin, "postprocess-module",  "yolov8");

g_object_set (mlbin,
    "inference-model",    appctx->model,
    "postprocess-labels", appctx->labels,
    NULL);

Property	Description
`inference-delegate`	Selects the hardware backend for model execution
`postprocess-module`	Specifies the post-processing algorithm (`yolov8` decoding)
`inference-model`	Path to the compiled TFLite model
`postprocess-labels`	Path to the class label file

Display sink — configure waylandsink for synchronized, full-screen rendering:

g_object_set (sink, "sync",       TRUE, NULL);
g_object_set (sink, "fullscreen", TRUE, NULL);

Property	Description
`sync`	Enables clock-synchronized rendering for smooth playback
`fullscreen`	Instructs the sink to occupy the full display surface

Step 5 — Add Elements to the Pipeline

gst_bin_add_many (GST_BIN (appctx->pipeline),
    source, demux, parse, decoder, mlbin, overlay, sink, NULL);

gst_bin_add_many() transfers ownership of all elements to the pipeline, making it responsible for their lifecycle — including state transitions and resource cleanup. At this point, elements share the same execution context but are not yet connected.

Step 6 — Link Static Pipeline Elements

gst_element_link (source, demux);
gst_element_link_many (parse, decoder, mlbin, overlay, sink, NULL);

This establishes the following static connections:

filesrc → qtdemux
h264parse → v4l2h264dec → qtimlvideotflitebin → qtivoverlay → waylandsink

The link between qtdemux and h264parse is intentionally omitted here. qtdemux exposes its output pads dynamically at runtime — this connection is established in Step 7 via dynamic pad handling.

Step 7 — Handle Dynamic Pads (Demuxer)

qtdemux determines the number and type of elementary streams only after parsing input data, dynamically creating output pads for each discovered stream. Since these pads do not exist at construction time, they cannot be linked statically.A callback is registered for the pad-added signal on the demuxer:

g_signal_connect (demux, "pad-added",
    G_CALLBACK (gst_app_qtdemux_pad_added_cb), appctx);

When a new pad is exposed at runtime, the callback inspects its type and links it to h264parse if it carries an H.264 video stream — all other stream types (e.g., audio) are ignored:

static void
gst_app_qtdemux_pad_added_cb (GstElement *element, GstPad *srcpad, gpointer userdata)
{
  gchar *name = gst_pad_get_name (srcpad);

  if (!g_str_has_prefix (name, "video/x-h264")) {
    g_free (name);
    return;
  }

  gst_app_link_dynamic_src_pad (srcpad, appctx->parse);
  g_free (name);
}

This completes the missing link:

qtdemux → h264parse

The full data path is now established end-to-end:

filesrc → qtdemux → h264parse → v4l2h264dec → qtimlvideotflitebin → qtivoverlay → waylandsink

Step 8 — Set Up Bus Message Handling

The application retrieves the pipeline bus and registers callbacks for error, EOS, and warning messages:

bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
gst_bus_add_signal_watch (bus);

g_signal_connect (bus, "message::error",   G_CALLBACK (gst_app_error_cb),   mloop);
g_signal_connect (bus, "message::eos",     G_CALLBACK (gst_app_eos_cb),     mloop);
g_signal_connect (bus, "message::warning", G_CALLBACK (gst_app_warning_cb), NULL);

Signal	Trigger	Action
`message::error`	Fatal error in a pipeline element	Log error and initiate shutdown
`message::eos`	All data processed; stream end reached	Exit main loop and proceed to teardown
`message::warning`	Non-fatal condition	Log warning; pipeline continues

Step 9 — Handle Ctrl+C (Interrupt Signal)

A SIGINT handler is registered using g_unix_signal_add() to route the OS signal safely into the GLib main loop:

g_unix_signal_add (SIGINT, gst_app_handle_interrupt_signal, &appctx);

The signal handler checks the current pipeline state and responds accordingly:

if (state == GST_STATE_PLAYING) {
  gst_element_send_event (appctx->pipeline, gst_event_new_eos ());
} else {
  g_main_loop_quit (appctx->mloop);
}

Pipeline State	Action
`GST_STATE_PLAYING`	Send EOS event — allows in-flight buffers to drain before stopping
Any other state	Quit main loop immediately

Sending EOS before stopping is particularly important for file-based outputs where finalization steps (e.g., writing file headers/footers) must complete.

Step 10 — Start the Pipeline

gst_element_set_state (pipeline, GST_STATE_PLAYING);

This single call propagates the state transition to all contained elements. Each element progresses through intermediate READY and PAUSED states before reaching PLAYING, at which point data begins to flow from filesrc through to waylandsink.

Step 11 — Run the Main Loop

g_main_loop_run (mloop);

The application enters the GLib main loop, which:

Blocks the main thread for the duration of pipeline execution
Dispatches bus messages and invokes registered callbacks as events occur
Exits only when explicitly quit by the EOS callback, error handler, or interrupt signal handler

Step 12 — Handle End-of-Stream

When the pipeline processes the last buffer and EOS propagates to the sink, an EOS bus message is delivered to the registered callback:

static void
gst_app_eos_cb (GstBus *bus, GstMessage *msg, gpointer userdata)
{
  g_main_loop_quit ((GMainLoop *) userdata);
}

The callback quits the main loop, allowing the main thread to resume and proceed with pipeline teardown and resource cleanup.

Step 13 — Handle Errors

If a fatal error occurs at any stage of pipeline execution, the error callback is invoked:

static void
gst_app_error_cb (GstBus *bus, GstMessage *msg, gpointer userdata)
{
  GError *err       = NULL;
  gchar  *debug_info = NULL;

  gst_message_parse_error (msg, &err, &debug_info);
  g_printerr ("ERROR: %s\n", err->message);

  if (debug_info)
    g_printerr ("Debug details: %s\n", debug_info);

  g_clear_error (&err);
  g_free (debug_info);

  g_main_loop_quit ((GMainLoop *) userdata);
}

Errors are treated as fatal. The callback:

Logs the error message and debug information for diagnostics
Quits the main loop to initiate controlled shutdown
The application subsequently transitions the pipeline to NULL and releases all resources

Step 14 — Teardown and Resource Cleanup

Once the main loop exits — whether due to EOS, error, or interrupt — the application performs an orderly teardown:

gst_element_set_state (pipeline, GST_STATE_NULL);
gst_object_unref (pipeline);
g_main_loop_unref (mloop);

Call	Effect
`gst_element_set_state(pipeline, GST_STATE_NULL)`	Transitions pipeline and all elements to `NULL` — stops processing, releases device handles, closes files, frees internal memory
`gst_object_unref(pipeline)`	Releases the application’s reference to the pipeline; destroys it and all owned elements if no other references are held
`g_main_loop_unref(mloop)`	Releases the main loop object and frees associated resources

This sequence ensures all resources are released in the correct order and that the application exits in a well-defined, clean state.

Summary

This guide covered the full implementation of a GStreamer YOLOv8 object detection application across 14 steps:

Phase	Steps	Key Operations
Initialization	1—2	Define app context, create pipeline
Element Setup	3—5	Instantiate, configure, and add elements
Linking	6—7	Static links + dynamic pad handling for `qtdemux`
Event Handling	8—9	Bus message callbacks, SIGINT handler
Execution	10—11	Set `PLAYING` state, run main loop
Shutdown	12—14	EOS/error handling, teardown and resource cleanup

​Overview

​Pipeline Diagram

​Implementation Steps

​Summary

Overview

Pipeline Diagram

Implementation Steps

Summary