Debugging Guide

This guide covers practical debugging techniques for QIM SDK AI pipelines — from enabling verbose logs to measuring per-stage latency, tracing buffers, and diagnosing camera or performance issues.

1. Enable Debug Logs

QIM SDK uses the standard GStreamer debug system. Pass --gst-debug=<level> to gst-launch-1.0, or set the GST_DEBUG environment variable for Python/C++ apps.

Debug Levels

Level	Name	Description
0	none	No debug information is output.
1	ERROR	Logs all fatal errors. These are errors that do not allow the core or elements to perform the requested action. The application can still recover if programmed to handle the conditions that triggered the error.
2	WARNING	Logs all warnings. Typically these are non-fatal, but user-visible problems are expected to happen.
3	FIXME	Logs all “fixme” messages. Those typically that a codepath that is known to be incomplete has been triggered. It may work in most cases, but may cause problems in specific instances.
4	INFO	Logs all informational messages. These are typically used for events in the system that only happen once, or are important and rare enough to be logged at this level.
5	DEBUG	Logs all debug messages. These are general debug messages for events that happen only a limited number of times during an object’s lifetime; these include setup, teardown, change of parameters, etc.
6	LOG	Logs all log messages. These are messages for events that happen repeatedly during an object’s lifetime; these include streaming and steady-state conditions. This is used for log messages that happen on every buffer in an element for example.
7	TRACE	Logs all trace messages. Those are message that happen very very often. This is for example is each time the reference count of a GstMiniObject, such as a GstBuffer or GstEvent, is modified.
9	MEMDUMP	Logs all memory dump messages. This is the heaviest logging and may include dumping the content of blocks of memory.

Global Level

gst-launch-1.0 --gst-debug=3 \
filesrc location=... ! ...

Per-Element Level

Override the global level for a specific plugin — useful for isolating one stage without drowning in output from everything else:

# Global = 2 (warnings), but inference plugin = 6 (verbose)
gst-launch-1.0 --gst-debug=2,qtimltflite:6 \
filesrc location=... ! ...

Common Starting Points

Flag	When to use
`--gst-debug=2`	Production runs — quiet, errors/warnings only
`--gst-debug=2,qtimltflite:5`	Prints all ERROR and WARNING messages globally, plus ERROR, WARNING, INFO, and DEBUG messages specifically from the qtimltflite plugin
`--gst-debug=2,qtimlvtransform:4,qtivcomposer:7`	Prints all ERROR and WARNING messages globally, ERROR, WARNING, and INFO messages from qtimlvtransform, and ERROR, WARNING, INFO, DEBUG, LOG, and TRACE messages from qtivcomposer

Python / C++ Apps

GST_DEBUG=3 python3 my_pipeline.py

GST_DEBUG=5 ./my_pipeline_app

Redirect to File

GST_DEBUG=5 GST_DEBUG_FILE=/data/gst.log GST_DEBUG_COLOR_MODE=off gst-launch-1.0 ...

2. Measure Pipeline KPIs (Latency per Stage)

Use the built-in GStreamer latency tracer to get per-element mean/min/max processing time for every stage — no custom instrumentation required.

Enable Tracing

GST_TRACERS="latency(flags=pipeline+element)" \
GST_DEBUG="GST_TRACER:7" \
gst-launch-1.0 -e \
  qtimlvconverter name=stage_01_preproc \
  qtimltflite name=stage_01_inference \
    model=$HOME/models/$MODEL_NAME \
    delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
    external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" \
  qtimlpostprocess name=stage_01_postproc module=yolov8 \
    labels=$HOME/labels/$LABELS_NAME \
    settings="{\"confidence\": 51.0}" \
  filesrc location=$HOME/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
  v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
  tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! \
    fpsdisplaysink name=display sync=false text-overlay=false \
      video-sink="waylandsink" \
  t. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! \
  stage_01_inference. stage_01_inference. ! queue ! \
  stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! obj_mux. \
2> /tmp/trace.log

Parse Results

gst-stats-1.0 /tmp/trace.log

Reading the Output

Element Latency Statistics:
  stage_01_preproc.src:   mean=2.19ms  min=1.24ms  max=12.2ms   ← preprocessing
  stage_01_inference.src: mean=5.71ms  min=4.95ms  max=6.78ms   ← inference (NPU)
  stage_01_postproc.src:  mean=0.24ms  min=0.12ms  max=0.44ms   ← postprocessing
  obj_mux.src:            mean=0.76ms  min=0.14ms  max=2.03ms   ← metadata sync

At 30 fps (33 ms budget per frame), the typical AI pipeline uses ~37% of the frame budget, leaving headroom for additional stages.

Measure FPS Live (On-Screen)

Add fpsdisplaysink in place of waylandsink to see FPS on the display in real time:

... ! qtivoverlay ! fpsdisplaysink sync=false video-sink="waylandsink sync=false"

3. Measure Inference Performance

GST_TRACER (inferences/sec)

From the tracer output, calculate inferences/sec:

inferences/sec = 1 / mean_inference_latency

Example: mean = 5.71 ms → 1 / 0.00571 ≈ 175 inferences/sec

pidstat (CPU/Memory alongside inference)

# Run in a second terminal while the pipeline is running
pidstat -u -r -p $(pgrep gst-launch-1.0) 1

This reports per-second CPU (%usr, %sys) and memory (RSS, VSZ) for the pipeline process.

4. Debug Performance Issues

Frame Drops

Frame drops manifest as:

Stuttering on the display
queue elements reporting overflows in debug logs (--gst-debug=4)
Tracer showing high max latency on inference (stage_01_inference)

Check for queue overflows:

gst-launch-1.0 --gst-debug=4,queue:5 ...

Look for lines like:

WARN   queue  gstqueue.c:... overrun ...

Fix: Add max-size-time=0 max-size-bytes=0 max-size-buffers=3 to queues around inference:

... ! queue max-size-buffers=3 ! stage_01_preproc. ...

Slow Throughput

If end-to-end latency is higher than expected:

Run the latency tracer to find which stage is the bottleneck.
Check if the inference delegate is correct — GPU delegate is much slower than HTP (NPU):

# Slow — GPU
qtimltflite delegate=gpu ...

# Fast — NPU via HTP
qtimltflite delegate=external \
  external-delegate-path=libQnnTFLiteDelegate.so \
  external-delegate-options="QNNExternalDelegate,backend_type=htp;"

Check model quantization — W8A8 (quantized) is significantly faster on the NPU than float32.

Memory Copies

Unnecessary buffer copies add latency. To detect them, enable memory tracing:

GST_DEBUG=4,GST_BUFFER:5 gst-launch-1.0 ... 2>&1 | grep -i "copy\|memcpy\|alloc"

Use capture-io-mode=4 output-io-mode=4 on decoder/encoder elements to enable zero-copy DMA buffer passing:

v4l2h264dec capture-io-mode=4 output-io-mode=4

5. Trace Buffers

Buffer tracing lets you follow a specific buffer through the pipeline to diagnose where it is dropped, delayed, or corrupted.

Enable Buffer Tracing

GST_DEBUG=GST_SCHEDULING:5 gst-launch-1.0 ... 2>&1 | head -200

Each line shows a buffer’s PTS (presentation timestamp), DTS, size, and which pad it passed through.

Check for PTS Discontinuities

If frames are out of order or the pipeline stalls, look for timestamp jumps:

GST_DEBUG=GST_SCHEDULING:5 gst-launch-1.0 ... 2>&1 | grep -i "pts\|discont\|gap"

Use the Dot Graph (Pipeline Visualization)

Dump the pipeline as a .dot file to visually inspect element connections:

GST_DEBUG_DUMP_DOT_DIR=/tmp gst-launch-1.0 -e ... &
# After pipeline is running:
dot -Tpng /tmp/*.dot -o pipeline.png

6. Debug Camera Issues

Common Errors

QMMF Recorder StartCamera Failed:

ERROR qticamsrc qmmf_source_context.cc: QMMF Recorder StartCamera Failed!
WARN  qticamsrc qmmf_source.c: error: Failed to Open Camera!

Fix: Kill and restart the camera server:

sudo killall cam-server

Then re-run the pipeline. The camera server restarts automatically on the next gst-launch-1.0 invocation.

Enable Camera Server Debug Logs

GST_DEBUG=2,qticamsrc:6 gst-launch-1.0 \
  qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
  waylandsink fullscreen=true sync=false

Camera Not Detected

# List available V4L2 devices
v4l2-ctl --list-devices

# Check camera capabilities
v4l2-ctl -d /dev/video0 --all

ISP Camera (Config #2 / qticamsrc)

To switch from libcamera (Config #1) to qticamsrc (Config #2), write "camx" to the EFI variable:

echo -n "camx" > /var/data
efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -w -f /var/data
sync && reboot

Verify Camera Stream is Running

# Quick sanity check — should show frames on display
gst-launch-1.0 \
  qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
  waylandsink fullscreen=true sync=false

7. Quick Diagnostic Checklist

Run this sequence when a pipeline doesn’t work:

# Step 1 — Check GStreamer can see the plugins
gst-inspect-1.0 qtimltflite
gst-inspect-1.0 qtimlvconverter
gst-inspect-1.0 qticamsrc

# Step 2 — Run with level 3 logs
gst-launch-1.0 --gst-debug=3 [your pipeline] 2>&1 | grep -E "ERROR|WARN|FIXME"

# Step 3 — Check camera server
sudo killall cam-server   # reset if camera errors appear

# Step 4 — Add latency tracing
GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG="GST_TRACER:7" \
  gst-launch-1.0 [your pipeline] 2> /tmp/trace.log
gst-stats-1.0 /tmp/trace.log

8. Report an Issue

When filing a bug or asking for help, include:

Platform — Device model (e.g. RB3 Gen 2), OS (Ubuntu / Qualcomm Linux), QIM SDK version
Full pipeline command — the exact gst-launch-1.0 or Python script
Debug log — captured with --gst-debug=5 2> debug.log
Tracer output — from gst-stats-1.0 if a performance issue
Error message — the exact error line from the log
Expected vs actual behaviour

# Collect everything in one command
GST_DEBUG=5 GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG_FILE=/tmp/debug.log \
  gst-launch-1.0 [your pipeline]
gst-stats-1.0 /tmp/debug.log >> /tmp/debug.log
# Attach /tmp/debug.log to your issue

Blog

Overview

Getting Started

Building AI Pipelines

Examples

Reference

Advanced Topics

1. Enable Debug Logs

Debug Levels

Global Level

Per-Element Level

Common Starting Points

Python / C++ Apps

Redirect to File

2. Measure Pipeline KPIs (Latency per Stage)

Enable Tracing

Parse Results

Reading the Output

Measure FPS Live (On-Screen)

3. Measure Inference Performance

GST_TRACER (inferences/sec)

pidstat (CPU/Memory alongside inference)

4. Debug Performance Issues

Frame Drops

Slow Throughput

Memory Copies

5. Trace Buffers

Enable Buffer Tracing

Check for PTS Discontinuities

Use the Dot Graph (Pipeline Visualization)

6. Debug Camera Issues

Common Errors

Enable Camera Server Debug Logs

Camera Not Detected

ISP Camera (Config #2 / qticamsrc)

Verify Camera Stream is Running

7. Quick Diagnostic Checklist

8. Report an Issue

​1. Enable Debug Logs

​Debug Levels

​Global Level

​Per-Element Level

​Common Starting Points

​Python / C++ Apps

​Redirect to File

​2. Measure Pipeline KPIs (Latency per Stage)

​Enable Tracing

​Parse Results

​Reading the Output

​Measure FPS Live (On-Screen)

​3. Measure Inference Performance

​GST_TRACER (inferences/sec)

​pidstat (CPU/Memory alongside inference)

​4. Debug Performance Issues

​Frame Drops

​Slow Throughput

​Memory Copies

​5. Trace Buffers

​Enable Buffer Tracing

​Check for PTS Discontinuities

​Use the Dot Graph (Pipeline Visualization)

​6. Debug Camera Issues

​Common Errors

​Enable Camera Server Debug Logs

​Camera Not Detected

​ISP Camera (Config #2 / qticamsrc)

​Verify Camera Stream is Running

​7. Quick Diagnostic Checklist

​8. Report an Issue

1. Enable Debug Logs

Debug Levels

Global Level

Per-Element Level

Common Starting Points

Python / C++ Apps

Redirect to File

2. Measure Pipeline KPIs (Latency per Stage)

Enable Tracing

Parse Results

Reading the Output

Measure FPS Live (On-Screen)

3. Measure Inference Performance

GST_TRACER (inferences/sec)

pidstat (CPU/Memory alongside inference)

4. Debug Performance Issues

Frame Drops

Slow Throughput

Memory Copies

5. Trace Buffers

Enable Buffer Tracing

Check for PTS Discontinuities

Use the Dot Graph (Pipeline Visualization)

6. Debug Camera Issues

Common Errors

Enable Camera Server Debug Logs

Camera Not Detected

ISP Camera (Config #2 / qticamsrc)

Verify Camera Stream is Running

7. Quick Diagnostic Checklist

8. Report an Issue