This guide covers practical debugging techniques for QIM SDK AI pipelines — from enabling verbose logs to measuring per-stage latency, tracing buffers, and diagnosing camera or performance issues.
1. Enable Debug Logs
QIM SDK uses the standard GStreamer debug system. Pass --gst-debug=<level> to gst-launch-1.0, or set the GST_DEBUG environment variable for Python/C++ apps.
Debug Levels
| Level | Name | Description |
|---|
| 0 | none | No debug information is output. |
| 1 | ERROR | Logs all fatal errors. These are errors that do not allow the core or elements to perform the requested action. The application can still recover if programmed to handle the conditions that triggered the error. |
| 2 | WARNING | Logs all warnings. Typically these are non-fatal, but user-visible problems are expected to happen. |
| 3 | FIXME | Logs all “fixme” messages. Those typically that a codepath that is known to be incomplete has been triggered. It may work in most cases, but may cause problems in specific instances. |
| 4 | INFO | Logs all informational messages. These are typically used for events in the system that only happen once, or are important and rare enough to be logged at this level. |
| 5 | DEBUG | Logs all debug messages. These are general debug messages for events that happen only a limited number of times during an object’s lifetime; these include setup, teardown, change of parameters, etc. |
| 6 | LOG | Logs all log messages. These are messages for events that happen repeatedly during an object’s lifetime; these include streaming and steady-state conditions. This is used for log messages that happen on every buffer in an element for example. |
| 7 | TRACE | Logs all trace messages. Those are message that happen very very often. This is for example is each time the reference count of a GstMiniObject, such as a GstBuffer or GstEvent, is modified. |
| 9 | MEMDUMP | Logs all memory dump messages. This is the heaviest logging and may include dumping the content of blocks of memory. |
Global Level
gst-launch-1.0 --gst-debug=3 \
filesrc location=... ! ...
Per-Element Level
Override the global level for a specific plugin — useful for isolating one stage without drowning in output from everything else:
# Global = 2 (warnings), but inference plugin = 6 (verbose)
gst-launch-1.0 --gst-debug=2,qtimltflite:6 \
filesrc location=... ! ...
Common Starting Points
| Flag | When to use |
|---|
--gst-debug=2 | Production runs — quiet, errors/warnings only |
--gst-debug=2,qtimltflite:5 | Prints all ERROR and WARNING messages globally, plus ERROR, WARNING, INFO, and DEBUG messages specifically from the qtimltflite plugin |
--gst-debug=2,qtimlvtransform:4,qtivcomposer:7 | Prints all ERROR and WARNING messages globally, ERROR, WARNING, and INFO messages from qtimlvtransform, and ERROR, WARNING, INFO, DEBUG, LOG, and TRACE messages from qtivcomposer |
Python / C++ Apps
GST_DEBUG=3 python3 my_pipeline.py
GST_DEBUG=5 ./my_pipeline_app
Redirect to File
GST_DEBUG=5 GST_DEBUG_FILE=/data/gst.log GST_DEBUG_COLOR_MODE=off gst-launch-1.0 ...
2. Measure Pipeline KPIs (Latency per Stage)
Use the built-in GStreamer latency tracer to get per-element mean/min/max processing time for every stage — no custom instrumentation required.
Enable Tracing
GST_TRACERS="latency(flags=pipeline+element)" \
GST_DEBUG="GST_TRACER:7" \
gst-launch-1.0 -e \
qtimlvconverter name=stage_01_preproc \
qtimltflite name=stage_01_inference \
model=$HOME/models/$MODEL_NAME \
delegate=external external-delegate-path=libQnnTFLiteDelegate.so \
external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" \
qtimlpostprocess name=stage_01_postproc module=yolov8 \
labels=$HOME/labels/$LABELS_NAME \
settings="{\"confidence\": 51.0}" \
filesrc location=$HOME/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \
v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \
tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! \
fpsdisplaysink name=display sync=false text-overlay=false \
video-sink="waylandsink" \
t. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! \
stage_01_inference. stage_01_inference. ! queue ! \
stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! obj_mux. \
2> /tmp/trace.log
Parse Results
gst-stats-1.0 /tmp/trace.log
Reading the Output
Element Latency Statistics:
stage_01_preproc.src: mean=2.19ms min=1.24ms max=12.2ms ← preprocessing
stage_01_inference.src: mean=5.71ms min=4.95ms max=6.78ms ← inference (NPU)
stage_01_postproc.src: mean=0.24ms min=0.12ms max=0.44ms ← postprocessing
obj_mux.src: mean=0.76ms min=0.14ms max=2.03ms ← metadata sync
At 30 fps (33 ms budget per frame), the typical AI pipeline uses ~37% of the frame budget, leaving headroom for additional stages.
Measure FPS Live (On-Screen)
Add fpsdisplaysink in place of waylandsink to see FPS on the display in real time:
... ! qtivoverlay ! fpsdisplaysink sync=false video-sink="waylandsink sync=false"
GST_TRACER (inferences/sec)
From the tracer output, calculate inferences/sec:
inferences/sec = 1 / mean_inference_latency
mean = 5.71 ms → 1 / 0.00571 ≈ 175 inferences/sec
pidstat (CPU/Memory alongside inference)
# Run in a second terminal while the pipeline is running
pidstat -u -r -p $(pgrep gst-launch-1.0) 1
%usr, %sys) and memory (RSS, VSZ) for the pipeline process.
Frame Drops
Frame drops manifest as:
- Stuttering on the display
queue elements reporting overflows in debug logs (--gst-debug=4)- Tracer showing high
max latency on inference (stage_01_inference)
Check for queue overflows:
gst-launch-1.0 --gst-debug=4,queue:5 ...
WARN queue gstqueue.c:... overrun ...
max-size-time=0 max-size-bytes=0 max-size-buffers=3 to queues around inference:
... ! queue max-size-buffers=3 ! stage_01_preproc. ...
Slow Throughput
If end-to-end latency is higher than expected:
- Run the latency tracer to find which stage is the bottleneck.
- Check if the inference delegate is correct — GPU delegate is much slower than HTP (NPU):
# Slow — GPU
qtimltflite delegate=gpu ...
# Fast — NPU via HTP
qtimltflite delegate=external \
external-delegate-path=libQnnTFLiteDelegate.so \
external-delegate-options="QNNExternalDelegate,backend_type=htp;"
- Check model quantization — W8A8 (quantized) is significantly faster on the NPU than float32.
Memory Copies
Unnecessary buffer copies add latency. To detect them, enable memory tracing:
GST_DEBUG=4,GST_BUFFER:5 gst-launch-1.0 ... 2>&1 | grep -i "copy\|memcpy\|alloc"
capture-io-mode=4 output-io-mode=4 on decoder/encoder elements to enable zero-copy DMA buffer passing:
v4l2h264dec capture-io-mode=4 output-io-mode=4
5. Trace Buffers
Buffer tracing lets you follow a specific buffer through the pipeline to diagnose where it is dropped, delayed, or corrupted.
Enable Buffer Tracing
GST_DEBUG=GST_SCHEDULING:5 gst-launch-1.0 ... 2>&1 | head -200
PTS (presentation timestamp), DTS, size, and which pad it passed through.
Check for PTS Discontinuities
If frames are out of order or the pipeline stalls, look for timestamp jumps:
GST_DEBUG=GST_SCHEDULING:5 gst-launch-1.0 ... 2>&1 | grep -i "pts\|discont\|gap"
Use the Dot Graph (Pipeline Visualization)
Dump the pipeline as a .dot file to visually inspect element connections:
GST_DEBUG_DUMP_DOT_DIR=/tmp gst-launch-1.0 -e ... &
# After pipeline is running:
dot -Tpng /tmp/*.dot -o pipeline.png
6. Debug Camera Issues
Common Errors
QMMF Recorder StartCamera Failed:
ERROR qticamsrc qmmf_source_context.cc: QMMF Recorder StartCamera Failed!
WARN qticamsrc qmmf_source.c: error: Failed to Open Camera!
gst-launch-1.0 invocation.
Enable Camera Server Debug Logs
GST_DEBUG=2,qticamsrc:6 gst-launch-1.0 \
qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
waylandsink fullscreen=true sync=false
Camera Not Detected
# List available V4L2 devices
v4l2-ctl --list-devices
# Check camera capabilities
v4l2-ctl -d /dev/video0 --all
ISP Camera (Config #2 / qticamsrc)
To switch from libcamera (Config #1) to qticamsrc (Config #2), write "camx" to the EFI variable:
echo -n "camx" > /var/data
efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -w -f /var/data
sync && reboot
Verify Camera Stream is Running
# Quick sanity check — should show frames on display
gst-launch-1.0 \
qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \
waylandsink fullscreen=true sync=false
7. Quick Diagnostic Checklist
Run this sequence when a pipeline doesn’t work:
# Step 1 — Check GStreamer can see the plugins
gst-inspect-1.0 qtimltflite
gst-inspect-1.0 qtimlvconverter
gst-inspect-1.0 qticamsrc
# Step 2 — Run with level 3 logs
gst-launch-1.0 --gst-debug=3 [your pipeline] 2>&1 | grep -E "ERROR|WARN|FIXME"
# Step 3 — Check camera server
sudo killall cam-server # reset if camera errors appear
# Step 4 — Add latency tracing
GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG="GST_TRACER:7" \
gst-launch-1.0 [your pipeline] 2> /tmp/trace.log
gst-stats-1.0 /tmp/trace.log
8. Report an Issue
When filing a bug or asking for help, include:
- Platform — Device model (e.g. RB3 Gen 2), OS (Ubuntu / Qualcomm Linux), QIM SDK version
- Full pipeline command — the exact
gst-launch-1.0 or Python script
- Debug log — captured with
--gst-debug=5 2> debug.log
- Tracer output — from
gst-stats-1.0 if a performance issue
- Error message — the exact error line from the log
- Expected vs actual behaviour
# Collect everything in one command
GST_DEBUG=5 GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG_FILE=/tmp/debug.log \
gst-launch-1.0 [your pipeline]
gst-stats-1.0 /tmp/debug.log >> /tmp/debug.log
# Attach /tmp/debug.log to your issue
