> ## Documentation Index > Fetch the complete documentation index at: https://imsdkdocs.qualcomm.com/llms.txt > Use this file to discover all available pages before exploring further. # Debugging Guide > Developer guide for debugging, profiling, and tracing QIM SDK GStreamer pipelines. This guide covers practical debugging techniques for QIM SDK AI pipelines — from enabling verbose logs to measuring per-stage latency, tracing buffers, and diagnosing camera or performance issues. *** ## 1. Enable Debug Logs QIM SDK uses the standard GStreamer debug system. Pass `--gst-debug=` to `gst-launch-1.0`, or set the `GST_DEBUG` environment variable for Python/C++ apps. ### Debug Levels | Level | Name | Description | | ----- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 0 | none | No debug information is output. | | 1 | ERROR | Logs all fatal errors. These are errors that do not allow the core or elements to perform the requested action. The application can still recover if programmed to handle the conditions that triggered the error. | | 2 | WARNING | Logs all warnings. Typically these are non-fatal, but user-visible problems are expected to happen. | | 3 | FIXME | Logs all "fixme" messages. Those typically that a codepath that is known to be incomplete has been triggered. It may work in most cases, but may cause problems in specific instances. | | 4 | INFO | Logs all informational messages. These are typically used for events in the system that only happen once, or are important and rare enough to be logged at this level. | | 5 | DEBUG | Logs all debug messages. These are general debug messages for events that happen only a limited number of times during an object's lifetime; these include setup, teardown, change of parameters, etc. | | 6 | LOG | Logs all log messages. These are messages for events that happen repeatedly during an object's lifetime; these include streaming and steady-state conditions. This is used for log messages that happen on every buffer in an element for example. | | 7 | TRACE | Logs all trace messages. Those are message that happen very very often. This is for example is each time the reference count of a GstMiniObject, such as a GstBuffer or GstEvent, is modified. | | 9 | MEMDUMP | Logs all memory dump messages. This is the heaviest logging and may include dumping the content of blocks of memory. | ### Global Level ```bash theme={null} gst-launch-1.0 --gst-debug=3 \ filesrc location=... ! ... ``` ### Per-Element Level Override the global level for a specific plugin — useful for isolating one stage without drowning in output from everything else: ```bash theme={null} # Global = 2 (warnings), but inference plugin = 6 (verbose) gst-launch-1.0 --gst-debug=2,qtimltflite:6 \ filesrc location=... ! ... ``` ### Common Starting Points | Flag | When to use | | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--gst-debug=2` | Production runs — quiet, errors/warnings only | | `--gst-debug=2,qtimltflite:5` | Prints all ERROR and WARNING messages globally, plus ERROR, WARNING, INFO, and DEBUG messages specifically from the qtimltflite plugin | | `--gst-debug=2,qtimlvtransform:4,qtivcomposer:7` | Prints all ERROR and WARNING messages globally, ERROR, WARNING, and INFO messages from qtimlvtransform, and ERROR, WARNING, INFO, DEBUG, LOG, and TRACE messages from qtivcomposer | ### Python / C++ Apps ```bash theme={null} GST_DEBUG=3 python3 my_pipeline.py ``` ```bash theme={null} GST_DEBUG=5 ./my_pipeline_app ``` ### Redirect to File ```bash theme={null} GST_DEBUG=5 GST_DEBUG_FILE=/data/gst.log GST_DEBUG_COLOR_MODE=off gst-launch-1.0 ... ``` *** ## 2. Measure Pipeline KPIs (Latency per Stage) Use the built-in GStreamer **latency tracer** to get per-element mean/min/max processing time for every stage — no custom instrumentation required. ### Enable Tracing ```bash theme={null} GST_TRACERS="latency(flags=pipeline+element)" \ GST_DEBUG="GST_TRACER:7" \ gst-launch-1.0 -e \ qtimlvconverter name=stage_01_preproc \ qtimltflite name=stage_01_inference \ model=$HOME/models/$MODEL_NAME \ delegate=external external-delegate-path=libQnnTFLiteDelegate.so \ external-delegate-options="QNNExternalDelegate,backend_type=htp,log_level=(string)1;" \ qtimlpostprocess name=stage_01_postproc module=yolov8 \ labels=$HOME/labels/$LABELS_NAME \ settings="{\"confidence\": 51.0}" \ filesrc location=$HOME/media/$SRC_VIDEO_NAME ! qtdemux ! h264parse ! \ v4l2h264dec capture-io-mode=4 output-io-mode=4 ! video/x-raw,format=NV12 ! queue ! \ tee name=t ! qtimetamux name=obj_mux ! qtivoverlay ! \ fpsdisplaysink name=display sync=false text-overlay=false \ video-sink="waylandsink" \ t. ! queue ! stage_01_preproc. stage_01_preproc. ! queue ! \ stage_01_inference. stage_01_inference. ! queue ! \ stage_01_postproc. stage_01_postproc. ! text/x-raw ! queue ! obj_mux. \ 2> /tmp/trace.log ``` ### Parse Results ```bash theme={null} gst-stats-1.0 /tmp/trace.log ``` ### Reading the Output ``` Element Latency Statistics: stage_01_preproc.src: mean=2.19ms min=1.24ms max=12.2ms ← preprocessing stage_01_inference.src: mean=5.71ms min=4.95ms max=6.78ms ← inference (NPU) stage_01_postproc.src: mean=0.24ms min=0.12ms max=0.44ms ← postprocessing obj_mux.src: mean=0.76ms min=0.14ms max=2.03ms ← metadata sync ``` At 30 fps (33 ms budget per frame), the typical AI pipeline uses \~37% of the frame budget, leaving headroom for additional stages. ### Measure FPS Live (On-Screen) Add `fpsdisplaysink` in place of `waylandsink` to see FPS on the display in real time: ```bash theme={null} ... ! qtivoverlay ! fpsdisplaysink sync=false video-sink="waylandsink sync=false" ``` *** ## 3. Measure Inference Performance ### GST\_TRACER (inferences/sec) From the [tracer output](debugging#enable-tracing), calculate inferences/sec: ``` inferences/sec = 1 / mean_inference_latency ``` Example: `mean = 5.71 ms` → `1 / 0.00571 ≈ 175 inferences/sec` ### pidstat (CPU/Memory alongside inference) ```bash theme={null} # Run in a second terminal while the pipeline is running pidstat -u -r -p $(pgrep gst-launch-1.0) 1 ``` This reports per-second CPU (`%usr`, `%sys`) and memory (`RSS`, `VSZ`) for the pipeline process. *** ## 4. Debug Performance Issues ### Frame Drops Frame drops manifest as: * Stuttering on the display * `queue` elements reporting overflows in debug logs (`--gst-debug=4`) * Tracer showing high `max` latency on inference (`stage_01_inference`) **Check for queue overflows:** ```bash theme={null} gst-launch-1.0 --gst-debug=4,queue:5 ... ``` Look for lines like: ``` WARN queue gstqueue.c:... overrun ... ``` **Fix:** Add `max-size-time=0 max-size-bytes=0 max-size-buffers=3` to queues around inference: ```bash theme={null} ... ! queue max-size-buffers=3 ! stage_01_preproc. ... ``` ### Slow Throughput If end-to-end latency is higher than expected: 1. Run the [latency tracer](debugging#2-measure-pipeline-kpis-latency-per-stage) to find which stage is the bottleneck. 2. Check if the inference delegate is correct — GPU delegate is much slower than HTP (NPU): ```bash theme={null} # Slow — GPU qtimltflite delegate=gpu ... # Fast — NPU via HTP qtimltflite delegate=external \ external-delegate-path=libQnnTFLiteDelegate.so \ external-delegate-options="QNNExternalDelegate,backend_type=htp;" ``` 3. Check model quantization — W8A8 (quantized) is significantly faster on the NPU than float32. ### Memory Copies Unnecessary buffer copies add latency. To detect them, enable memory tracing: ```bash theme={null} GST_DEBUG=4,GST_BUFFER:5 gst-launch-1.0 ... 2>&1 | grep -i "copy\|memcpy\|alloc" ``` Use `capture-io-mode=4 output-io-mode=4` on decoder/encoder elements to enable zero-copy DMA buffer passing: ```bash theme={null} v4l2h264dec capture-io-mode=4 output-io-mode=4 ``` *** ## 5. Trace Buffers Buffer tracing lets you follow a specific buffer through the pipeline to diagnose where it is dropped, delayed, or corrupted. ### Enable Buffer Tracing ```bash theme={null} GST_DEBUG=GST_SCHEDULING:5 gst-launch-1.0 ... 2>&1 | head -200 ``` Each line shows a buffer's `PTS` (presentation timestamp), `DTS`, `size`, and which pad it passed through. ### Check for PTS Discontinuities If frames are out of order or the pipeline stalls, look for timestamp jumps: ```bash theme={null} GST_DEBUG=GST_SCHEDULING:5 gst-launch-1.0 ... 2>&1 | grep -i "pts\|discont\|gap" ``` ### Use the Dot Graph (Pipeline Visualization) Dump the pipeline as a `.dot` file to visually inspect element connections: ```bash theme={null} GST_DEBUG_DUMP_DOT_DIR=/tmp gst-launch-1.0 -e ... & # After pipeline is running: dot -Tpng /tmp/*.dot -o pipeline.png ``` *** ## 6. Debug Camera Issues ### Common Errors **QMMF Recorder StartCamera Failed:** ``` ERROR qticamsrc qmmf_source_context.cc: QMMF Recorder StartCamera Failed! WARN qticamsrc qmmf_source.c: error: Failed to Open Camera! ``` **Fix:** Kill and restart the camera server: ```bash theme={null} sudo killall cam-server ``` Then re-run the pipeline. The camera server restarts automatically on the next `gst-launch-1.0` invocation. ### Enable Camera Server Debug Logs ```bash theme={null} GST_DEBUG=2,qticamsrc:6 gst-launch-1.0 \ qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \ waylandsink fullscreen=true sync=false ``` ### Camera Not Detected ```bash theme={null} # List available V4L2 devices v4l2-ctl --list-devices # Check camera capabilities v4l2-ctl -d /dev/video0 --all ``` ### ISP Camera (Config #2 / qticamsrc) To switch from libcamera (Config #1) to `qticamsrc` (Config #2), write `"camx"` to the EFI variable: ```bash theme={null} echo -n "camx" > /var/data efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -w -f /var/data sync && reboot ``` ### Verify Camera Stream is Running ```bash theme={null} # Quick sanity check — should show frames on display gst-launch-1.0 \ qticamsrc name=camsrc ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \ waylandsink fullscreen=true sync=false ``` *** ## 7. Quick Diagnostic Checklist Run this sequence when a pipeline doesn't work: ```bash theme={null} # Step 1 — Check GStreamer can see the plugins gst-inspect-1.0 qtimltflite gst-inspect-1.0 qtimlvconverter gst-inspect-1.0 qticamsrc # Step 2 — Run with level 3 logs gst-launch-1.0 --gst-debug=3 [your pipeline] 2>&1 | grep -E "ERROR|WARN|FIXME" # Step 3 — Check camera server sudo killall cam-server # reset if camera errors appear # Step 4 — Add latency tracing GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG="GST_TRACER:7" \ gst-launch-1.0 [your pipeline] 2> /tmp/trace.log gst-stats-1.0 /tmp/trace.log ``` *** ## 8. Report an Issue When filing a bug or asking for help, include: 1. **Platform** — Device model (e.g. RB3 Gen 2), OS (Ubuntu / Qualcomm Linux), QIM SDK version 2. **Full pipeline command** — the exact `gst-launch-1.0` or Python script 3. **Debug log** — captured with `--gst-debug=5 2> debug.log` 4. **Tracer output** — from `gst-stats-1.0` if a performance issue 5. **Error message** — the exact error line from the log 6. **Expected vs actual behaviour** ```bash theme={null} # Collect everything in one command GST_DEBUG=5 GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG_FILE=/tmp/debug.log \ gst-launch-1.0 [your pipeline] gst-stats-1.0 /tmp/debug.log >> /tmp/debug.log # Attach /tmp/debug.log to your issue ```