deepstream-dev

# DeepStream Development Skill

When this skill is active, **ALWAYS read the relevant reference documents** before generating code. Do NOT rely on memory - the reference documents contain critical details about exact property names, correct API usage, and common pitfalls.

## SDK and Architecture Quick Reference

### DeepStream SDK Version Requirements

- **GStreamer**: 1.24.2
- **NVIDIA Driver**: 590+
- **CUDA**: 13.1
- **TensorRT**: 10.14.1.48
- **Platforms**: Ubuntu 24.04 (x86_64 and ARM64/Jetson)

### Typical Pipeline Flow

```
Source → Stream Muxer → Inference → [Tracker] → OSD → Renderer
```
Components in `[brackets]` are **optional** -- only add them when the user explicitly requests them.

| Stage | Role | Key Element(s) | Required? |
|-------|------|-----------------|-----------|
| Source | Input from files, RTSP, cameras | `nvurisrcbin` (preferred), `nvmultiurisrcbin`, `filesrc` | Yes |
| Stream Muxer | Batches streams for inference | `nvstreammux` | Yes |
| Inference | TensorRT model execution | `nvinfer`, `nvinferserver` | Yes |
| Tracker | Multi-object tracking across frames | `nvtracker` | **Only if requested** |
| OSD | Draws bounding boxes, labels, overlays | `nvosdbin` | Yes (for visualization) |
| Renderer | Display or save output | `nveglglessink`, `nv3dsink`, `filesink` | Yes |

### Memory Model

DeepStream uses NVIDIA Video Memory Manager (NVMM) for zero-copy GPU buffer transfers. Caps strings use `memory:NVMM` to indicate GPU memory (e.g., `video/x-raw(memory:NVMM), format=NV12`).

## Critical Rules

1. **Only Add Requested Components**: Do NOT add pipeline elements the user did not ask for.
   - **Tracker (`nvtracker`)**: Only add when the user explicitly requests tracking or object IDs across frames
   - **Secondary GIEs**: Only add when the user requests classification or attribute extraction
   - **Analytics (`nvdsanalytics`)**: Only add when the user requests line crossing, ROI counting, etc.
   - **Message broker (`nvmsgbroker`/`nvmsgconv`)**: Only add when the user requests Kafka/cloud messaging
   - When in doubt, build the **minimal working pipeline** and let the user ask for additions

2. **Default to `nvurisrcbin` for Sources**: When the user says "camera", "stream", "video", or provides a file path:
   - Always use `nvurisrcbin` -- it handles RTSP, HTTP, and local files (`file://`) transparently
   - Only use `filesrc` + `qtdemux` + parser when the user explicitly needs raw file source control
   - For RTSP/live sources, also set `live-source=1` on `nvstreammux` and `sync=0` on the sink
   - Convert local paths to URI: `"file://" + os.path.abspath(path)`

3. **Metadata Iteration**: Use `.frame_items` and `.object_items` (returns iterators, NOT lists)
   - NEVER use `len()` on these - iterate to count
   - Iterator can only be consumed once

4. **Request Pad Syntax**: Use `"sink_%u"` template, NEVER literal pad names
   ```python
   pipeline.link(("decoder", "mux"), ("", "sink_%u"))  # CORRECT
   # pipeline.link(("decoder", "mux"), ("", "sink_0"))  # WRONG - will fail
   ```

5. **Platform Detection for Sinks**:
   ```python
   import platform
   sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
   ```

6. **Buffer Cloning**: Always clone buffers for async processing
   ```python
   tensor = buffer.extract(0).clone()  # CRITICAL
   ```

7. **Queue Types**:
   - `queue.Queue` → Use with `threading.Thread`
   - `multiprocessing.Queue` → Use with `multiprocessing.Process`
   - Using wrong type causes silent data loss!

8. **nvinfer Config Format**:
   - YAML: Use `property:` section (NOT `model:`), `key: value` with space after colon
   - INI: Use `[property]` section, `key=value` with equals sign
   - Section MUST be named `property`

9. **nvmsgbroker is a SINK**: Cannot have downstream elements - use `tee` to split pipeline

10. **ALL Sinks Need async=0 for Tee Splits or Dynamic Sources**: CRITICAL for state transitions
    ```python
    # When using tee splits OR dynamic sources, ALL sinks MUST have async=0
    pipeline.add("nveglglessink", "sink", {
        "sync": 0, "qos": 0,
        "async": 0  # CRITICAL - prevents state transition deadlock
    })
    ```
    **Symptom if missing**: Pipeline stays in PAUSED state, no video displays.

11. **Built-in Probe Attachment**: `measure_fps_probe` can only be attached to processing elements (e.g., `nvinfer`, `nvosdbin`), **NOT** to sink elements. Attaching to a sink raises `RuntimeError: Probe failure`.

12. **Dynamic ONNX Models Require `infer-dims`**: When the ONNX model has dynamic input shapes (e.g., exported with `dynamic=True` in Ultralytics YOLO, or with dynamic batch/height/width axes), you **MUST** add `infer-dims=C;H;W` to the nvinfer config. Without it, TensorRT sees `-1` for dynamic dimensions and fails with `setDimensions: Error Code 3`. Common values:
    - YOLO models (640 input): `infer-dims=3;640;640`
    - Models with 416 input: `infer-dims=3;416;416`
    - Models with 1280 input: `infer-dims=3;1280;1280`

13. **Ultralytics YOLO Output Format Depends on Model Generation** — newer models (v10+/v26+) output post-NMS results; older models (v8/v11) output raw pre-NMS tensors. The custom parser and `cluster-mode` **must** match the actual output:

   | Model generation | Output tensor shape | Fields | `cluster-mode` |
   |------------------|--------------------|---------------------------------|----------------|
   | v8 / v11 | `[batch, 84, 8400]` | `[features(4+80), anchors]` — raw cx/cy/w/h + class scores, no NMS | `2` (NMS) |
   | v10 / v26+ | `[batch, 300, 6]` | `[max_det, (x1,y1,x2,y2,conf,cls)]` — already post-NMS, pixel coords | `4` (none) |

   **How to identify at runtime**: log `inferDims.d[0]` and `inferDims.d[1]` inside the custom parser.
   - `d={84, 8400}` → pre-NMS (v8/v11 style)
   - `d={300, 6}` → post-NMS (v10/v26+ style)

   **Symptom of mismatch**: If `cluster-mode: 2` is used with a post-NMS `[N, 6]` output, bounding boxes appear shifted by 45°