deepstream-sop

# DeepStream SOP Inference Microservice Skill

This skill guides AI coding assistants in building, extending, and debugging the
**NVIDIA DeepStream SOP (Standard Operating Procedure) Inference Microservice** —
a GPU-accelerated pipeline for temporal action detection and VLM-based SOP compliance
monitoring on industrial video feeds.

**Reference repository**: https://github.com/NVIDIA/sop-monitoring-blueprints/tree/main/microservices/sop-inference-bp
**Local reference code**: `sop-inference-bp/` directory (from a local clone of the repository)

---

## Models

Model-agnostic at both inference stages — swap via env var (and Triton dir for GEBD).

| Stage | Role | Model class | Default | Swap via |
|------|------|-------------|---------|----------|
| Stage 1 (CV) | Per-frame boundary scoring → chunk segmentation | **Generic Event Boundary Detection (GEBD)** | **DDM** ([MCG-NJU/DDM](https://github.com/MCG-NJU/DDM)) via Triton Python backend | Replace `triton_model_repo/<model>/` + `DDM_MODEL_PATH` (§ 5) |
| Stage 3 (VLM) | Per-chunk action classification | Vision-language model via vLLM | **Cosmos Reason 1 7B** (Reason 2 also supported) | Set `VLLM_MODEL_PATH` to a different HF ID or local path |

"GEBD" = swappable Stage-1 slot; "DDM" = the default architecture (terms used interchangeably).

**Chunking is selectable per request** (§ 2): default `ddm-net` uses GEBD; `uniform` produces fixed-length chunks and **bypasses Stage-1 GEBD** (§ 3, § 6). DDM temporal window is configurable via `FRAMES_PER_SIDE` / `SEQUENCE_BATCH` (§ 4, § 5), with optional **TensorRT** (§ 5).

---

## Architecture Overview

Runs in a Docker container (`nvds-action-sop`) alongside a Kafka container. Full diagram: [`references/sop_architecture.svg`](references/sop_architecture.svg).

**Data flow through the 4-stage `SOPVideoProcessor` pipeline (per-request):**

```
Input Sources                    Docker Container: nvds-action-sop
─────────────                    ──────────────────────────────────────────────────
Video Files ──┐                  FastAPI Server (port 8300)
RTSP Streams ─┤── base64/       ├─ /v1/chat/completions → SOPProcessManager
Basler Camera ┘   file/rtsp/       │
                  camera           │ ModelInitializer: VLM first, then DDM dummy pipeline
                                   │ 4 Thread Pools: cv(32), clip(32), vlm(64), vlm_req(64)
                                   │
                                   ▼ SOPVideoProcessor (per-request)
                                   ┌────────────────────────────────────────────────┐
                                   │ Stage 1: DeepStream Pipeline (GPU)             │
                                   │   Source → nvstreammux → tee1                  │
                                   │    ├─[inference] queue1 → nvdspreprocess       │
                                   │    │  → nvinferserver (Triton CAPI + DDM)      │
                                   │    │  → InferOutputTensorParser → score_queue  │
                                   │    ├─[frames]  queue3 → nvvideoconvert         │
                                   │    │  → capsfilter → appsink                   │
                                   │    │  → DecodedFrameRetriever → frame_queue    │
                                   │    └─[RTSP out] queue → convert → H.264 enc    │  (optional, § 18)
                                   │       → rtppay → udpsink → RTSPServer (§ 18)   │  opt-in only
                                   │              │ boundary scores                 │
                                   │              ▼                                 │
                                   │ Stage 2: Clip Post-Process                     │
                                   │   Boundary detection → chunk segmentation      │
                                   │              │ video frames + timestamps        │
                                   │              ▼                                 │
                                   │ Stage 3: VLM Inference                         │
                                   │   Embedded vLLM (Cosmos Reason 1/2)            │
                                   │   Frame sampling at VLM_FPS → classification   │
                                   │              │ action labels                    │
                                   │              ▼                                 │
                                   │ Stage 4: SOP Checker                           │
                                   │   Sequence validation → missing/misordered     │
                                   │              │ chunk results                    │
                                   │              ▼                                 │
                                   │         final_queue                            │
                                   └────────────────────────────────────────────────┘
                                          │
Output                                    ▼
──────                             ┌─────────────────┐
SSE Stream (chat.completion.chunk) │ Kafka Messages   │
Non-streaming (chat.completion)    │ (JSON/Protobuf)  │
Prometheus metrics (/v1/metrics)   └────────┬────────┘
                                            ▼
                                   Docker Container: kafka
                                   (apache/kafka:3.7.0)
```

---

## Section Index

Each section is a standalone file in `references/` — load only what your task needs.

| § | File | Responsibility |
|---|------|---------------|
| 1 | [`skill_01_fastapi_endpoints.md`](references/skill_01_fastapi_endpoints.md) | FastAPI endpoints, server init, Prometheus metrics |
| 2 | [`skill_02_pydantic_schemas.md`](references/skill_02_pydantic_schemas.md) | Request/response Pydantic models (`api_types.py`) |
| 3 | [`skill_03_deepstream_pipeline.md`](references/skill_03_deepstream_pipeline.md) | DeepStream pyservicemaker pipeline, tensor parser, dummy pipeline |
| 4 | [`sk