doca-dpa-hl-tracer

# DOCA DPA High-Level Tracer

**Where to start:** This is a tool skill for invoking
`doca_dpa_hl_tracer` — the documented host-side CLI that
captures DPA-side execution traces in higher-level terms
(DPA programming events: kernel entry / exit, sync points,
comm primitive calls, RDMA WR submission, completions) rather
than raw cycle counts. Open [`TASKS.md`](TASKS.md) and start at
[`## configure`](TASKS.md#configure) for the
mode-vs-overhead decision and the JSON config layout, then
[`## run`](TASKS.md#run) for the
capture → decode → render pipeline. Open
[`CAPABILITIES.md`](CAPABILITIES.md) when the question is
*what does this tool actually trace*, *which DPA programming
events does it expose*, *what is the trace-overhead vs
fidelity tradeoff*, or *how does it slot into a DPA debug
loop alongside [`doca-dpa`](../../libs/doca-dpa/SKILL.md)
and [`doca-debug`](../../doca-debug/SKILL.md)*. If DPA is not
the right surface for the user's question (e.g. the bug is
host-side, the bug is in the DPACC-produced image, the user
wants raw cycle counts), the path-selection rule in
[`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
routes the agent before any capture is attempted.

## Example questions this skill answers well

The CLASSES of `doca_dpa_hl_tracer` questions this skill is
built to answer, each with one worked example. The class is
the load-bearing piece; the worked example is one instance.

- **"My DPA kernel is doing the wrong thing — where do I
  look?"** — worked example: *"my host-side
  `doca_dpa_kernel_launch_update_*` completes, but the
  kernel's reported result is wrong; no host-side
  `DOCA_ERROR_*`"*. Answered by the *when DPA-side
  high-level tracing is the right surface* gate in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  + the
  capture → decode → render flow in
  [`TASKS.md ## run`](TASKS.md#run) + the
  *which DPA programming events to focus on* rule in
  [`TASKS.md ## debug`](TASKS.md#debug).
- **"My DPA kernel is slow at a granularity that doesn't
  show up in cycle profiles — how do I see kernel-entry to
  first-comm-call latency?"** — worked example: *"my DPA
  kernel runs but the time between launch and the first
  RDMA WR submission is bigger than I expected"*. Answered
  by the event-taxonomy table in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  + the iterative loop in
  [`TASKS.md ## test`](TASKS.md#test) which treats trace
  overhead, mode (`TRACE` vs `CRIT`), and capture window
  as axes to tune.
- **"How do I capture a trace without burying the DPA in
  observation overhead?"** — worked example: *"`TRACE` mode
  is producing too much data and my measured DPA latency
  went up by 2x compared to without the tracer"*. Answered
  by the mode-vs-overhead tradeoff in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  + the `CRIT`-first guidance in
  [`TASKS.md ## configure`](TASKS.md#configure) (start
  with critical-events-only; widen to `TRACE` only when the
  bug demands per-event detail).
- **"My trace file got truncated mid-run — how should I
  configure the file-size limits?"** — worked example:
  *"binary trace file hit 5 GB and the capture stopped"*.
  Answered by the `log_file_max_size_in_bytes` /
  `bin_file_max_size_in_bytes` / `file_size_limit_policy`
  triple in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  + the JSON config layout in
  [`TASKS.md ## configure`](TASKS.md#configure).
- **"Is the tracer on my install, and is it paired with the
  matching `doca-dpa` library and DPACC compiler
  version?"** — worked example: *"is the tracer ABI on my
  install compatible with the DPA image my DPACC just
  produced?"*. Answered by the version-overlay in
  [`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility),
  which redirects to the canonical
  [`doca-version`](../../doca-version/SKILL.md) chain and
  adds the *tracer ↔ `doca-dpa` library ↔ DPACC compiler*
  match rule.
- **"The capture file looks empty / decode failed — is the
  install broken, no events fired, or am I tracing the
  wrong thing?"** — worked example: *"`doca_dpa_hl_tracer`
  ran, wrote a file, but the parser shows zero events"*.
  Answered by the layered error taxonomy in
  [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy)
  (install / device-binding / DPA-image-instrumented /
  capture-window / decode-vs-elf / overhead-saturated /
  version / cross-cutting) + the layered walk in
  [`TASKS.md ## debug`](TASKS.md#debug).

## Audience

This skill serves **external developers, platform operators,
and AI agents who have already brought up a DPA-side workload
through [`doca-dpa`](../../libs/doca-dpa/SKILL.md) and now
need higher-level visibility into what the DPA kernel is
actually doing on the wire** — DPA programming events
ordering, sync gaps, comm-call latencies, RDMA-WR / completion
timing — without dropping all the way down to raw cycle
counters. Concretely:

- A DPA developer who can launch their kernel cleanly from
  the host side but whose kernel's *result* is wrong or
  whose *DPA-side performance* is below expectation, and
  who needs a DPA-side ground truth before triaging.
- A platform operator running a DPA-using workload (RDMA
  offload from accelerator, custom CC algorithm via
  `doca-pcc`) and needs to localize a regression to the
  DPA side without instrumenting the application.
- An AI agent producing a *DPA-side trace report* as
  evidence for the host-side
  [`doca-dpa TASKS.md ## debug`](../../libs/doca-dpa/TASKS.md#debug)
  ladder when the host side reports clean completions but
  the DPA-side behaviour is wrong.

It is **not** for users debugging the tracer binary itself,
**not** a substitute for the live public DOCA DPA Tools
guide, **not** the right place for users learning how to
write a DPA kernel (that audience b