doca-gpunetio-ib-write-lat

# DOCA GPUNetIO ib_write_lat

**Where to start:** This is a tool skill for the GPUNetIO-
flavored `ib_write_lat` benchmark shipped under
`doca/tools/gpunetio_ib_write_lat/` (a client + server pair,
built from source against the installed DOCA via `meson`).
It measures the latency of an RDMA WRITE work request when
the WR is posted **from a CUDA kernel through the
doca-gpunetio device-side surface**, in a ping-pong cadence.
Open [`TASKS.md`](TASKS.md) and start at
[`## configure`](TASKS.md#configure) for the GPU-NIC pairing
precondition and the build pattern; jump to
[`## run`](TASKS.md#run) for the single-iteration smoke
flow. Open [`CAPABILITIES.md`](CAPABILITIES.md) when the
question is *what this tool actually measures*, *how it
differs from the GPI sister tool on the same physical
operation*, or *how to interpret the half-iter / full-iter
/ CUDA-side usec output and the median / p99 / jitter
characterization*. If DOCA is not installed yet, route to
[`doca-setup`](../../doca-setup/SKILL.md) first; if the
user is still deciding between GPUNetIO and GPI as a
programming surface, the picture in
[`../../libs/doca-gpunetio/CAPABILITIES.md#capabilities-and-modes`](../../libs/doca-gpunetio/CAPABILITIES.md#capabilities-and-modes)
and
[`../../libs/doca-gpi/CAPABILITIES.md#capabilities-and-modes`](../../libs/doca-gpi/CAPABILITIES.md#capabilities-and-modes)
is the first stop.

## Example questions this skill answers well

The CLASSES of `doca-gpunetio-ib-write-lat` questions this
skill is built to answer, each with one worked example. The
class is the load-bearing piece; the worked example is one
instance.

- **"What GPU-init RDMA-WRITE latency / jitter can the
  GPUNetIO path deliver for a real-time / control-loop
  workload?"** — worked example: *"measure per-iteration
  WRITE latency between two hosts with an H100 +
  ConnectX-7 on each side, target the median and the p99
  separately"*. Answered by the GPU-NIC pairing
  precondition in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  + the bring-up flow in
  [`TASKS.md ## configure`](TASKS.md#configure) +
  [`TASKS.md ## run`](TASKS.md#run).
- **"This is the GPUNetIO tool — how does the latency
  number differ from the GPI programming surface?"** —
  worked example: *"the team is using GPI; should I expect
  GPUNetIO to beat / tie / lose vs GPI?"*. Answered by the
  *"same physical operation, different runtime framework"*
  rule in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  + the cross-link to the GPI library skill
  [`../../libs/doca-gpi/CAPABILITIES.md`](../../libs/doca-gpi/CAPABILITIES.md)
  (note: `doca/tools/` ships no GPI `ib_write_lat`
  benchmark binary — GPI is a programming surface, not a
  shipped benchmark tool).
- **"Median vs p99 vs jitter — which one is the actual
  answer for a real-time control loop?"** — worked
  example: *"my control loop has a deadline; the median
  is well under the budget but p99 spikes; do I quote
  the median or the p99?"*. Answered by the
  median-vs-p99-vs-jitter rule in
  [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability)
  + the eval-loop overlay in
  [`TASKS.md ## test`](TASKS.md#test).
- **"What is the latency-vs-batching trade-off specific
  to GPU-init RDMA?"** — worked example: *"my CUDA kernel
  could batch multiple WRs to amortize the GPU-side
  overhead; what does that buy me on latency vs what does
  it cost me?"*. Answered by the
  latency-vs-batching trade-off in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes).
- **"What version of DOCA + CUDA Toolkit do I need for
  this binary to build and run?"** — worked example: *"my
  install has DOCA at one semver and CUDA at another;
  will the ToT-shipped `gpunetio_ib_write_lat` even
  link?"*. Answered by the version overlay in
  [`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility).
- **"How do I read the half-iter / full-iter / CUDA-side
  usec columns?"** — worked example: *"the binary printed
  half-iter, full-iter, and a CUDA-side number — what is
  the right column to quote for one-way latency vs
  round-trip vs cross-check?"*. Answered by the column-
  semantics rule in
  [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability).

## Audience

This skill serves **external developers and performance
engineers who need a reproducible measurement of the
latency of an RDMA WRITE WR when the WR is posted from a
CUDA kernel through doca-gpunetio**, on the user's actual
install and GPU-NIC pair. Concretely:

- A developer designing a GPU-resident real-time control
  loop and deciding whether the GPUNetIO path's tail
  latency fits the deadline.
- A platform operator validating a tuning change (NUMA
  pinning, GPU PCIe placement, IB device choice, GID
  index, NIC firmware burn) by re-running this benchmark
  against the new state.
- An SRE / performance engineer producing a *"this is the
  GPUNetIO-driven WRITE latency on this GPU-NIC pair
  today, with median + p99 + jitter"* artifact downstream
  consumers can cite.
- An AI agent answering *"is the doca-gpunetio latency
  budget acceptable for this real-time workload class"*
  honestly — with measured numbers, the build +
  invocation that produced them, and the GPU + NIC +
  DOCA version that scopes them — rather than guessing.

It is **not** for users debugging the `doca-gpunetio`
library itself (route to
[`../../libs/doca-gpunetio/SKILL.md`](../../libs/doca-gpunetio/SKILL.md)),
and **not** a substitute for the `perftest` upstream
`ib_write_lat` (which measures CPU-initiated WRITE
latency).

## Language scope

The `doca-gpunetio-ib-write-lat` tool is shipped as **C
plus CUDA `.cu` translation units** under
`doca/tools/gpunetio_ib_write_lat/`, split into a
`client/` subtree, a `server/` subtree, and a `common/`
subtree shared between them (per the verified file layout:
`client/{main.c,perftest.{c,h},meson.bu