api-relay-audit

# API Relay Audit for Hermes Agent

## Overview

This skill runs `api-relay-audit`, a zero-dependency 14-step security audit for third-party AI API relays and proxy services. It checks relay behavior while keeping API relay audit, prompt injection audit, model substitution signals, and Web3 relay audit as separate query families with separate evidence boundaries.

Use the standalone `audit.py` path by default. It only needs Python 3 and `curl`, which makes it suitable for local Hermes terminal sessions and sandboxed execution. On Windows, run the POSIX shell recipes from Git Bash or an equivalent shell; direct `python audit.py ...` invocations also work from PowerShell when the environment variables are set.

## When to Use

- The user asks whether an AI API relay, proxy API, resale key, or "API relay" is safe.
- The user provides a relay base URL and wants an evidence-based risk report.
- The user suspects hidden prompts, identity substitution, response tampering, context truncation, tool-call/package substitution, or stream anomalies.
- The user wants to audit Web3/wallet safety behavior with `--profile web3` or `--profile full`.

Do not use this skill for general model benchmarking, provider price comparison, or legal/security certification. The output is a technical audit report, not a guarantee that a service is safe.

Keep these query families separate:

| Query family | Use when | Profile / evidence boundary |
|---|---|---|
| API relay audit | The user wants a local report for a relay, mirror, gateway, LLM proxy, or resale API. | Default `general`; report is evidence, not certification. |
| Prompt injection audit | The user asks about hidden prompt injection, prompt leakage, instruction override, or extraction behavior. | Steps 3-6; do not publish private prompts or secrets. |
| Model substitution signals | The user suspects model identity, route, latency, or upstream channel mismatch. | Signals from Steps 5, 10, 13, and 14 require corroboration; self-ID and fingerprints are not standalone provider proof. |
| Web3 relay audit | The user is testing wallet-sensitive agent workflows. | Use `--profile web3` or `--profile full`; Step 11 is profile-gated. |

## Install or Share

After this file is merged to the public repository, Hermes users can install it as a tap skill:

```bash
hermes skills tap add toby-bridges/api-relay-audit
hermes skills install toby-bridges/api-relay-audit/api-relay-audit
```

For direct testing without adding the whole tap:

```bash
hermes skills install toby-bridges/api-relay-audit/skills/api-relay-audit
```

## Required Inputs

| Input | How to provide it | Notes |
|---|---|---|
| Relay API key | Prefer `$API_RELAY_AUDIT_KEY` via Hermes secure env setup | Use a temporary or low-scope key when possible. |
| Base URL | Ask the user or use `$API_RELAY_AUDIT_URL` if already set | Example: `https://relay.example.com/v1`. |
| Model | Optional; default is `claude-opus-4-6` | Use the model the user plans to rely on. |
| Profile | Optional; default is `general` | Use `web3` for wallet users, `full` for complete coverage. |

Never print the raw API key in summaries, filenames, reports, shell traces, or GitHub comments. If the user pasted a key into chat, avoid repeating it and recommend rotating it after the audit if exposure matters.

## Standard Workflow

1. Confirm the target base URL, model, and profile.
2. Ensure the key is available as `$API_RELAY_AUDIT_KEY`. If it is missing, ask the user to configure it through Hermes secure setup or local `.env`, not by committing it.
3. Download the standalone script into a temporary directory unless the current repo already contains `audit.py`.
4. Run the audit and write a Markdown report.
5. Summarize only evidence from the generated report. Do not overstate safety or make policy promises.

## One-Shot Audit Recipe

Use this when the user provides a base URL and wants a normal audit:

This recipe is POSIX shell. On Windows Hermes hosts, use Git Bash for this
one-shot form; if the repository already has `audit.py`, PowerShell can run the
direct local command shown after the recipe.

```bash
set -euo pipefail

: "${API_RELAY_AUDIT_KEY:?Set API_RELAY_AUDIT_KEY through Hermes secure env setup first}"
: "${API_RELAY_AUDIT_URL:?Set API_RELAY_AUDIT_URL to the relay base URL}"

MODEL="${API_RELAY_AUDIT_MODEL:-claude-opus-4-6}"
PROFILE="${API_RELAY_AUDIT_PROFILE:-general}"
WORKDIR="$(mktemp -d)"
REPORT="$PWD/api-relay-audit-report.md"
AUDIT_SCRIPT_REF=v2.3.0

curl -fsSL \
  "https://raw.githubusercontent.com/toby-bridges/api-relay-audit/${AUDIT_SCRIPT_REF}/audit.py" \
  -o "$WORKDIR/audit.py"

python3 "$WORKDIR/audit.py" \
  --key "$API_RELAY_AUDIT_KEY" \
  --url "$API_RELAY_AUDIT_URL" \
  --model "$MODEL" \
  --profile "$PROFILE" \
  --output "$REPORT"

printf 'Report written to %s\n' "$REPORT"
```

If the current working tree is the `api-relay-audit` repository and `audit.py` exists, prefer the local file:

```bash
python3 audit.py \
  --key "$API_RELAY_AUDIT_KEY" \
  --url "$API_RELAY_AUDIT_URL" \
  --model "${API_RELAY_AUDIT_MODEL:-claude-opus-4-6}" \
  --profile "${API_RELAY_AUDIT_PROFILE:-general}" \
  --output api-relay-audit-report.md
```

## Profiles and Cost Controls

| Scenario | Recommended flags |
|---|---|
| Fast first pass | `--skip-infra --skip-context --skip-latency-variance` |
| Normal relay audit | `--profile general` |
| Web3 or wallet relay | `--profile web3` |
| Complete audit | `--profile full` |
| Suspicious relay with request-count gating | `--warmup 5` to `--warmup 20` |
| Avoid intentionally broken requests | `--skip-error-leakage` |
| Avoid streaming checks | `--skip-stream-integrity` |
| Avoid upstream channel classification | `--skip-channel-classifier` |

Warn the user before enabling `--aggressive-error-probes` because oversized probes can create metered usage on pay-as-you-go relays.

## What the 14 Steps Cover

| Step | Area | Purpose |
|---|---|---|
| 1 | Infrastructure recon | DNS, WHOIS, SSL, H