hf-cloud-sagemaker-production-defaults

# SageMaker Production Defaults

The difference between a demo endpoint and one you can leave running is: it scales with traffic, it tells you when it breaks, and you can debug it later. This skill makes those three the default rather than optional extras.

By the time this skill runs, the planner has chosen a real-time endpoint, IAM has a usable role, and image-selection has resolved a container URI + AMI version. This skill turns those into an actual deployment.

## What gets created

For every endpoint, the skill creates these as a unit:

1. **SageMaker Model** — image + env vars + execution role + S3 artifacts
2. **Endpoint config** — instance type, initial count, optional data capture
3. **Endpoint** — the real-time endpoint serving inference
4. **Autoscaling target + policy** — target tracking on invocations per instance
5. **CloudWatch alarms** — latency, errors, platform overhead

Data capture (logging requests/responses to S3) is **off by default** — useful for debugging but creates ongoing S3 costs the user didn't necessarily ask for. Enable with `--enable-data-capture`.

All resources get a consistent tag set including `CreatedBy=agentic-deploy-skills` for later cleanup.

Defaults and reasoning in `references/deployment-template.md`.

## Running the deployment

For a text-generation LLM (vLLM):

```bash
python scripts/deploy.py \
    --model-name qwen3-medical \
    --image-uri "$IMAGE_URI" \
    --inference-ami-version "$AMI" \
    --role-arn "$ROLE_ARN" \
    --instance-type ml.g5.xlarge \
    --region "$REGION" \
    --env SM_VLLM_MODEL=Qwen/Qwen3-0.6B \
    --env SM_VLLM_HOST=0.0.0.0 \
    --env SM_VLLM_TRUST_REMOTE_CODE=true \
    --env SM_VLLM_MAX_MODEL_LEN=4096
```

For an embedding model (TEI, often on CPU):

```bash
python scripts/deploy.py \
    --model-name bge-large-embeddings \
    --image-uri "$IMAGE_URI" \
    --role-arn "$ROLE_ARN" \
    --instance-type ml.c6i.2xlarge \
    --region "$REGION" \
    --env HF_MODEL_ID=BAAI/bge-large-en-v1.5
```

Note: TEI deployments **do not** need `--inference-ami-version`. That flag is vLLM-specific. TEI env vars are also simpler (`HF_MODEL_ID` instead of `SM_VLLM_*`, no host or trust-remote-code to configure).

Where each value comes from:

| Parameter | Source |
|---|---|
| `--image-uri` | `hf-cloud-serving-image-selection` — agent reads from the AWS DLC catalog page |
| `--inference-ami-version` | `hf-cloud-serving-image-selection` — required for vLLM tags containing cu130+ |
| `--role-arn` | `hf-cloud-sagemaker-iam-preflight` (`check_role.py`) |
| `--region` | `hf-cloud-aws-context-discovery` |
| `--instance-type` | User input or planner recommendation |
| `--env` | Model-specific; see `hf-cloud-serving-image-selection` for required `SM_VLLM_*` vars |
| `--model-s3-uri` | Optional — S3 path to model artifacts; omit if loading from HF Hub |

The script creates resources in order with error handling, waits for `InService` (up to 30 min), surfaces failure reasons, registers autoscaling and alarms, and prints a summary including the teardown command. Outputs a JSON blob on stdout with endpoint/config/model names for downstream scripting.

The scripts ship with this skill. If the installed copy is missing the `scripts/` directory (some harnesses copy only SKILL.md on install), fetch them from the source repo rather than re-implementing them from this description.

**Cold-start expectation**: when the model loads from HF Hub, the download happens inside the container after the endpoint starts — 5–15+ minutes to InService is normal, not a failure. `deploy.py` waits 30 minutes; if you write custom wait code, don't time out at 15. Pre-staging weights in S3 (`--model-s3-uri`) cuts this and removes the Hub dependency.

## InService is not success — smoke-test before declaring victory

`InService` only means the container answered `/ping`. In MMS-based containers (HF Inference Toolkit) the Java front-end answers pings even while the Python worker crash-loops — an endpoint can be InService and serve nothing. Two checks, always:

1. **One real invocation.**
   - Real-time: `invoke_endpoint.py` (below) with a minimal payload; require an HTTP 200 with a sane body.
   - Async: upload one input to S3, call `invoke-endpoint-async`, poll the output URI for a few minutes (see "Invoking async endpoints"). A result object = success; an object at the failure URI, or nothing appearing, = broken.
2. **Scan the endpoint logs for worker-crash markers** — catches the crash-loop case even when the smoke request merely times out:

   ```bash
   aws logs filter-log-events \
       --log-group-name /aws/sagemaker/Endpoints/<endpoint-name> \
       --filter-pattern '?"Worker died" ?"Load model failed" ?"ImportError"' \
       --region <region> --max-items 5
   ```

Only report the deployment complete after both pass. If the log scan hits, surface the actual traceback from CloudWatch — not the InService status.

## Testing a real-time endpoint

Once the endpoint is `InService`, test it with the bundled helper. It is cross-platform and **BOM-safe** — use it instead of hand-writing a payload file and calling `invoke-endpoint` directly:

```bash
# macOS / Linux
python3 scripts/invoke_endpoint.py \
    --endpoint-name <endpoint-name> \
    --payload '{"inputs": "Hello"}' \
    --region "$REGION"
```

```powershell
# Windows (PowerShell)
python scripts\invoke_endpoint.py `
    --endpoint-name <endpoint-name> `
    --payload-file payload.json `
    --region $REGION
```

It accepts either `--payload '<json>'` (inline) or `--payload-file <path>`, validates JSON, writes the request body as plain UTF-8, invokes the endpoint, and prints the response body to stdout.

### The UTF-8 BOM gotcha (Windows)

If you write the request payload yourself on Windows, **do not** use `Set-Content -Encoding UTF8` — depending on the PowerShell version it prepends a UTF-8 byte-order mark (BOM). SageMaker's JSON parser rejects a BOM with a 400 `ModelError`:

```
Unexpected UTF-8 BOM (d