remote-compute

# Remote compute over SSH

Run heavy work on the user's own machines over SSH with their own keys or their
own interactive sign-in — you never install anything remote and never handle
credentials yourself. A machine may be a plain server (CPU or GPU, no scheduler)
or a Slurm cluster.

## 0 · Always run ssh through the app's config

Every `ssh`, `scp` and `rsync` below passes `-F "$OPENSCIENCE_SSH_CONFIG"`. Write
it as `$SSHCFG` after defining it once per command you send:

```bash
SSHCFG="${OPENSCIENCE_SSH_CONFIG:+-F $OPENSCIENCE_SSH_CONFIG}"
```

That config makes your connection reuse the one the app already authenticated,
which is what lets clusters with two-factor authentication work at all: the user
signs in once, and every command after that needs no password or code. Without
it, such a host refuses every command you send.

### When a host asks for credentials

Many institutional clusters require a password or a one-time code on EVERY new
connection. If a command fails with a permission or authentication error
(`Permission denied`, `no supported authentication methods`), do NOT retry, and
do not report the failure yet: call the **`ssh_connect`** tool with that host.
It asks the desktop UI to prompt the user, and returns once the shared
connection is up — then retry your command unchanged.

Two errors that sign-in cannot fix, so do not call the tool for them: a host-key
error (`Host key verification failed` — the user must verify the fingerprint in
their own terminal once) and a network error (unreachable, timed out, unknown
host).

## 1 · Pick the machine

1. `cat .openscience/compute.json` in the workspace (the app keeps this file in
   sync from the user's settings — read it directly; the directory is hidden).
   It looks like:
   `{"machines":[{"host":"home-3090","label":"8x3090",
     "caps":{"cores":16,"mem_total_bytes":...,"gpus":["RTX 3090",...],"slurm":null}}]}`
   The directory is hidden — read the file directly.
2. If the file is missing or has no machines, ask the user to add one in
   **Settings → Remote compute**, or give you a `user@host`. Do not guess.
3. Choose by the task's needs and each machine's `caps`: a GPU job → a machine
   whose `caps.gpus` is non-empty; a CPU job → any reachable machine. If several
   fit, or none clearly does, ask the user which to use.
4. Confirm it's reachable and check live headroom before launching:
   `ssh $SSHCFG -o BatchMode=yes -o ConnectTimeout=8 <host> "nproc; free -h; nvidia-smi 2>/dev/null | head -15"`.
   On "Permission denied", the host wants credentials interactively: call the
   `ssh_connect` tool for it (see §0), then retry. Never send a password
   yourself and never disable host-key checking.

If the chosen machine's `caps.slurm` is set, use **§2-Slurm**. Otherwise use
**§2-Direct**.

## 2-Direct · Run on a plain server (no Slurm)

Long jobs must outlive the SSH connection. Use a per-job dir + a fully detached
process, mirroring how the app tracks runs.

1. Pick a job name and build the remote dir path (remember the literal string —
   shell variables do not survive between separate ssh calls):
   `REMOTE=openscience/jobs/<name>-<YYYYmmdd-HHMMSS>`
2. Create it and copy inputs (confirm with the user before copying > ~100 MB):
   ```bash
   ssh $SSHCFG -o BatchMode=yes <host> "mkdir -p <remote-dir>"
   scp $SSHCFG -o BatchMode=yes run.sh <input files> <host>:<remote-dir>/
   ```
   Write `run.sh` in the workspace first (so it is versioned in provenance);
   it should `cd` into the job dir and run the actual commands, e.g. use
   `CUDA_VISIBLE_DEVICES` to select GPUs. On a plain box the software
   environment is ambient (whatever is installed) — not declared anywhere — so
   pin it at run time by having `run.sh` write a manifest as its first step (it
   is fetched and recorded in §3–4). Keep it fail-safe so provenance never
   aborts the job:
   ```bash
   { python3 -V; echo "PLATFORM=$(uname -s)-$(uname -m)"; \
     echo '--- pip freeze ---'; python3 -m pip freeze; } > env.txt 2>&1 || true
   ```
3. Launch fully detached and capture the PID:
   ```bash
   ssh $SSHCFG -o BatchMode=yes <host> "cd <remote-dir> && \
     setsid bash -c 'bash run.sh >log 2>&1; echo \$? > exit_code' </dev/null >/dev/null 2>&1 & \
     echo \$! > pid; cat pid"
   ```
   Report the PID and the remote dir to the user.
4. **Track:**
   - Running? `ssh $SSHCFG <host> "kill -0 \$(cat <remote-dir>/pid) 2>/dev/null && echo RUNNING || echo DONE"`.
   - Progress: `ssh $SSHCFG <host> "tail -n 30 <remote-dir>/log"`; GPU use:
     `ssh $SSHCFG <host> "nvidia-smi"`.
   - Finished: `ssh $SSHCFG <host> "cat <remote-dir>/exit_code"` — `0` = success, other
     = failure. Do not assume success from an empty queue. When a run finishes
     you MUST complete §3 (fetch) **and** §4 (record) — every time, including a
     quick re-run or re-fetch. A run you don't record is invisible in Runs
     (neither the global view nor the session), so it may as well not exist.
   - Long jobs: report the PID + running state and stop; the user can ask you to
     check again later. Do not poll in a loop for more than ~2 minutes.
5. **Cancel** (only jobs you launched, or a PID/dir the user names): kill the
   whole process group so children die too:
   `ssh $SSHCFG <host> "kill -- -\$(cat <remote-dir>/pid) 2>/dev/null || kill \$(cat <remote-dir>/pid)"`.

## 2-Slurm · Run on a Slurm cluster

Use this only when `caps.slurm` is set.

1. Write `slurm/<job-name>.sbatch` in the workspace:
   ```bash
   #!/bin/bash
   #SBATCH --job-name=<job-name>
   #SBATCH --output=slurm-%j.out
   #SBATCH --error=slurm-%j.err
   #SBATCH --time=01:00:00

   set -euo pipefail
   cd "$SLURM_SUBMIT_DIR"
   <the actual commands>
   ```
   Only add `--partition/--gres/--mem/--cpus-per-task` when the user asks or the
   cluster rejects the default. Load modules (`module load …`) the user names.
2. Submit:
   ```bash
   REMOTE=openscience/jobs/<job-name>-$(date +%Y%m%d-%H%M%S)
   ssh $SSHCFG -o BatchMode=yes