setup-osworld
Provision and verify an OSWorld-V2 checkout after clone. Use when the user asks for OSWorld-V2 setup, installation, onboarding, AWS provider setup, Docker provider setup, mocked website server setup, GitLab server setup, gated task download, CUA-Harness hybrid experiment setup, or a final runnable export block. The skill should install/configure the selected supported infrastructure where possible, ask for user confirmation or credentials when required, and report what is fully configured versus still blocked.
git clone --depth 1 https://github.com/AMAP-ML/LongHorizon-Harness /tmp/setup-osworld && cp -r /tmp/setup-osworld/eval/OSWorldv2-harness/OSWorld-V2/.codex/skills/setup-osworld ~/.claude/skills/setup-osworldSKILL.md
# Setup OSWorld
Use this skill to make a cloned OSWorld-V2 checkout runnable, not merely to
list environment variables. Provision every selected component that can be
configured safely from the current machine, and ask the user whenever an action
requires credentials, paid resources, DNS changes, SSH keys, secrets, or a
destructive/cloud operation.
## Supported Scope
Supported provider setup:
- `aws`: fully provision required AWS network resources with AWS CLI, then
export the values OSWorld needs.
- `docker`: install/verify Docker and KVM where possible, then verify the
Docker provider can run.
Unsupported provider setup for now:
- `vmware`, `virtualbox`, `azure`, `gcp`, `aliyun`, and `volcengine`.
For unsupported providers, load `references/unsupported-providers.md`, state
that this skill does not automate them yet, and ask whether the user wants AWS
or Docker instead.
Optional service setup:
- OSWorld-web mocked website server, using `Task-Web/OSWorld-web`.
- GitLab server, using `Task-Web/gitlab`.
- Gated task class download from Hugging Face.
- Task-scoped proxy setup following upstream OSWorld section 2.3, plus optional
host-side proxy or HF mirror setup.
- CUA-Harness hybrid experiment overlay when this checkout includes
`experiments/osworld_v2_hybrid` and the sibling `cua-harness/` source tree.
## Start With Intake
Ask before doing infrastructure work unless the user has already supplied the
answers.
Ask for:
- provider: `aws` or `docker`
- whether to use existing AWS resources or create new ones
- AWS region, VPC/subnet preference, and whether AWS charges are acceptable
- whether to set up OSWorld-web, and on existing server or new AWS EC2 host
- website domain mode: existing wildcard domain, `nip.io`, or user-provided
host suffix
- whether to set up GitLab, and whether it must be a separate server/domain
- whether to download gated task classes, latest or a benchmark release
- whether task-scoped proxy, host-side proxy, or HF mirror is needed
- whether to configure the CUA-Harness hybrid experiment overlay
- whether to run real smoke tests that create/start resources
If the user says "do it unattended", still stop for any cloud spend, DNS,
GitHub private-repo authorization, generated secret disclosure, or destructive
operation.
## Workflow
1. Read `README.md`, `docs/PROVIDER_SETUP.md`, and the selected references.
2. Load `references/common.md` and complete package/tool setup.
3. Load exactly one provider reference:
- `references/provider-aws.md`
- `references/provider-docker.md`
- `references/unsupported-providers.md`
4. For OSWorld-web, load `references/website.md`.
5. For GitLab, load `references/gitlab.md`.
6. For gated task classes, load `references/tasks.md`.
7. For proxy needs, load `references/proxy.md`.
8. For CUA-Harness hybrid experiments, load `references/cua-harness.md`.
9. Load `references/verify.md` and run verification for each selected surface.
10. Write final setup outputs:
- copy-pasteable `export ...` block for non-secret values
- secret exports in a chmod 600 local file unless the user explicitly asks
to print them
- status report for configured, partially configured, skipped, and blocked
items
## Operating Rules
- Use `uv`, `uvx`, and `uv run python`; do not rely on bare `python` or `pip`.
- Prefer existing repo docs and current provider code over memory.
- Do not stop at "set this variable"; if the user selected a supported setup,
create or verify the backing resource that makes the variable valid.
- Use AWS CLI for AWS resources. Do not hand-wave AWS console steps unless the
user chose manual setup.
- Use Docker Compose for OSWorld-web and GitLab exactly as their READMEs
describe, with `HOST_SUFFIX`, `GITLAB_URL`, and `GITLAB_PRIVATE_TOKEN`
backed by reachable services.
- Keep model/API-key setup separate from provider/runtime setup unless the user
asks to run an agent or evaluator.
- Never commit secrets, `.env` files, generated private keys, or setup export
files.
- If an external private repository is inaccessible, report the exact blocker
and ask the user to authorize GitHub access instead of inventing commands.
- At the end, be explicit: "configured and verified", "configured but not
smoke-tested", "not configured", or "blocked awaiting user action".Check OSWorld tasks. Validate the evaluation function, verify that the instruction is feasible given the task setup and agent-visible files, inspect setup artifacts when needed, and produce both markdown and structured JSON reports.
Analyze OSWorld-V2 agent trajectory logs and task results to produce actionable insights. Use this skill whenever the user wants to understand agent performance on OSWorld tasks — including analyzing trajectories, reviewing task results, finding error patterns, comparing code vs GUI strategies, identifying which tools/commands the agent used, or deciding which task types to scale up in the benchmark.
Migrate an agent from upstream OSWorld into this OSWorld-V2 repository, add matching evaluation entrypoints, and verify the integration.
Reproduce CUA-Harness experiments on WeaveBench from a GitHub checkout. Use when the user wants an AI coding agent to set up dependencies, download WeaveBench assets, prepare the 120G VM, configure Qwen/Anthropic-compatible APIs, run smoke tests, launch full or subset evaluations, inspect logs, or summarize scores for this repository.