using-webctl

# Using webctl in Claude.ai Containers

Browser automation using webctl CLI with automatic proxy authentication handling for Claude.ai's egress-controlled environment.

## When This Skill Applies

- User requests browser automation, web scraping, or page interaction
- webctl commands fail with `ERR_TUNNEL_CONNECTION_FAILED`
- Playwright/Chromium needs to work through authenticated HTTP proxy

## Core Problem

Claude.ai containers route traffic through an authenticated egress proxy (`HTTP_PROXY` env var with JWT credentials). Chromium doesn't properly handle proxy authentication for HTTPS CONNECT tunnels, causing all HTTPS navigation to fail even though curl works.

## Solution

A local forwarding proxy (port 18080) intercepts Chromium connections and injects `Proxy-Authorization` headers before forwarding to the real egress proxy.

## Setup Procedure

### 1. Install webctl

```bash
pip install webctl --break-system-packages
webctl setup  # Downloads Chromium if needed
```

### 2. Deploy Auth Proxy Module

Copy `scripts/auth_proxy.py` to webctl's daemon directory:

```bash
cp /mnt/skills/user/using-webctl/scripts/auth_proxy.py \
   /usr/local/lib/python3.12/dist-packages/webctl/daemon/
```

### 3. Patch Session Manager

Apply this patch to `/usr/local/lib/python3.12/dist-packages/webctl/daemon/session_manager.py`.

Recent webctl versions split context creation into two branches (mobile emulation vs desktop). Both must receive the proxy config. Find the block (search for `cfg = WebctlConfig.load()` followed by the `if mode == "unattended" and cfg.mobile_emulation:` branch — around line 190):

```python
cfg = WebctlConfig.load()
if mode == "unattended" and cfg.mobile_emulation:
    # Mobile emulation: cleaner pages, fewer ads, simpler layouts
    device = self._playwright.devices["Pixel 7"]
    context = await browser.new_context(
        storage_state=storage_state,
        **device,
    )
else:
    context = await browser.new_context(
        storage_state=storage_state, viewport={"width": 1280, "height": 720}
    )
```

Replace with:
```python
cfg = WebctlConfig.load()
from .auth_proxy import get_local_proxy_url
proxy_url = get_local_proxy_url()
proxy_config = {"server": proxy_url} if proxy_url else None
if mode == "unattended" and cfg.mobile_emulation:
    # Mobile emulation: cleaner pages, fewer ads, simpler layouts
    device = self._playwright.devices["Pixel 7"]
    context = await browser.new_context(
        storage_state=storage_state,
        proxy=proxy_config,
        **device,
    )
else:
    context = await browser.new_context(
        storage_state=storage_state, viewport={"width": 1280, "height": 720},
        proxy=proxy_config
    )
```

If the file only has a single `new_context` call (older webctl), inject `proxy=proxy_config` into that one and place the three `proxy_url` lines just above it.

### 4. Verify

```bash
webctl start --mode unattended
webctl --quiet navigate "https://github.com"
webctl snapshot --interactive-only --limit 10
webctl stop
```

## Quick Reference

### Session Management
```bash
webctl start --mode unattended    # Headless browser
webctl stop                       # Full shutdown (use --keep-daemon to leave daemon running)
webctl status                     # Current state + console error counts
```

### Navigation
```bash
webctl navigate "https://..."
webctl back / webctl forward / webctl reload
```

### Observation
```bash
webctl snapshot --interactive-only --limit 30   # Buttons, links, inputs
webctl snapshot --within "role=main"            # Scope to container
webctl query "role=button name~=Submit"         # Debug queries
webctl screenshot --path shot.png
```

### Interaction
```bash
webctl click 'role=button name~="Submit"'
webctl type 'role=textbox name~="Email"' "user@example.com"
webctl type 'role=textbox name~="Search"' "query" --submit
webctl select 'role=combobox name~="Country"' --label "Germany"
```

### Query Syntax
- `role=button` — By ARIA role
- `name~="partial"` — Contains (preferred, more robust)
- `name="exact"` — Exact match
- `nth=0` — Select first when multiple matches

### Wait Conditions
```bash
webctl wait network-idle
webctl wait 'exists:role=button name~="Continue"'
webctl wait 'url-contains:"/dashboard"'
```

## Troubleshooting

### ERR_TUNNEL_CONNECTION_FAILED
Auth proxy not loaded. Verify:
1. `auth_proxy.py` exists in webctl daemon directory
2. Session manager is patched
3. Restart daemon: `webctl stop && webctl start --mode unattended`

### Multiple matches error
Add specificity or use `nth=0`:
```bash
webctl click 'role=link name="Sign in" nth=0'
```

### Verify proxy is running
```bash
netstat -tlnp | grep 18080
```

## Limitations

- Patch is session-local (container resets clear it)
- Only allowed domains in network config are accessible
- No Firefox support (download blocked by egress policy)

## Domain Restrictions

Check `<network_configuration>` in system prompt for allowed domains. Common allowed: `*.github.com`, `*.bsky.app`, allowed API endpoints.

## Output Filtering

Reduce context consumption:
```bash
webctl snapshot --interactive-only --limit 30    # Cap elements
webctl snapshot | grep -i "submit"               # Unix filtering
webctl --quiet navigate "..."                    # Suppress events
```