async-jobs

# Async Jobs

Background task processing with Celery, ARQ, Redis and Temporal. This skill is a wrapper, not a
manual: Celery and ARQ document their own product well, so what lives here is our delta, the
thresholds, working config, ordering constraints and tool-choice rules we picked. Product
mechanics are linked, not restated.

Start with `Read("${CLAUDE_PLUGIN_ROOT}/skills/async-jobs/references/ork-delta.md")`.

## Quick Reference

| Topic | Where our part lives |
|-------|----------------------|
| [Configuration](#configuration) | `references/celery-config.md`, `rules/jobs-task-queue.md` |
| [Task Routing](#task-routing) | `references/ork-delta.md` (queue taxonomy, prefetch tiers, Redis priority) |
| [Canvas Workflows](#canvas-workflows) | `rules/celery-canvas.md` |
| [Retry Strategies](#retry-strategies) | `references/ork-delta.md` (backoff cap, idempotency layers, lock TTLs) |
| [Scheduling](#scheduling) | `rules/jobs-scheduling.md`, `references/ork-delta.md` (beat process model) |
| [Monitoring](#monitoring) | `references/ork-delta.md` (alert thresholds, histogram buckets) |
| [Result Backends](#result-backends) | `rules/jobs-monitoring.md`, `references/ork-delta.md` (return contract) |
| [ARQ Patterns](#arq-patterns) | `rules/jobs-task-queue.md`, `references/ork-delta.md` (budgets, pool ownership) |
| [Temporal Workflows](#temporal-workflows) | `rules/temporal-workflows.md` |
| [Temporal Activities](#temporal-activities) | `rules/temporal-activities.md` |

10 topic areas, 6 rule files in `rules/`, house delta in `references/ork-delta.md`.

## Quick Start

```python
@app.task(bind=True, max_retries=3, default_retry_delay=60)
def process_payment(self, order_id: str):
    try:
        return gateway.charge(order_id)
    except TransientError as exc:
        raise self.retry(exc=exc, countdown=2 ** self.request.retries * 60)
```

Load more examples: `Read("${CLAUDE_PLUGIN_ROOT}/skills/async-jobs/references/quick-start-examples.md")` for Celery
retry task and ARQ/FastAPI integration patterns.

## Upstream coverage (do not restate)

Fetch these when you need product mechanics. The right-hand column is the part we keep, because
it is a house threshold, a working config or an ordering constraint that upstream cannot know.

| Topic | First-party source | House subset stays in |
|-------|--------------------|-----------------------|
| Celery settings, serializers, time limits, worker flags | https://docs.celeryq.dev/en/stable/userguide/configuration.html and .../optimizing.html | `references/celery-config.md`, `rules/jobs-task-queue.md` |
| Queue declarations, router classes, Redis priority mechanics | https://docs.celeryq.dev/en/stable/userguide/routing.html | `references/ork-delta.md` |
| chain / group / chord / signature semantics | https://docs.celeryq.dev/en/stable/userguide/canvas.html | `rules/celery-canvas.md` keeps the house canvas subset. Its `si()`-in-chords guidance is UNVERIFIED and contested: confirm the argument-passing behaviour against the upstream canvas page before relying on it |
| `autoretry_for`, `retry_backoff`, `Reject`, task base classes | https://docs.celeryq.dev/en/stable/userguide/tasks.html | `references/ork-delta.md`, `rules/jobs-task-queue.md` |
| Beat schedules, crontab syntax, DatabaseScheduler | https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html and https://django-celery-beat.readthedocs.io/en/latest/ | `rules/jobs-scheduling.md` keeps our `beat_schedule` shapes; `references/ork-delta.md` keeps the process model |
| Flower flags, `inspect`, signal names | https://docs.celeryq.dev/en/stable/userguide/monitoring.html, https://docs.celeryq.dev/en/stable/userguide/signals.html, https://flower.readthedocs.io/en/latest/config.html | `references/ork-delta.md` |
| Result backend, `AsyncResult`, custom states | https://docs.celeryq.dev/en/stable/userguide/configuration.html | `rules/jobs-monitoring.md` keeps our status endpoints and `update_state()` usage |
| Per-task `rate_limit`, `control.rate_limit`, Redis Lua | https://docs.celeryq.dev/en/stable/userguide/workers.html, https://redis.io/docs/latest/develop/programmability/eval-intro/ | `references/ork-delta.md` |
| ARQ `WorkerSettings`, `enqueue_job`, `_defer_by` / `_defer_until`, `Job` status | https://arq-docs.helpmanual.io/ | `rules/jobs-task-queue.md` keeps the worker skeleton; `references/ork-delta.md` keeps the budgets |
| FastAPI lifespan and dependency wiring | https://fastapi.tiangolo.com/advanced/events/ | `references/ork-delta.md` |
| Distributed locks with `SET NX EX` | https://redis.io/docs/latest/commands/set/ | `references/ork-delta.md` |

## Configuration

Load: `Read("${CLAUDE_PLUGIN_ROOT}/skills/async-jobs/references/celery-config.md")`.

| Decision | Recommendation |
|----------|----------------|
| Serializer | JSON (never pickle) |
| Ack mode | Late ack (`task_acks_late=True`) |
| Prefetch | 1 for fair, 4-8 for throughput |
| Time limit | soft < hard (540 / 600) |
| Timezone | UTC always |

## Task Routing

| Decision | Recommendation |
|----------|----------------|
| Queue count | 5: critical / high / default / low / bulk |
| Priority levels | 0-9, with all four Redis priority switches set together |
| Worker assignment | Dedicated worker per queue |
| Prefetch | 1 critical, 2 high, 4 default, 8 low/bulk |
| Routing | Router class once past 5 routing rules |

## Canvas Workflows

Load: `Read("${CLAUDE_PLUGIN_ROOT}/skills/async-jobs/rules/celery-canvas.md")`.

| Decision | Recommendation |
|----------|----------------|
| Sequential | Chain with `s()` |
| Parallel | Group for independent tasks |
| Fan-in | Chord (all header tasks must succeed for the body to run) |
| Ignore input | Use `si()` immutable signature |
| Error in chain | `Reject` stops the chain, `retry` continues it |
| Partial failures | Return an error dict from chord header tasks |

## Retry Strategies

| Decision | Recommendation |
|----------|----------------|
| Retry delay | Exponential backoff, jitter on, cappe