docker-hybrids3

# HybridS3

Lightweight object storage that speaks S3 (boto3/AWS SDK compatible), plain HTTP, and MCP. SQLite for metadata, flat files on disk.

---

## Table of Contents

- [Why](#why)
- [Quick Start](#quick-start)
- [Running](#running)
- [Configuration](#configuration)
- [Keys and Access](#keys-and-access)
  - [Key pair model](#key-pair-model)
  - [Master key](#master-key)
  - [Bucket visibility](#bucket-visibility)
- [HTTP API](#http-api)
  - [Authentication](#authentication)
  - [Endpoints](#endpoints)
  - [Object keys](#object-keys)
  - [Response format](#response-format)
- [S3 Interface](#s3-interface)
  - [Authentication](#authentication-1)
  - [Operations](#operations)
- [MCP Interface](#mcp-interface)
  - [Connecting](#connecting)
  - [Authentication](#authentication-2)
  - [Tools](#tools)
- [Presigned URLs](#presigned-urls)
- [MIME Type Detection](#mime-type-detection)
- [TTL and Expiry](#ttl-and-expiry)
- [Locking](#locking)
- [Logging](#logging)
- [Behind a Reverse Proxy](#behind-a-reverse-proxy)
- [Security](#security)
- [Testing](#testing)
- [Development](#development)
- [Architecture](#architecture)
- [Limitations](#limitations)
- [License](#license)

---

## Why

**Most self-hosted S3-compatible storage is designed for large-scale deployments.** Distributed erasure coding, IAM policies, WORM compliance, full web consoles — useful if you're running a cloud, overkill if you just want a place to put files that various services and AI agents can read and write.

**AWS Sig V4 breaks behind reverse proxy path prefixes.** Most implementations verify signatures using the full original upstream path. Put them behind nginx at `/storage/`, nginx strips the prefix, the server sees `/bucket/key` instead of `/storage/bucket/key`, the signature check fails. HybridS3 has a `path_prefix` config option — set it to `/storage` and all routes move under that prefix. No path stripping, no special proxy headers. boto3's signed path matches what the server sees.

**Three interfaces, one service.** boto3 works out of the box. Plain HTTP with `curl` works. AI agents connect via MCP and get structured tool definitions. No separate services for different clients.

**Buckets are configuration, not state.** There is no API to create or delete buckets. They live in the YAML config file. You always know exactly what exists, it's version-controlled, and there are no surprise buckets accumulating garbage.

**TTL expiry is built in.** Set `ttl: 24h` on a bucket and objects expire automatically after their last write. No lifecycle policies, no cron jobs, no separate process.

**Readable and modifiable.** Small enough to understand in an afternoon.

---

## Quick Start

```bash
# get the example config
wget -O config.yaml https://raw.githubusercontent.com/psyb0t/docker-hybrids3/master/config.example.yaml

# edit it — set your keys and define your buckets
vi config.yaml

# run
docker run -d --name hybrids3 \
    -p 8080:8080 \
    -v ./config.yaml:/config/config.yaml:ro \
    -v hybrids3-data:/data \
    psyb0t/hybrids3

# verify
curl http://localhost:8080/health
```

---

## Running

The container expects:

- Config file at `/config/config.yaml`
- Data directory at `/data`
- Port `8080`

Runs as UID 1000.

**docker run:**

```bash
docker run -d --name hybrids3 \
    -p 8080:8080 \
    -v ./config.yaml:/config/config.yaml:ro \
    -v hybrids3-data:/data \
    psyb0t/hybrids3
```

**docker-compose:**

```yaml
services:
  hybrids3:
    image: psyb0t/hybrids3
    ports:
      - "8080:8080"
    volumes:
      - ./config.yaml:/config/config.yaml:ro
      - hybrids3-data:/data
    restart: unless-stopped

volumes:
  hybrids3-data:
```

---

## Configuration

```yaml
# Cross-bucket god key. Works on every bucket for every operation.
# Only credential that can list all buckets. Keep secret.
master_key: "change-me-to-something-secret"

# Non-secret identifier paired with master_key in S3 auth (aws_access_key_id).
# Safe to share — it grants nothing without the master_key.
master_public_key: "master"

# How often the background loop runs to delete expired objects and orphan files.
# Accepts human-readable durations: 30s, 5m, 1h, or raw seconds: 60
cleanup_interval: 1m

# How long a request may wait to acquire a lock on an object before giving up with 503.
# Accepts seconds as a float. Applies per object key, globally across all buckets.
lock_acquire_timeout: 30

# How long a request may hold a lock before it is forcibly released and the request gets 503.
# Protects against slow or stalled uploads blocking writes on the same key indefinitely.
lock_hold_timeout: 300

# Maximum number of requests that may queue for the same object key at once.
# Once the queue is full, new requests are rejected immediately with 503.
lock_max_waiters: 100

# Serve all routes under this path prefix.
# Set this when running behind a reverse proxy at a subpath (e.g. /storage).
# Accepts "/storage" or "/storage/" — both are normalized.
# Leave empty or omit to serve at the root.
# path_prefix: /storage

# TTL formats:   30s  5m  1h  1h30m12s  1d  2d12h  0 (never expire)
# Size formats:  500B  50KB  10MB  1GB  0 (no limit)

buckets:
  uploads:
    # true  — anyone can read (GET/HEAD/LIST) without authentication
    # false — authentication required for all operations
    public: true

    # Private key for this bucket.
    # Used as the Bearer token value and as aws_secret_access_key in S3 auth.
    # Never transmitted — only used to sign or verify request signatures.
    # Keep secret.
    key: "uploads-secret"

    # Public identifier for this bucket in S3 auth (aws_access_key_id).
    # Also appears in presigned URL Credential= fields.
    # Safe to share — it identifies the bucket but grants no access.
    # Defaults to the bucket name if not set.
    public_key: "uploads-id"

    # Objects expire this long after their last write. Overwriting resets the clock.
    # 0 means objects never expire.
    ttl: 24h

    # Maximum upload size. Requests over this limit are rejected with 413.
    # 0 means no limit.
    max_file_size: 50MB

  permanent:
    public: false
    key: "perm-secret"
    public_key: "permanent-id"
    ttl: 0
    max_file_size: 100MB
```

---

## Keys and Access

### Key pair model

Each bucket has two keys defined in config:

| Config field | Role                                                                                                                                               | Keep secret?       |
| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ |
| `key`        | The private key. Used to authenticate Bearer requests and to sign S3 signatures. Never transmitted — only used locally to compute or verify HMACs. | Yes                |
| `public_key` | The public identifier. Used as `aws_access_key_id` in S3 auth and appears in presigned URL `Credential=` fields. Grants nothing on its own.        | No — safe to share |

The split is what makes presigned URLs work safely. A presigned URL must embed an identifier in the `Credential=` field so the server knows which key to verify against — that identifier is the `public_key`. Since it is non-secret, having it in the URL is fine. The private key signs the URL on the server and never appears in it.

### Master key

The `master_key` is a cross-bucket credential that works on every bucket for every operation, without needing individual bucket keys. Two situations call for it:

- You need to operate across multiple buckets from a single client
- You need to list all configured buckets — bucket keys only see their own bucket in `list_buckets`

The `master_public_key` is the non-secret identifier that pairs with `master_key` in S3 auth (used as `aws_access_key_id`).

Do not embed the master key in client-facing code. Use per-bucket keys for that — they limit access to exactly one bucket.

### Bucket visibility

| Setting         | GET / HEAD / LIST                              | PUT                                              | DELETE / presign         |
| --------------- | ---------------------------------------------- | ------------------------------------------------ | ------------------------ |
| `public: true`  | no authentication required                     | bucket key, master key, or valid presigned PUT   | bucket key or master key |
| `public: false` | bucket key, master key, or valid presigned GET | bucket key, master key, or valid presigned PUT   | bucket key or master key |

---

## HTTP API

### Authentication

HTTP requests authenticate using a Bearer token in the `Authorization` header. Pass the bucket's private `key`, or the master key for cross-bucket operations.

```
Authorization: Bearer <private_key>
```

- **Public bucket reads** (GET, HEAD, LIST) — no authentication required.
- **All writes** (PUT, DELETE) and **reads on private buckets** — require the bucket key or master key.
- **Listing buckets** (`GET /`) — master key lists all buckets; bucket key lists only its own bucket.
- **Presign endpoint** (`POST /presign/...`) — requires the bucket key or master key.

```bash
# write to a bucket
curl -X PUT http://localhost:8080/uploads/file.txt \
  -H "Authorization: Bearer uploads-secret" \
  -d "hello"

# read from a public bucket — no auth needed
curl http://localhost:8080/uploads/file.txt

# read from a private bucket
curl http://localhost:8080/permanent/doc.pdf \
  -H "Authorization: Bearer perm-secret"

# list all buckets — master key sees all
curl http://localhost:8080/ \
  -H "Authorization: Bearer your-master-key"

# list buckets — bucket key sees only its own bucket
curl http://localhost:8080/ \
  -H "Authorization: Bearer uploads-secret"
```

---

### Endpoints

| Method   | Path                      | Auth                 | Description                                                        |
| -------- | ------------------------- | -