fanout

# Fanout

**Single-binary, agent-native OpenTelemetry investigation.**

[![CI](https://github.com/labstack/fanout/actions/workflows/ci.yml/badge.svg)](https://github.com/labstack/fanout/actions/workflows/ci.yml)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![Go Reference](https://pkg.go.dev/badge/github.com/labstack/fanout.svg)](https://pkg.go.dev/github.com/labstack/fanout)

Fanout ingests OpenTelemetry data, durably publishes it as atomic Parquet
batches, and puts an AI agent in front of it — in one Go process with no
external dependencies to operate. Persistent trace indexes serve targeted
reads; embedded DuckDB handles SQL, broad scans, and rebuildable rollups. Point
an SDK or Collector at it, open the browser, and ask questions about your
telemetry in plain language.

There is no separate ingester, query service, metadata database, object store,
or dashboard server to deploy. One binary, one data directory.

## Architecture

One process owns ingest, storage, query, alerting, the agent runtime, and an
MCP server. Everything below the dashed boundary is compiled into a single
executable, including the React client.

![Fanout architecture](docs/diagrams/architecture.svg)

Telemetry lands over OTLP/gRPC or OTLP/HTTP. Concurrent small requests may
share a group-commit batch, while up to four workers independently encode and
durably publish atomic Parquet directories with persistent trace indexes.
Targeted trace reads go through those indexes; DuckDB scans the same Parquet
for SQL and maintains rebuildable service, endpoint, and edge rollups. The
browser client, an in-process agent, and any
external MCP host all reach the same typed observability contract rather than
issuing raw SQL.

Parquet is authoritative telemetry, DuckDB query state is rebuildable, and
SQLite is reserved for transactional product state. Native compaction
prepares replacements while reads continue and briefly gates readers only for
the crash-safe namespace swap:

![Fanout persistence](docs/diagrams/persistence.svg)

Application state (users, sessions, dashboards, alert rules, agent threads)
lives in the control SQLite database and never sits on the telemetry write
path. The published Parquet directories are self-describing, so startup can
discover the authoritative batch set directly from the filesystem.

## Performance

The independent [Fanout Bench](https://github.com/labstack/fanout-bench)
project measures authenticated ingest and optional dashboard read load against
your hardware. It uses the official OpenTelemetry generator and publishes raw,
reproducible evidence separately from the production binary. Ingest, indexed
reads, DuckDB analytics, and native Parquet maintenance have separate
coordination paths but still compete for the same CPU, memory bandwidth,
filesystem cache, and disk.

The current publication candidate is **296,196 accepted OpenTelemetry items per
second** sustained for five minutes with traces, logs, and metrics arriving
together on a machine with eight logical CPUs and 15.6 GiB of memory. It is a
single run, and the benchmark harness that produced it carried uncommitted
local changes, so it is not Fanout's official headline yet. The [performance methodology](https://fanout.run/explanation/performance)
shows the signal breakdown, quality gates, limitations, and publication bar.

## How it compares

Fanout is a single node holding traces, logs, and metrics for a system you can
reason about from one place. That premise, rather than any single feature, is
what separates it from its neighbours.

| If you use | Where Fanout differs |
| --- | --- |
| **Grafana with Loki, Tempo, and Mimir** | That stack keeps a service and a query language per signal, plus object storage underneath. Fanout keeps one process, one data directory, and one typed contract across all three signals, at the cost of the horizontal scale those components are built for. |
| **SigNoz** | Both are OTLP-native and self-hosted. SigNoz composes a collector, ClickHouse, and query services; Fanout compiles ingest, authoritative Parquet storage, indexed trace reads, DuckDB analytics, alerting, and the browser client into one binary. |
| **Jaeger** | Jaeger covers traces and expects a storage backend you run separately. Fanout ingests traces, logs, and metrics into the same store, with nothing else to deploy. |
| **Prometheus with Grafana** | Prometheus pulls metrics and is excellent at them. Fanout accepts pushed OTLP for all three signals and is built around investigating a specific incident rather than maintaining long-range metric series. |
| **Datadog**, **Honeycomb**, **Grafana Cloud** | Those are managed services: someone else runs the storage, the scaling, and the upgrades, and your telemetry leaves your network to get there. Fanout is a binary you run, on data that stays on your disk. |
| **An OpenTelemetry Collector piped into ClickHouse** | The same shape, assembled by hand: collector, database, dashboards, and the glue between them. Fanout is that assembly as one program, with an agent and an MCP server already wired to the same query contract. |

Fanout is a single node. It has no clustering, no replication, and no object
tier; a deployment that outgrows one machine's disk and CPU has outgrown
Fanout.

## Requirements

- **Go and a C compiler** with `CGO_ENABLED=1` — DuckDB is a cgo dependency
- **[Bun](https://bun.sh)** — compiles the browser assets
- **[just](https://just.systems)** — task runner
- A 32-character **authentication code secret**

SMTP and an AI provider are optional. Without SMTP, an operator can mint a
short-lived login link from the local Fanout binary. Without an AI key, ingest,
dashboards, traces, logs, metrics, and MCP continue to work; only investigation
chat and AI-assisted controls are hidden.

## Quick start

### Native binary

Release archives support Linux and macOS on amd64 and arm64. The installer
verifies the selected archive against the release checksum before extracting:

```sh
curl -fsSL https://raw.githubusercontent.com/labstack/fanout/main/scripts/install.sh | sh
```

Set `FANOUT_VERSION=v{YYYY.M}.{N}` to pin a release and `FANOUT_PREFIX` to
choose the installation directory.

### Docker

```sh
docker run --name fanout -p 7520:7520 \
  -v fanout-data:/var/lib/fanout/data \
  -e FANOUT_AUTH_CODE_SECRET=$(openssl rand -hex 32) \
  labstack/fanout:latest
```

Open the one-time setup URL printed by the container and create the first
administrator. Fanout displays the ingest token exactly once; save it with
your collector secrets. A standard OTLP/HTTP exporter can then use:

```sh
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:7520
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer%20$INGEST_TOKEN"
```

For a Collector on the same private container network, the forwarding side is:

```yaml
exporters:
  otlp_http/fanout:
    endpoint: http://fanout:7520
    headers:
      Authorization: "Bearer ${env:INGEST_TOKEN}"

service:
  pipelines:
    traces:  { receivers: [otlp], exporters: [otlp_http/fanout] }
    metrics: { receivers: [otlp], exporters: [otlp_http/fanout] }
    logs:    { receivers: [otlp], exporters: [otlp_http/fanout] }
```

This assumes the Collector's existing `otlp` receiver and an environment
variable containing the one-time token. Replace `fanout` with the private
hostname reachable from that Collector.

The equivalent OTLP/gRPC endpoint is `localhost:7520` (plaintext locally).
Behind a TLS proxy, both transports use the public origin; gRPC requires
HTTP/2 to Fanout. See the [single-port migration notes](docs/operations.md#single-port-migration).
For later sign-in
without SMTP, mint a 15-minute, single-use link against the running
container's control database:

```sh
docker exec fanout fanout --config /etc/fanout/fanout.yaml \
  login-link admin@example.com
```

Add `FANOUT_AI_API_KEY` to enable chat. Configure all four SMTP settings
(`FANOUT_SMTP_HOST`, `FANOUT_SMTP_USERNAME`, `FANOUT_SMTP_PASSWORD`, and
`FANOUT_SMTP_FROM`) to enable email-code login.

The distroless image runs unprivileged as UID 65532. A bind-mounted host directory at
`/var/lib/fanout/data` must be writable by that user; a named volume, as above,
needs no such handling.

The image selects `/etc/fanout/fanout.yaml` by default. That file contains only
the container listener and data-directory defaults; it does not contain
credentials. Start a container-specific document from that file so it retains
the shared listener and persistent data-directory settings:

```sh
cp fanout.docker.yaml fanout.yaml
# Add the remaining settings, then mount it over the image document:
docker run -v ./fanout.yaml:/etc/fanout/fanout.yaml:ro \
  labstack/fanout:latest
```

All surfaces use `server.addr` (`:7520` by default). Publish only that
port; OTLP/HTTP and OTLP/gRPC use the same address as the application.

### From source

```sh
git clone https://github.com/labstack/fanout.git
cd fanout
just install   # browser dependencies and git hooks
just build     # browser assets, then the binaries
```

Run it with the minimum configuration:

```sh
export FANOUT_AUTH_CODE_SECRET=$(openssl rand -hex 32) # must be 32+ characters

./bin/fanout
```

Optionally set `FANOUT_AI_API_KEY` for chat and the SMTP variables shown above
for email-code login. Without SMTP, run `./bin/fanout login-link
admin@example.com` from the same configuration and data directory.

Fanout serves the UI, API, MCP, OTLP/gRPC and OTLP/HTTP on
<http://localhost:7520>. The first account
created becomes the administrator and receives the ingest token once.

Point any OpenTelemetry collector or SDK at either OTLP endpoint with the
ingest token. Use the separate
[Fanout Bench](https://github.com/labstack/fanout-bench) project for controlled
capacity tests.

## Configuration

Configuration is resolved once at startup and validated before Fanout opens
data files or listeners. Sources apply i