Agent observability in Grafana: the Hermes case

On Sunday 16 August, at 08:40 UTC, someone posted a repository to Show HN called grafana-agento11y-hermes: a Grafana based observability layer for the Hermes agent. As we write this it has one point and zero comments. There is no thread to pull on, no viral demo, no discussion. Just the code, at grafana-agento11y-hermes, and the Hacker News submission where it went unnoticed.

That is exactly why it caught our eye. Agent observability is one of those problems nobody treats as urgent until they have four agents in production, a token bill that does not add up and no way to tell which one went into a loop at three in the morning.

What is actually in the repository

It pays to be precise about what can be claimed. The project presents itself as Grafana based observability for one specific agent, Hermes, not as a standard or as a product with a roadmap. With no comment thread and no release notes, the honest reading is the plain one: someone instrumented their own agent, built the panels they needed and decided to publish them. That "here is my stack, take it if it helps" format has historically moved the ecosystem more than most corporate announcements.

Why measuring an agent matters

An agent does not break the way an HTTP service breaks. An endpoint returns 500 and you see it. An agent returns 200, takes forty seconds, calls the same tool eleven times, corrects itself, burns 180,000 tokens and delivers something that almost works. On a classic application dashboard, that is a successful request.

The signals that do help are different:

input and output tokens per run, not per day
number of loop iterations before finishing or giving up
latency and error rate broken down per tool, not aggregated
how many runs end by hitting the step limit instead of producing an answer
* cost per completed task, usually the only figure the person paying understands

OpenTelemetry has been working for a while on semantic conventions specific to GenAI systems (gen-ai semconv), and that is the boring, decisive part of all this. If telemetry comes out in a common format, the Grafana panel is almost a presentation detail. The real work is instrumenting the agent loop, not picking the chart colour.

How it maps onto the Claude stack

For anyone working with Claude Code, the pattern travels with little effort. Hooks (PreToolUse, PostToolUse, Stop) are natural attachment points: a shell command that writes one line per tool call, with name, duration and result, already gives you much of what a first dashboard needs. Subagents lend themselves to being modelled as nested spans, because every delegation has a start, an end and a cost of its own. And MCP servers are the most profitable boundary to measure: nearly all of an agent's odd failures show up when it crosses over to an external tool, not inside the model.

None of this forces you to adopt the Hermes repository. It forces you to accept that an agent in production is a distributed system with a non deterministic component, and to treat it accordingly.

Who it is useful for

If you are testing agents locally, you do not need this yet. If you have agents running unsupervised, on cron, in CI or in front of a client, you are already paying the cost of not measuring, only in the form of debugging hours and invoices explained after the fact. It also works as a shopping list: even if you never clone the repository, looking at the panels someone else built is the fast way to find out which metrics you are missing.

It is a small repository, with no traction and tied to one specific agent, and even so it points at the right place: agent tooling has run far ahead of agent instrumentation. At ElephantPink we would rather have an ugly dashboard that works today than wait for the definitive standard to show up.