k8s-troubleshoot-mcp

# k8s-troubleshoot-mcp

A read-only [MCP (Model Context Protocol)](https://modelcontextprotocol.io/)
server that lets an AI assistant diagnose a Kubernetes cluster. Ask why a pod is
crash-looping instead of running six `kubectl` commands and correlating the
output by hand.

**Read-only is a structural property, not a promise.** There are no write tools,
and the RBAC manifests grant no write verbs. See [Security model](#security-model).

## What It Does

The server exposes 16 diagnostic tools over stdio. Connect it to Claude Desktop,
VS Code, or Kiro, and you can ask things like:

- "Why is the `checkout` pod in `staging` not ready?"
- "Show me the last 50 lines of logs from the `api` container"
- "Which nodes are not Ready, and what are their taints?"
- "Is the `web` HPA scaling, and what does its current metric say?"
- "What events fired in `production` in the last few minutes?"

The assistant calls the tools, the server queries the Kubernetes API with a
scoped ServiceAccount token, and every response comes back as a structured dict —
including errors, which are never raised as exceptions into the MCP layer.

## Architecture

```
┌──────────────────┐    stdio (JSON-RPC)   ┌──────────────────────────┐
│   MCP Client     │◄─────────────────────►│   MCP Server             │
│  Claude Desktop  │                       │   (this project)         │
│  VS Code / Kiro  │                       │                          │
└──────────────────┘                       │  config.py   ─ validate  │
                                           │  server.py   ─ 16 tools  │
                                           │  tools/*.py  ─ read+shape│
                                           │  response.py ─ escape +  │
                                           │                structure │
                                           └───────────┬──────────────┘
                                                       │ HTTPS, explicit
                                                       │ KUBECONFIG only
                                           ┌───────────▼──────────────┐
                                           │  Kubernetes API server   │
                                           │  ── RBAC boundary ──     │
                                           │  ServiceAccount:         │
                                           │  get/list/watch only     │
                                           └──────────────────────────┘
```

Configuration is validated once at startup. If anything is wrong — `KUBECONFIG`
unset, the file unreadable or malformed, `ALLOWED_NAMESPACES` missing or
containing a wildcard — the process writes one line to stderr and exits 1. It
never starts in a partially-valid state.

## Available Tools

Arguments marked `?` are optional.

| Tool | Description | Parameters |
|------|-------------|------------|
| `get_pod_status` | Phase, conditions, container statuses, QoS class and node for a pod | `pod_name`, `namespace` |
| `get_pod_logs` | Recent log lines from a pod container. Content is untrusted — see [Reading `get_pod_logs` output](#reading-get_pod_logs-output) | `pod_name`, `namespace`, `container?`, `previous?`, `tail_lines?` |
| `get_pod_events` | Recent events for a pod, newest first, with `total_available` | `pod_name`, `namespace` |
| `list_pods` | Pods in a namespace with phase, restart count and readiness | `namespace`, `label_selector?` |
| `get_node_status` | Conditions, capacity, allocatable, taints and roles for a node | `node_name` |
| `list_nodes` | Cluster nodes with readiness, roles, age and kubelet version | none |
| `get_deployment_status` | Replica counts, conditions and rollout strategy | `deployment_name`, `namespace` |
| `list_deployments` | Deployments in a namespace with replica counts and availability | `namespace` |
| `get_statefulset_status` | Replica counts, revisions and update strategy | `statefulset_name`, `namespace` |
| `get_daemonset_status` | Scheduling counts and update strategy | `daemonset_name`, `namespace` |
| `get_service` | Type, ClusterIP, ports, selector and ready endpoint count | `service_name`, `namespace` |
| `get_endpoints` | Ready and not-ready endpoint addresses backing a service | `service_name`, `namespace` |
| `get_pvc_status` | Phase, capacity, binding and resize state for a PVC | `pvc_name`, `namespace` |
| `get_hpa_status` | Replica bounds, current metrics and conditions for an HPA | `hpa_name`, `namespace` |
| `get_namespace_events` | Recent events across a namespace, newest first, with `total_available` | `namespace`, `limit?` |
| `list_namespaces` | The namespaces this server is permitted to read | none |

Every namespaced tool validates its `namespace` argument **before** making any
API call, so a disallowed namespace produces a structured error and no network
request.

### Deliberately absent

No `get_secrets`, `get_configmap`, `exec_into_pod`, `port_forward`, or any
`create`/`update`/`patch`/`delete` tool. These are excluded from all versions
unless a new threat-model review is conducted and documented — they are not
backlog items. The reasoning for each is in
[SECURITY.md](SECURITY.md#what-this-server-is-not--deliberate-exclusions).

## Security model

Full detail is in [SECURITY.md](SECURITY.md). The summary:

### The boundary is Kubernetes RBAC

**Everything this server does in application code is defense-in-depth. The
enforcement boundary is the ServiceAccount's RBAC bindings.** If the bindings
grant more than intended, the application-layer allowlist is all that stands in
the way, and it is not a boundary you should rely on.

Provisioning is split by scope so that the cluster-scoped grant is minimal:

| Manifest | Scope | Grants |
|----------|-------|--------|
| `clusterrole.yaml` + `clusterrolebinding.yaml` | cluster | `get`/`list`/`watch` on `nodes` and `namespaces` only |
| `role.yaml` | namespace | `get`/`list`/`watch` on the diagnostic resources |
| `rolebinding.yaml.template` | namespace | binds the Role, one namespace at a time |

Applying the cluster-scoped pair makes **no namespace readable**. A namespace
becomes readable only when a Role *and* a RoleBinding exist in it. A namespace
listed in `ALLOWED_NAMESPACES` but never bound stays unreadable — RBAC wins.

`pods/log` is granted in its own rule block, never folded into the `pods` rule,
because Kubernetes subresources do not inherit from their parent.

### Defense-in-depth layers

| Layer | What it does | What it is not |
|-------|--------------|----------------|
| **RBAC** | Grants read verbs on diagnostic resources in bound namespaces only | — this *is* the boundary |
| **Explicit kubeconfig** | Reads `KUBECONFIG` from an exact path; no `~/.kube/config`, no in-cluster config, no fallback chain | Not a permission check — it prevents silently picking up an ambient credential |
| **Namespace allowlist** | Rejects wildcards, strips `kube-system`/`kube-public`, validates before every call | Advisory; a bug here is contained by RBAC |
| **Output escaping** | All cluster-authored free text routed through `serialize_log_content` | Prevents breaking out of a JSON string; cannot stop a model acting on legible instructions |
| **Structured errors** | Every failure returns a dict; no exception reaches the MCP layer | — |

### Prompt injection is mitigated, not eliminated

Pod logs and event messages are written by workloads in the cluster. A container
can print anything, including text shaped like instructions to the model reading
it. Escaping keeps injected text inside its JSON string; it cannot stop a model
from acting on instructions it reads as data. **Treat tool output as untrusted
input to whatever consumes it.** This residual risk is accepted and documented.

## Prerequisites

1. **A Kubernetes cluster** and a `kubectl` context with enough permission to
   create a ServiceAccount, Role, RoleBinding, ClusterRole and
   ClusterRoleBinding — you need this once, to provision. The server itself
   never uses your admin credential.
2. **Kubernetes 1.24+** — `scripts/generate-kubeconfig.sh` mints a token via the
   TokenRequest API, not a legacy auto-mounted Secret.
3. **Python 3.11+**
4. **uv** — `curl -LsSf https://astral.sh/uv/install.sh | sh` (or use Docker,
   which needs neither Python nor uv on the host)

## Setup

```bash
git clone https://github.com/NanaGyamfiPrempeh30/k8s-troubleshoot-mcp.git
cd k8s-troubleshoot-mcp

# Install dependencies (uv creates .venv automatically)
uv sync

# Run the test suite
uv run pytest tests/ -q
```

### Provision RBAC and mint a kubeconfig

```bash
scripts/generate-kubeconfig.sh /secure/path/k8s-mcp-kubeconfig.yaml staging production
```

The first argument is where to write the kubeconfig; the rest are the namespaces
the server may read. Pass the same set you intend to put in
`ALLOWED_NAMESPACES` — RBAC is the enforcement boundary, and a namespace bound
here but absent from the allowlist (or the reverse) is a mismatch between real
permission and configured capability.

The script applies the cluster-scoped manifests together, then applies
`role.yaml` with an explicit `-n <namespace>` and renders a RoleBinding per
namespace. On success it prints the kubeconfig path to stdout and nothing else;
all diagnostics go to stderr. It also asserts after provisioning that
`kubectl auth can-i get secrets` returns `no`, and aborts if it does not.

> **Do not run `kubectl apply -f kubernetes/`.** It does not fail — it reports
> success while creating `role.yaml` in the *current* namespace and skipping
> `rolebinding.yaml.template` entirely, because `kubectl apply -f <dir>` only
> reads `.yaml`/`.yml`/`.json`. The result is a server that looks provisioned
> and can read nothing. Verified against a v1.35 API server with
> `--dry-run=server`: 5 resources applied, not 6.

The generated kubeconfig is written with `umask 077` and `chmod 600`. Keep it
out of the repository — the script warns if the output path is inside a
repository and not covered by `.gitignore`.

### Run