MCP Prometheus

MCP Prometheus exposes 18 read-only MCP tools that wrap the Prometheus HTTP API: instant and range PromQL queries, metric/label/series discovery, target and runtime information, TSDB stats, alerting rules, and exemplars.

When deployed with OAuth enabled it acts as a fullOAuth 2.1 Authorization Server(backed by Dex/OIDC), so MCP clients authenticate with the server before any tool call. The server then resolves the authenticated user's Mimir tenant IDs and enforces them on every query.

MCP Client (Claude, muster, …) │ OAuth 2.1 + MCP over HTTP ▼ ┌──────────────────────────────────┐ │ mcp-prometheus │ │ │ │ ┌────────────┐ ┌─────────────┐ │ │ │ OAuth 2.1 │ │ MCP Tools │ │ │ │ server │ │ (PromQL, │ │ │ │ (mcp-oauth)│ │ labels, …)│ │ │ └─────┬──────┘ └──────┬──────┘ │ │ │ │ │ │ ┌─────▼──────┐ ┌──────▼──────┐ │ │ │ Dex OIDC │ │ Tenancy │ │ │ │ provider │ │ resolver │ │ │ └────────────┘ └──────┬──────┘ │ └─────────────────────────┼────────┘ │ ┌───────────────┴──────────────┐ │ │ ┌──────▼───────┐ ┌───────▼──────┐ │ GrafanaOrg │ │ Prometheus │ │ CRDs (k8s) │ │ / Mimir │ └──────────────┘ └──────────────┘

- :8080— MCP + OAuth endpoints (served to clients)
- :9091— observability:/metrics,/healthz,/readyz(internal only)

git clone https://github.com/giantswarm/mcp-prometheus.git cd mcp-prometheus go build -o mcp-prometheus ./...

All configuration is via environment variables.

Start the server withserve --transport <mode>:

# Local stdio (no OAuth) ./mcp-prometheus serve # In-cluster HTTP with OAuth ./mcp-prometheus serve --transport streamable-http --http-addr :8080 --enable-oauth

MCP Client mcp-prometheus Dex OIDC │ │ │ │ 1. GET /.well-known/… │ │ │──────────────────────────>│ │ │ server metadata │ │ │<──────────────────────────│ │ │ │ │ │ 2. POST /oauth/register │ │ │──────────────────────────>│ │ │ client_id + secret │ │ │<──────────────────────────│ │ │ │ │ │ 3. GET /oauth/authorize │ │ │──────────────────────────>│ │ │ │ 4. redirect to Dex │ │ │───────────────────────>│ │<──────────────────────────────────────────────────│ │ browser: Dex login UI │ │ │ (user authenticates) │ │ │ │ │ │ │ 5. callback + code │ │ │<───────────────────────│ │ │ │ │ 6. POST /oauth/token │ │ │──────────────────────────>│ │ │ access_token + refresh │ │ │<──────────────────────────│ │ │ │ │ │ 7. MCP tool call │ │ │ Authorization: Bearer … │ │ │──────────────────────────>│ │ │ │ 8. validate token │ │ │ resolve tenants │ │ │ forward to Mimir │

The access token is a short-lived JWT signed by mcp-prometheus and validated on every request. Refresh token rotation is enabled — every refresh issues a new refresh token.

- memory(default): in-process, lost on restart. Suitable for single-replica deployments and development.
- valkey: production-grade Redis/Valkey backend. Required for multi-replica deployments.

WhenDEX_ISSUER_URLuses an internal DNS name that resolves to a private IP (RFC-1918 range), the built-in SSRF protection in the OIDC discovery client would reject the connection.

SetMCP_OAUTH_ALLOW_PRIVATE_URLS=trueto inject an HTTP client that allows private-IP connections for OIDC discovery. TLS verification is still enforced.

MCP_OAUTH_ALLOW_PRIVATE_URLS=true DEX_ISSUER_URL=https://dex.mc.my-cluster.example.io # resolves to 10.x.x.x

In Helm:app.oauth.allowPrivateURLs: true

When Dex is served with a certificate from a private/internal CA (e.g. private management clusters), TLS verification fails withx509: certificate signed by unknown authority. PointDEX_CA_FILEat a PEM CA file to add that CA on top of the system trust store. The pool verifies the Dex provider connection (OIDC discovery, code flow, userinfo), the forwarded-ID-token JWKS endpoint, and trusted-issuer JWKS endpoints.

DEX_CA_FILE=/etc/ssl/certs/dex-ca/ca.crt

In Helm, reference a Secret holding the CA certificate:

app: oauth: dexCASecret: name: mcp-prometheus-dex-ca key: ca.crt

ConfigureOAUTH_TRUSTED_AUDIENCESwith a comma-separated list of the aggregator's OAuth client IDs:

OAUTH_TRUSTED_AUDIENCES=muster-client,my-aggregator

- Detect that the incoming token's audience matches a trusted client ID
- Verify the token signature against Dex's JWKS endpoint
- Accept the token and proceed with tenant resolution

Tokensmuststill originate from the configured Dex issuer.

When OAuth is enabled, every tool call is scoped to the authenticated user's allowed Mimir tenant IDs. The user can pass an explicitorg_idparameter; the server validates it against their allowed tenants. If noorg_idis given, all allowed tenants are injected as a Mimir pipe-separated multi-tenant selector.

Two resolution modes are available, selected with--tenancy-mode(orapp.tenancy.modein Helm).

ReadsGrafanaOrganizationcustom resources from the Kubernetes API. Each CR declares which Dex groups have access (spec.rbac) and which Mimir tenant IDs map to it (spec.tenants).

apiVersion: observability.giantswarm.io/v1alpha1 kind: GrafanaOrganization metadata: name: team-platform spec: rbac: - groupName: github-org:team-platform # Dex group from LDAP/GitHub tenants: - prod-eu-west - prod-us-east

- The server reads thegroupsclaim from their Dex token
- It looks up allGrafanaOrganizationCRs where any group inspec.rbacmatches
- It collects all tenant IDs fromspec.tenantsacross matching CRs
- Results are cached per group-set for 60 seconds

The Helm chart creates aClusterRole+ClusterRoleBindinggrantingget, list, watchongrafanaorganizations.observability.giantswarm.iowhen this mode is active.

No Kubernetes API access required. Tenants are configured statically.

# All authenticated users get these tenant IDs --static-tenants=prod-eu,prod-us

app: tenancy: mode: static static: tenants: "prod-eu,prod-us"

Group mapping: per-group tenant assignment

WhenTENANCY_STATIC_GROUP_MAPis set (orapp.tenancy.static.groupsin Helm), tenant IDs are resolved per group: