Kafka MCP Server

A Model Context Protocol (MCP) server for Apache Kafka implemented in Go, leveraging franz-go and mcp-go.

This server provides an implementation for interacting with Kafka via the MCP protocol, enabling LLM models to perform common Kafka operations through a standardized interface.

Overview

The Kafka MCP Server bridges the gap between LLM models and Apache Kafka, allowing them to:

- Produce and consume messages from topics
- List, describe, and manage topics
- Monitor and manage consumer groups
- Assess cluster health and configuration
- Execute standard Kafka operations

All through the standardized Model Context Protocol (MCP).

Architecture

graph TB
    subgraph "MCP Client (AI Applications)"
        A[Claude Desktop]
        B[Cursor]
        C[Windsurf]
        D[ChatWise]
    end
    
    subgraph "Kafka MCP Server"
        E[MCP Protocol Handler]
        F[Tools Registry]
        G[Resources Registry]
        H[Prompts Registry]
        I[Kafka Client Wrapper]
    end
    
    subgraph "Apache Kafka Cluster"
        J[Broker 1]
        K[Broker 2]
        L[Broker 3]
        M[Topics & Partitions]
        N[Consumer Groups]
    end
    
    A --> E
    B --> E
    C --> E
    D --> E
    
    E --> F
    E --> G
    E --> H
    
    F --> I
    G --> I
    H --> I
    
    I --> J
    I --> K
    I --> L
    
    J --> M
    K --> M
    L --> M
    
    J --> N
    K --> N
    L --> N
    
    classDef client fill:#e1f5fe
    classDef mcp fill:#f3e5f5
    classDef kafka fill:#fff3e0
    
    class A,B,C,D client
    class E,F,G,H,I mcp
    class J,K,L,M,N kafka

How it works:
1. MCP Clients (AI applications) connect to the Kafka MCP Server via stdio or HTTP transport
2. MCP Server exposes three types of capabilities:
- Tools - Direct Kafka operations (produce/consume messages, describe topics, etc.)
- Resources - Cluster health reports and diagnostics
- Prompts - Pre-configured workflows for common operations
3. Kafka Client Wrapper handles all Kafka communication using the franz-go library
4. Apache Kafka Cluster processes the actual message streaming and storage

Transport Modes:
- STDIO: Default mode, ideal for local MCP clients (Claude Desktop, Cursor, etc.)
- HTTP: Enables remote access with optional OAuth 2.1 authentication

Key Features

- Kafka Integration: Implementation of common Kafka operations via MCP
- Security:
- Support for SASL (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512) and TLS authentication
- OAuth 2.1 authentication for HTTP transport (Native and Proxy modes)
- Support for Okta, Google, Azure AD, and HMAC providers
- Flexible Transport: STDIO for local clients, HTTP for remote access
- Error Handling: Error handling with meaningful feedback
- Configuration Options: Customizable for different environments
- Pre-Configured Prompts: Set of prompts for common Kafka operations
- Compatibility: Works with MCP-compatible LLM models

Getting Started

Prerequisites

- Go 1.24 or later
- Docker (for running integration tests)
- Access to a Kafka cluster

Installation

Homebrew (macOS and Linux)

The easiest way to install kafka-mcp-server is using Homebrew:

```bash