The agent didn't make a mistake in reasoning. It made a mistake in vocabulary. It had no way to know that the field was called total_amount and not total_price, because nothing told it the schema. The agent was schema-blind.

This is the default state of every AI agent interacting with Kafka today.

What schema blindness actually costs you​

Schema blindness manifests in three ways, and all of them are worse than they sound.

1. Agents hallucinate field names. Large language models are autocomplete machines. When an agent needs to produce a message to a topic called orders.created, it guesses what the fields should be. Sometimes it guesses correctly. Sometimes it produces order_id when the schema says orderId. Without the actual schema definition, the agent is operating on vibes.

2. Agents can't validate what they read. When an agent consumes from a Kafka topic, it receives raw bytes. If the topic uses Avro, those bytes are Avro-encoded. If the agent doesn't know the schema, it can't decode the message. Most ad-hoc integrations skip Avro entirely and configure the consumer to treat everything as JSON — which works until someone changes the serialization format.

3. Schema changes break agents silently. Schema evolution is the norm in production Kafka. Fields get added, defaults change, optional fields become required. If your agent doesn't know the registry exists, it can't detect that a schema changed, can't verify backward compatibility, and can't adapt its behavior. The agent is the last to know.

The concrete damage​

Consider a data quality agent that monitors a user_events topic. Its job: consume recent messages, validate field presence, and alert on anomalies.

Without schema awareness, the agent is checking fields by guessing. If the team adds a new required field — say, session_id — the agent doesn't know. It continues reporting everything is fine. Meanwhile, 5% of producers haven't upgraded yet and are sending messages without session_id, which fail downstream.

With schema awareness, the agent reads the schema from the Schema Registry, sees that session_id was added in v3 with no default value, and immediately flags: "New required field session_id added to user_events schema v3. Checking producer compliance."

Same agent. Same Kafka topic. Radically different outcomes based entirely on whether the agent knows the schema.

How KafkaMCP solves this​

KafkaMCP integrates schema awareness into every relevant operation, not as an add-on.

Schema discovery. kafka_list_schemas returns all Schema Registry subjects with their latest version, type (AVRO/PROTOBUF/JSON), and compatibility level. kafka_get_schema returns the full schema definition with a generated field summary:

{
  "subject": "orders.created-value",
  "version": 3,
  "schema_id": 42,
  "schema_type": "AVRO",
  "compatibility_level": "BACKWARD",
  "field_summary": "OrderCreated { order_id: string, customer_id: string, total_amount: double, currency: string (default: USD), items: array<OrderItem>, created_at: timestamp-millis }"
}

That field_summary field is key. It gives the agent a single-line mental model of what the message looks like. The agent doesn't need to parse nested Avro JSON — it reads a type signature.

Compatibility checking. Before a team adopts a proposed producer schema, an agent can call kafka_check_schema_compatibility against the registry's actual policy.

Schema evolution workflow. The agent can retrieve the current definition, construct a concrete candidate schema, and check that proposal before registration.

The difference in practice​

Without (schema-blind):

1. Agent samples a DLQ payload and sees plausible field names
2. It guesses that the payload is valid
3. The mismatch with the registered schema is missed
4. A human later compares the payload and schema manually

With KafkaMCP (schema-aware):

1. Agent calls kafka_analyze_dlq and kafka_get_schema
2. It compares the sampled payload with the field summary
3. It identifies the likely producer/schema mismatch
4. It reports concrete evidence and a remediation path

Schema blindness is the difference between agents that work in staging and agents that survive production.

KafkaMCP exposes schema evidence; it does not silently validate or wire-serialize every produced payload.

KafkaMCP is open source under the Apache 2.0 license. GitHub · Docs · Discussions

Here's what we're building:

Claude Desktop ←→ MCP (stdio) ←→ KafkaMCP ←→ Your Kafka Cluster

Prerequisites​

You need two things:

• A running Kafka cluster (local localhost:9092 is fine)
• Go 1.22+ installed (for go install)

If you're running Kafka locally and don't have one handy, the fastest path:

docker run -d --name kafka -p 9092:9092 \
  -e KAFKA_CFG_NODE_ID=0 \
  -e KAFKA_CFG_PROCESS_ROLES=controller,broker \
  -e KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@localhost:9093 \
  -e KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \
  -e KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
  -e KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \
  -e KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER \
  bitnami/kafka:3.7

Minute 1: Install KafkaMCP​

go install github.com/josedab/kafkamcp/cmd/kafkamcp@latest

Verify it's working:

kafkamcp version
# kafkamcp 0.1.0

That's it. Single binary, no dependencies.

Minute 2: Write the config​

Create a file called kafkamcp.yaml in a location you'll remember (I use ~/.config/kafkamcp/kafkamcp.yaml):

server:
  transport: stdio
  log_level: info

clusters:
  - name: local
    bootstrap_servers: "localhost:9092"
    security_protocol: PLAINTEXT
    default: true

audit:
  enabled: true
  max_entries: 10000

Eight lines of meaningful config. That's the entire setup.

Validate it:

kafkamcp validate --config ~/.config/kafkamcp/kafkamcp.yaml
# Configuration is valid.

Using Confluent Cloud?​

Replace the cluster block:

clusters:
  - name: confluent
    bootstrap_servers: "${CONFLUENT_BOOTSTRAP_SERVERS}"
    security_protocol: SASL_SSL
    sasl_mechanism: PLAIN
    sasl_username: "${CONFLUENT_API_KEY}"
    sasl_password: "${CONFLUENT_API_SECRET}"
    schema_registry:
      url: "${CONFLUENT_SR_URL}"
      auth:
        username: "${CONFLUENT_SR_API_KEY}"
        password: "${CONFLUENT_SR_API_SECRET}"
    default: true

KafkaMCP expands ${VAR} and ${VAR:-default} patterns from your environment. Set the variables, run kafkamcp validate, and you're done.

Minute 3: Connect to Claude Desktop​

Open your Claude Desktop configuration file:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

Add the KafkaMCP entry:

{
  "mcpServers": {
    "kafkamcp": {
      "command": "kafkamcp",
      "args": ["--config", "/Users/you/.config/kafkamcp/kafkamcp.yaml"]
    }
  }
}

Use the full path to your config file. Restart Claude Desktop.

Minute 4: Verify the connection​

In Claude, you should see the MCP server indicator showing KafkaMCP is connected. If it doesn't appear, check Claude's MCP logs (Help → MCP Logs) for connection errors.

Now ask Claude:

List all Kafka topics in my cluster

Claude calls kafka_list_topics and returns your topics with partition counts, replication factors, and estimated message counts.

Minute 5: Do something useful​

Read recent messages:

Show me the last 5 messages from orders.created

Inspect a consumer group:

Describe the consumer group order-processor and show me the lag per partition

Produce a message:

Publish a test message to the test-events topic with key "test-1" and value {"event": "hello", "source": "claude"}

Search messages:

Search for messages in orders.created where the value contains "refund"

That's five minutes. Your AI agent now has native, authenticated, schema-aware access to your Kafka cluster — and you didn't write a single line of integration code.

Adding access control​

If you don't want Claude (or other agents) to have full access, add a policy:

policies:
  default_deny: true
  agents:
    - id: "anonymous"
      topics:
        - pattern: "orders.*"
          permissions: [read]
        - pattern: "test-*"
          permissions: [read, write]
      consumer_groups:
        permissions: [describe]
      schemas:
        permissions: [read]
      rate_limit:
        requests_per_minute: 60

Now Claude can read from orders.* topics and read/write test-* topics, but nothing else. Rate-limited to 60 requests per minute.

Next steps​

• Access Control Guide — restrict what agents can do
• Multi-Cluster Setup — connect to multiple Kafka clusters
• API Reference — full tool catalog

KafkaMCP is open source under the Apache 2.0 license. GitHub · Docs · Discussions

Eleven days. One topic.

She wrote a Python consumer that subscribed to orders.dlq, deserialized Avro messages using a hand-managed schema cache, dumped the results into a REST endpoint, wrote an MCP tool definition that called the REST endpoint, added error handling for when the consumer fell behind, and then spent two more days debugging a memory leak in the consumer process. When she was done, the agent could sample dead-letter queue messages. Read-only. No schema awareness. No consumer group inspection. And when the team wanted the agent to also read from payments.failed, the whole process started over.

This is the default experience for every team that wants AI agents to interact with Kafka. It's a hidden tax that compounds with every new topic and every new agent, and it's entirely unnecessary.

The three-layer wrapper pattern​

Here's what the integration looks like for most teams today:

1. Human writes Python/Go consumer → dumps to file or REST API
2. Human writes MCP tool wrapper around the REST API  
3. Agent gets stale, batch-mode, schema-blind access to one topic
4. Repeat for every additional topic

I've seen this pattern at companies running anywhere from 50 to 2,000+ Kafka topics. Each integration takes non-trivial engineering time, produces multiple layers of bespoke code, and delivers an agent that can read but not write, can't discover new topics, has zero understanding of message schemas, and requires its own monitoring and on-call coverage independent of Kafka itself.

Multiply that by the 5-15 agents a typical platform team deploys, and you're looking at a full-time headcount consumed by wrapper maintenance.

Why existing alternatives don't solve this​

The obvious question: why not use Confluent's REST Proxy, or ksqlDB, or embed a Kafka client directly in the agent?

Kafka REST Proxy is a fine HTTP adapter. I've operated it at scale. But it was designed for human-built services, not AI agents. It's not MCP-compatible, so you still need an MCP tool wrapper. It has no schema introspection — your agent receives raw bytes. It has no topic discovery — the agent must be told which topics exist. And every new agent needs its own integration, because REST Proxy has no concept of per-agent access control.

ksqlDB is a stream processing engine. Using it for ad-hoc agent reads is like using a forklift to pick up a pen. It requires a persistent deployment, ongoing operational investment, and still doesn't speak MCP. It solves a different problem.

Embedding a Kafka client directly (kafka-python, librdkafka) creates a tight coupling between the agent framework and Kafka internals. Every agent now needs to manage connections, handle rebalances, negotiate SASL credentials, and deserialize wire-format schemas. This violates separation of concerns and turns every agent developer into a Kafka specialist.

LangChain's Kafka integration exists, but it's a community-contributed wrapper with 5 basic tools, no schema support, no consumer group management, and maintenance that has stalled. It's a starting point, not a solution.

The gap that shouldn't exist​

Apache Kafka processes trillions of messages daily and is used across a broad range of production deployments. MCP adoption is growing rapidly. These are two enormous, converging ecosystems.

Existing open-source Kafka MCP servers provide basic produce/consume with a handful of tools. No schema awareness. No access control. No consumer group operations. No multi-cluster support. Most are starter projects suitable for experimentation, not production governance.

The result: data engineering teams at companies operating hundreds of Kafka topics are individually building the same three-layer wrapper, with the same blind spots.

What a real solution looks like​

I've been operating Kafka at scale for years — 47-broker clusters, controller failovers, ZooKeeper-to-KRaft migrations, thousands of topics across multiple clusters. I built KafkaMCP to be the bridge I wished existed.

It's a single Go binary. You write 8 lines of YAML, point it at your Kafka cluster, and your agent gets MCP tools covering:

• Discovery: list topics and schemas, inspect consumer groups, and retrieve cluster metadata
• Data access: consume with offset/timestamp/latest positioning, produce with delivery confirmation, search by key/header/value/time range
• Operations: create/alter/delete topics, reset consumer group offsets, analyze dead-letter queues
• Governance: schema compatibility checks, approvals, policy evaluation, and live Kafka ACL reconciliation

Every tool call goes through a middleware chain: verified identity → rate limiting → per-agent policy enforcement → execution → audit logging → Prometheus metrics. Guarded packs require production-safe policy and approval configuration.

It's Apache 2.0 licensed, pure Go (no CGO, no JVM, no native deps), and runs on Linux, macOS, and Windows.

KafkaMCP is open source under the Apache 2.0 license. GitHub repository · Documentation · Discussions