Skip to main content

Configuration

KafkaMCP is configured with a single YAML file, usually named kafkamcp.yaml.

Top-level structure​

server:
clusters:
policies:
schema_cache:
audit:
masking:
telemetry:

KafkaMCP validates the file at startup and applies defaults for omitted values where appropriate.

server​

Controls how KafkaMCP listens for MCP traffic.

FieldTypeRequiredDefaultDescription
transportstringNostdioMCP transport. Valid values: stdio, sse, streamable-http.
hoststringNo0.0.0.0Bind host for HTTP transports. Ignored for stdio.
portintegerNo3100Bind port for HTTP transports.
metrics_hoststringNo127.0.0.1Bind host for metrics and health endpoints. Use 0.0.0.0 only behind network controls.
metrics_portintegerNoport + 1Bind port for metrics, /livez, /healthz, and /ready.
log_levelstringNoinfoLog level: debug, info, warn, error.
max_message_sizeintegerNo1048576Max MCP message size in bytes (1 MB). Used to validate produce message payloads.
tls_certstringNo—Path to TLS certificate for HTTPS.
tls_keystringNo—Path to TLS private key for HTTPS.

Notes​

  • tls_cert and tls_key must both be set to enable TLS.
  • The inbound MCP HTTPS server requires TLS 1.3. Kafka broker and Connect clients default to TLS 1.2 for provider compatibility and can be raised to TLS 1.3 per cluster.
  • stdio is the right choice for local MCP subprocess clients such as Claude Desktop.
  • sse and streamable-http are the right choices for network-accessible deployments.

clusters​

Defines one or more Kafka clusters.

At least one cluster is required.

FieldTypeRequiredDefaultDescription
namestringYes—Unique cluster name used by the cluster parameter and kafka://clusters.
bootstrap_serversstringYes—Comma-separated Kafka broker list.
security_protocolstringNoPLAINTEXTKafka security mode: PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. SASL modes require a mechanism and credentials.
sasl_mechanismstringNoemptySASL mechanism such as PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512.
sasl_usernamestringConditionallyemptySASL username. Required when sasl_mechanism is set.
sasl_passwordstringConditionallyemptySASL password. Required when sasl_mechanism is set.
ssl_ca_locationstringNoemptyPath to a CA bundle used for TLS broker validation.
tls_min_versionstringNo1.2Minimum broker TLS version: 1.2 or 1.3.
schema_registryobjectNo—Optional Schema Registry configuration for this cluster.
connect_urlstringNoemptyOptional Kafka Connect base URL for connector tools.
connect_tlsobjectNo—Optional TLS settings for the Kafka Connect REST API.
defaultbooleanNofirst cluster if none setMarks the default cluster.

clusters[].schema_registry​

FieldTypeRequiredDefaultDescription
urlstringYes—Base URL of the Schema Registry.
authobjectNo—Optional basic-auth credentials.

clusters[].schema_registry.auth​

FieldTypeRequiredDefaultDescription
usernamestringNoemptySchema Registry username.
passwordstringNoemptySchema Registry password.

clusters[].connect_tls​

FieldTypeRequiredDefaultDescription
ca_filestringNoemptyPath to a CA certificate file for verifying the Connect REST API server.
insecure_skip_verifybooleanNofalseSkip TLS certificate verification (not recommended for production).
min_versionstringNocluster tls_min_versionMinimum Connect TLS version: 1.2 or 1.3.
tip

If no cluster is marked default: true, KafkaMCP automatically uses the first cluster in the list as the default.

policies​

Defines the per-agent authorization and rate limiting model.

FieldTypeRequiredDefaultDescription
default_denybooleanNofalseDeny requests unless an explicit rule allows them.
agentsarrayNoemptyList of per-agent policy blocks.

policies.agents[]​

FieldTypeRequiredDefaultDescription
idstringYes—Agent identifier matched against X-Agent-ID, then X-Client-ID, then anonymous.
rolestringNoemptyOptional human-readable role label.
topicsarrayNoemptyTopic permission rules keyed by glob pattern.
consumer_groupsobjectNoemptyConsumer group permissions for the agent.
schemasobjectNoemptySchema permissions for the agent.
rate_limitobjectNoemptyPer-agent request budget.

policies.agents[].topics[]​

FieldTypeRequiredDefaultDescription
patternstringYes—Glob pattern such as orders.*, *.dlq, or *.
permissionsarray of stringsYes—Topic permissions: read, write, create, delete, admin.

policies.agents[].consumer_groups​

FieldTypeRequiredDefaultDescription
permissionsarray of stringsNoemptyConsumer group permissions: describe, reset.

policies.agents[].schemas​

FieldTypeRequiredDefaultDescription
permissionsarray of stringsNoemptySchema permissions. Current schema tools use read.

policies.agents[].rate_limit​

FieldTypeRequiredDefaultDescription
requests_per_minuteintegerNo0Per-agent request cap. 0 means no explicit limit is installed.

schema_cache​

Controls the in-memory Schema Registry cache.

FieldTypeRequiredDefaultDescription
max_entriesintegerNo10000Maximum cached schema entries.
ttl_secondsintegerNo300Cache TTL in seconds.

audit​

Controls the audit trail.

FieldTypeRequiredDefaultDescription
enabledbooleanNofalseEnables the audit logger.
max_entriesintegerNo10000Size of the in-memory ring buffer.
log_filestringNoemptyOptional JSONL file path for durable audit logs.

masking​

Controls data masking and PII redaction for consumed messages.

FieldTypeRequiredDefaultDescription
enabledbooleanNofalseEnable data masking / PII redaction.
rulesarrayNoemptyList of masking rule definitions.

masking.rules[]​

FieldTypeRequiredDefaultDescription
namestringYes—Rule identifier.
typestringYes—Rule type: email, ssn, credit_card, phone, regex, ner.
field_pathstringYes—JSON path pattern (e.g., $.user.email, **email**).
patternstringConditionally—Custom regex pattern. Required when type is regex.
replacestringNo***REDACTED***Replacement string for masked values.
agentsarray of stringsNoempty (all agents)Apply only to these agent IDs. Empty means all agents.

Example​

masking:
enabled: true
rules:
- name: redact-emails
type: email
field_path: "**email**"
replace: "***EMAIL***"
- name: redact-ssn
type: ssn
field_path: "$.user.ssn"
- name: custom-api-key
type: regex
field_path: "$.headers.authorization"
pattern: "Bearer [A-Za-z0-9._~+/-]+"
replace: "Bearer ***REDACTED***"
agents: ["incident-agent"]
tip

Masking rules are evaluated in order. The first matching rule wins for each field path.

telemetry​

Controls OpenTelemetry distributed tracing.

FieldTypeRequiredDefaultDescription
enabledbooleanNofalseEnable OpenTelemetry tracing.
endpointstringConditionally—OTLP collector endpoint (e.g., localhost:4317). Required when enabled is true.
insecurebooleanNofalseUse insecure (non-TLS) connection to collector.
service_namestringNokafkamcpService name reported in traces.
sample_ratefloatNo1.0Trace sampling rate (0.0–1.0).

Example​

telemetry:
enabled: true
endpoint: "localhost:4317"
insecure: true
service_name: kafkamcp-production
sample_rate: 0.5
info

KafkaMCP uses the OTLP exporter. Point endpoint at any OpenTelemetry Collector, Jaeger, or compatible backend.

Environment variable expansion​

KafkaMCP expands environment variables before parsing the YAML file.

Supported forms:

  • ${VAR}
  • ${VAR:-default}

Example​

clusters:
- name: production
bootstrap_servers: "${KAFKA_BOOTSTRAP_SERVERS}"
sasl_username: "${KAFKA_USERNAME}"
sasl_password: "${KAFKA_PASSWORD}"
connect_url: "${CONNECT_URL:-http://connect:8083}"

Behavior:

  • ${VAR} fails if VAR is not set
  • ${VAR:-default} uses default when VAR is not set
warning

If you use ${VAR} and the environment variable is missing, KafkaMCP refuses to start.

Runtime transport override​

The KAFKAMCP_TRANSPORT environment variable overrides server.transport from the config file at runtime.

Valid values: stdio, sse, streamable-http.

KAFKAMCP_TRANSPORT=sse ./bin/kafkamcp --config kafkamcp.yaml

Insecure TLS override​

The KAFKAMCP_ALLOW_INSECURE environment variable must be set to true to allow insecure_skip_verify: true in connect_tls settings. This is a safety mechanism requiring explicit opt-in for skipping TLS certificate verification on Kafka Connect REST API connections.

warning

Setting KAFKAMCP_ALLOW_INSECURE=true disables TLS certificate verification for Connect REST API calls. Do not use in production. This is intended for development and testing environments only.

KAFKAMCP_ALLOW_INSECURE=true ./bin/kafkamcp --config kafkamcp.yaml

Log level override​

The KAFKAMCP_LOG_LEVEL environment variable overrides server.log_level from the config file at runtime. Valid values: debug, info, warn, error.

The --log-level CLI flag takes precedence over this environment variable.

KAFKAMCP_LOG_LEVEL=debug ./bin/kafkamcp --config kafkamcp.yaml

Environment variables reference​

VariableDescription
KAFKAMCP_TRANSPORTOverrides server.transport from the config file at runtime. Valid values: stdio, sse, streamable-http.
KAFKAMCP_ALLOW_INSECURESet to true to allow insecure_skip_verify in connect_tls settings. Required as an explicit opt-in for skipping TLS certificate verification.
KAFKAMCP_LOG_LEVELOverrides server.log_level from the config file at runtime. Valid values: debug, info, warn, error. The --log-level CLI flag takes precedence over this variable.

Config validation​

KafkaMCP validates the configuration at startup and refuses to start if any rule fails. This helps catch misconfigurations early before connecting to Kafka.

RuleError Message
At least one cluster requiredat least one cluster must be configured
Cluster name requiredcluster[N]: name is required
Cluster names must be uniquecluster[N]: duplicate cluster name "X"
Bootstrap servers required per clustercluster[N] (name): bootstrap_servers is required
SASL credentials required when mechanism setcluster[N] (name): sasl_username and sasl_password are required when sasl_mechanism is set
Transport must be validinvalid transport "X"; must be one of: stdio, sse, streamable-http
tip

Run ./bin/kafkamcp validate --config kafkamcp.yaml to check your config without starting the server.

Complete annotated example​

server:
transport: sse # stdio | sse | streamable-http
host: 0.0.0.0 # Bind host for HTTP transports
port: 3100 # Main MCP port; metrics will be 3101
log_level: info # debug | info | warn | error
tls_cert: "${TLS_CERT:-}" # Optional HTTPS certificate path
tls_key: "${TLS_KEY:-}" # Optional HTTPS private key path

clusters:
- name: production
bootstrap_servers: "${PROD_BOOTSTRAP_SERVERS}"
security_protocol: SASL_SSL
sasl_mechanism: SCRAM-SHA-512
sasl_username: "${PROD_KAFKA_USERNAME}"
sasl_password: "${PROD_KAFKA_PASSWORD}"
ssl_ca_location: "${PROD_CA_FILE:-/etc/ssl/certs/ca-certificates.crt}"
schema_registry:
url: "${PROD_SCHEMA_REGISTRY_URL}"
auth:
username: "${PROD_SR_USERNAME}"
password: "${PROD_SR_PASSWORD}"
connect_url: "${PROD_CONNECT_URL:-http://connect:8083}"
default: true

- name: staging
bootstrap_servers: "${STAGING_BOOTSTRAP_SERVERS:-staging-kafka:9092}"
security_protocol: PLAINTEXT
schema_registry:
url: "${STAGING_SCHEMA_REGISTRY_URL:-http://staging-schema-registry:8081}"
connect_url: "${STAGING_CONNECT_URL:-http://staging-connect:8083}"

policies:
default_deny: true
agents:
- id: incident-agent
role: reader
topics:
- pattern: "orders.*"
permissions: [read]
- pattern: "*.dlq"
permissions: [read]
consumer_groups:
permissions: [describe]
schemas:
permissions: [read]
rate_limit:
requests_per_minute: 120

- id: pipeline-agent
role: admin
topics:
- pattern: "*"
permissions: [admin]
consumer_groups:
permissions: [describe, reset]
schemas:
permissions: [read]
rate_limit:
requests_per_minute: 600

schema_cache:
max_entries: 10000
ttl_seconds: 300

audit:
enabled: true
max_entries: 10000
log_file: /var/log/kafkamcp/audit.jsonl

masking:
enabled: true
rules:
- name: redact-emails
type: email
field_path: "**email**"
- name: redact-ssn
type: ssn
field_path: "$.user.ssn"

telemetry:
enabled: true
endpoint: "${OTEL_ENDPOINT:-localhost:4317}"
insecure: true
service_name: kafkamcp
sample_rate: 1.0

Validate before startup​

Always validate the file before deploying:

kafkamcp validate --config /path/to/kafkamcp.yaml

That catches:

  • missing clusters
  • duplicate cluster names
  • invalid transports
  • incomplete SASL settings

A clean config keeps KafkaMCP startup predictable and safe.