apache/kafka · error · KafkaException

Failed to construct kafka producer

Error message

Failed to construct kafka producer

What it means

Catch-all wrapper thrown from the KafkaProducer constructor's outer try/catch (KAFKA-2121). Any Throwable raised while building the producer — config parsing, Metrics, serializers, transaction manager, network client, Sender thread start — is first cleaned up via close(Duration.ZERO, true) to avoid leaking threads/sockets, then rethrown wrapped as KafkaException with this message and the original as cause. The actual reason is in getCause().

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:524

                        PRODUCER_METRIC_GROUP_NAME,
                        time,
                        transactionManager,
                        new BufferPool(this.totalMemorySize, batchSize, metrics, time, PRODUCER_METRIC_GROUP_NAME, BufferPool.AllocationMode.FULL));
            }

            this.errors = this.metrics.sensor("errors");
            this.sender = newSender(logContext, kafkaClient, this.metadata);
            String ioThreadName = NETWORK_THREAD_PREFIX + " | " + clientId;
            this.ioThread = new Sender.SenderThread(ioThreadName, this.sender, true);
            this.ioThread.start();
            config.logUnused();
            AppInfoParser.registerAppInfo(JMX_PREFIX, clientId, metrics, time.milliseconds());
            log.debug("Kafka producer started");
        } catch (Throwable t) {
            // call close methods if internal objects are already constructed this is to prevent resource leak. see KAFKA-2121
            close(Duration.ofMillis(0), true);
            // now propagate the exception
            throw new KafkaException("Failed to construct kafka producer", t);
        }
    }

    // visible for testing
    KafkaProducer(ProducerConfig config,
                  LogContext logContext,
                  Metrics metrics,
                  Serializer<K> keySerializer,
                  Serializer<V> valueSerializer,
                  ProducerMetadata metadata,
                  RecordAccumulator accumulator,
                  TransactionManager transactionManager,
                  Sender sender,
                  ProducerInterceptors<K, V> interceptors,
                  Partitioner partitioner,
                  Time time,
                  Sender.SenderThread ioThread,
                  Optional<ClientTelemetryReporter> clientTelemetryReporter) {

View on GitHub (pinned to c31c9215e1)

Solutions

1. Read the wrapped cause — exception.getCause() (or the log line just above) carries the real reason; act on that, not on this message.
2. If the cause is ConfigException, fix the named property; if ClassNotFoundException, add the missing dependency (serializer, login module, metrics reporter).
3. Validate producer properties with ProducerConfig.parseAndValidate(props) or a small smoke-test main() before deploying.
4. For security-related causes (SASL/SSL/Kerberos), confirm the JAAS/config files are present and readable by the JVM at construction time.
5. Check for conflicting jars on the classpath (multiple kafka-clients versions, shaded serializer duplicates).

Example fix

// before
try {
    Producer<String, byte[]> p = new KafkaProducer<>(props);
} catch (KafkaException e) {
    log.error("producer failed", e); // message is generic, root cause hidden
}

// after — unwrap and surface the real reason
try {
    Producer<String, byte[]> p = new KafkaProducer<>(props);
} catch (KafkaException e) {
    Throwable cause = e.getCause() != null ? e.getCause() : e;
    log.error("producer construction failed: {}", cause.getMessage(), cause);
    throw new RuntimeException("cannot start producer: " + cause.getMessage(), cause);
}

// KafkaProducer's constructor wraps ANY failure (bad config, missing class,
// serializer error, security, ...) in KafkaException("Failed to construct kafka
// producer", cause). You cannot fully pre-validate every internal step, so the
// reliable defense is to catch and inspect getCause().
// Useful pre-check: validate the Properties via ProducerConfig without building
// the producer, which surfaces most ConfigException issues early.
try {
    org.apache.kafka.clients.producer.ProducerConfig me =
        new org.apache.kafka.clients.producer.ProducerConfig(props);
    me.values();   // throws ConfigException on bad/unknown keys
} catch (org.apache.kafka.common.config.ConfigException e) {
    log.error("Invalid producer config, will not attempt construction", e);
}

// Construction can fail for many reasons; always inspect the cause and free
// any partial resources (the producer itself closes them, but your code must
// not retain a half-built reference).
KafkaProducer<K,V> producer;
try {
    producer = new KafkaProducer<>(props, keySer, valSer);
} catch (org.apache.kafka.common.KafkaException e) {
    Throwable c = e.getCause();
    if (c instanceof org.apache.kafka.common.config.ConfigException) {
        log.error("Bad producer config", c);
    } else if (c instanceof ClassNotFoundException) {
        log.error("Serializer/partitioner class not on classpath", c);
    } else {
        log.error("Producer construction failed", c);
    }
    throw e;            // or fall back to a different config / fail fast
}

When it happens

Trigger: Any failure during new KafkaProducer<>(props): invalid/unknown config keys, missing or unserializable key/value.serializer.class, bad bootstrap.servers format, SSL/SASL/JAAS misconfiguration, Kerberos login failure, Metrics/MetricReporter instantiation error, enable.idempotence with an incompatible acks/max.in.flight, transactional.id set without proper broker support, classpath issues loading plugins.

Common situations: Wrong serializer class name; typo in bootstrap.servers (e.g. missing port); JAAS config not on classpath or wrong path; Kerberos ticket expired at producer creation; serializer jar not on classpath in a fat-jar that excluded it; conflicting client library versions; producer config copied from another service with environment-specific values that don't resolve.

Related errors

• The ${ProducerConfig.BUFFER_MEMORY_ALLOCATION_STRATEGY_INCRE
• ${ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG} should be equal
• Transaction already started
• Producer closed while allocating memory
• Producer closed while send in progress

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/e0bff175ca09e421.json. Report an issue: GitHub.