What causes "Failed to close kafka consumer" in apache/kafka?

Inspect the wrapped cause (KafkaException.getCause()) — the real failure is the cause, not the wrapper; fix that root component.

apache/kafka · error · org.apache.kafka.common.KafkaException

Failed to close kafka consumer

Error message

Failed to close kafka consumer

What it means

A wrapping KafkaException ('Failed to close kafka consumer') thrown at the end of the internal close flow when one of the sub-components (metrics, deserializers, telemetry reporter, rebalance callbacks, etc.) threw during shutdown and the error is not being intentionally swallowed. It collects the first exception via a firstException reference and rethrows it as a KafkaException so callers see a single, coherent failure rather than swallowing resource-cleanup errors. InterruptException is rethrown unwrapped; everything else is wrapped here.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java:1687

            backgroundEventReaper.reap(backgroundEventQueue);

        closeQuietly(interceptors, "consumer interceptors", firstException);
        closeQuietly(kafkaConsumerMetrics, "kafka consumer metrics", firstException);
        closeQuietly(asyncConsumerMetrics, "async consumer metrics", firstException);
        closeQuietly(fetchMetricsManager, "consumer fetch metrics", firstException);
        closeQuietly(rebalanceCallbackMetricsManager, "consumer rebalance callback metrics");
        closeQuietly(metrics, "consumer metrics", firstException);
        closeQuietly(deserializers, "consumer deserializers", firstException);
        clientTelemetryReporter.ifPresent(reporter -> closeQuietly(reporter, "async consumer telemetry reporter", firstException));

        AppInfoParser.unregisterAppInfo(CONSUMER_JMX_PREFIX, clientId, metrics);
        log.debug("Kafka consumer has been closed");
        Throwable exception = firstException.get();
        if (exception != null && !swallowException) {
            if (exception instanceof InterruptException) {
                throw (InterruptException) exception;
            }
            throw new KafkaException("Failed to close kafka consumer", exception);
        }
    }

    private Timer createTimerForCloseRequests(Duration timeout) {
        // this.time could be null if an exception occurs in constructor prior to setting the this.time field
        final Time time = (this.time == null) ? Time.SYSTEM : this.time;
        return time.timer(Math.min(timeout.toMillis(), requestTimeoutMs));
    }

    private void autoCommitOnClose(final Timer timer) {
        if (groupMetadata.get().isEmpty() || applicationEventHandler == null)
            return;

        if (autoCommitEnabled)
            commitSyncAllConsumed(timer);

        applicationEventHandler.add(new CommitOnCloseEvent());
    }

View on GitHub (pinned to c31c9215e1)

Solutions

Inspect the wrapped cause (KafkaException.getCause()) — the real failure is the cause, not the wrapper; fix that root component.
For deserializer/telemetry close errors, ensure the underlying client (schema registry, metrics backend) is still alive when the consumer closes; close consumers before shutting down shared registries.
If shutting down the whole JVM, prefer closing in a controlled order (consumer -> producer -> registry -> broker client) rather than concurrent shutdown hooks.
If the cause is an interrupted close, handle InterruptException at the call site and decide whether to suppress or propagate.

Example fix

// before
try {
    consumer.close();
} catch (KafkaException e) {
    log.error("close failed", e); // only sees the wrapper
}

// after
try {
    consumer.close();
} catch (KafkaException e) {
    Throwable cause = e.getCause() != null ? e.getCause() : e;
    log.error("close failed: {}", cause.toString(), cause);
}

Defensive patterns

Strategy: try-catch

Try / catch

try {
    consumer.close(Duration.ofSeconds(30));
} catch (InterruptException e) {
    Thread.currentThread().interrupt(); // restore flag, then suppress further close work
} catch (KafkaException e) {
    // Wraps the first exception raised during shutdown (network, deserializer close, etc.).
    log.error("Consumer close failed; resources may leak", e);
} catch (RuntimeException e) {
    log.error("Unexpected error closing consumer", e);
}

Prevention

Always wrap consumer.close() in try/catch during shutdown so one failing component does not abort the whole teardown.
Restore the interrupt status when InterruptException is caught, so higher-level shutdown logic still works.
Idempotently attempt close in a finally block and guard against double-close (the consumer marks itself closed).
Investigate the wrapped cause — it usually points to a network issue or a misbehaving close callback.

When it happens

Trigger: Close-time cleanup of an AsyncKafkaConsumer where closeQuietly captured a non-interrupt exception: e.g. the ClientTelemetryReporter failed to deregister, AppInfoParser.unregisterAppInfo threw, or a rebalance callback running on close raised. The firstException is then wrapped and rethrown.

Common situations: JVM shutdown hooks racing with the consumer close; metrics registry/MBean server MBean unregistration failures under security managers or duplicate registrations; deserializer close() throwing (e.g. Avro/Protobuf serializer closing a pooled registry client); interrupted threads during shutdown that surface as non-InterruptException in a callback; partial constructor failure leaving some fields null so close partially fails.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/3f17945038ad99b8.json. Report an issue: GitHub.