apache/kafka · critical · KafkaException

Requested metadata update after close

Error message

Requested metadata update after close

What it means

Thrown by ProducerMetadata.awaitUpdate after the loop breaks because isClosed() became true while waiting for a metadata version bump. It indicates a race where the application closed the producer (or it was closed by an error path) while a send() call was still blocked on metadata. The producer refuses to serve metadata to a closed instance because further sends would be impossible anyway.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerMetadata.java:169

    /**
     * Wait for metadata update until the current version is larger than the last version we know of
     */
    public synchronized void awaitUpdate(final int lastVersion, final Timer timer) throws InterruptedException {
        while (true) {
            // Throw fatal exceptions, if there are any. Recoverable topic errors will be handled by the caller.
            maybeThrowFatalException();
            if (updateVersion() > lastVersion || isClosed())
                break;

            timer.update();
            if (timer.isExpired())
                throw new TimeoutException("Failed to update metadata after " + timer.timeoutMs() + " ms.");

            wait(timer.remainingMs());
        }

        if (isClosed())
            throw new KafkaException("Requested metadata update after close");
    }

    @Override
    public synchronized void update(int requestVersion, MetadataResponse response, boolean isPartialUpdate, long nowMs) {
        super.update(requestVersion, response, isPartialUpdate, nowMs);
        errors = response.errors();

        // Remove all topics in the response that are in the new topic set. Note that if an error was encountered for a
        // new topic's metadata, then any work to resolve the error will include the topic in a full metadata update.
        if (!newTopics.isEmpty()) {
            for (MetadataResponse.TopicMetadata metadata : response.topicMetadata()) {
                newTopics.remove(metadata.topic());
            }
        }

        notifyAll();
    }

View on GitHub (pinned to c31c9215e1)

Solutions

1. Ensure no thread is calling send() when close() is invoked: drain in-flight sends (flush + callback completion) before close.
2. Do not share a single KafkaProducer instance across threads that can independently close it; own the lifecycle in one place.
3. Use a coordinated shutdown (e.g. close(Duration)) after a final flush() and after all send-calling workers have stopped.
4. Catch and treat this exception as a shutdown signal rather than a retryable error — the producer is gone.

Example fix

// before
new Thread(() -> producer.send(rec)).start();
producer.close(); // races with the send above

// after
producer.send(rec, cb -> { latch.countDown(); });
producer.flush();
latch.await();
producer.close(Duration.ofSeconds(10));

// KafkaProducer does not expose isClosed(); track it yourself at the single ownership point.
private final java.util.concurrent.atomic.AtomicBoolean closed = new java.util.concurrent.atomic.AtomicBoolean();
...
if (closed.get()) throw new IllegalStateException("producer already closed");
producer.partitionsFor(topic); // or any metadata-dependent call

try {
    producer.partitionsFor(topic);
} catch (org.apache.kafka.common.KafkaException ke) {
    if (ke.getMessage() != null && ke.getMessage().contains("after close")) {
        // producer was closed concurrently; stop using it
    }
}

When it happens

Trigger: One thread calls producer.close() while another thread is inside producer.send() blocked in awaitUpdate(); or close() races with an inflight metadata refresh triggered by an unknown topic. Also reachable if the producer's metadata instance is closed by a fatal error path concurrently with send().

Common situations: Concurrent use of a KafkaProducer from multiple threads where one thread tears it down (shutdown hook, container stop, bean @PreDestroy); producers shared across request threads in a web app that also closes on undeploy; mis-ordered shutdown that closes the producer before draining in-flight sends.

Related errors

• Producer closed while allocating memory
• Producer closed while send in progress
• Producer closed while send in progress
• Timeout after waiting for {timeoutMillis} ms.
• Failed to update metadata after {timeoutMs} ms.

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/c0b4703c7af2b9ec.json. Report an issue: GitHub.