{"id":"9e67498f9d7d88c3","repo":"apache/kafka","slug":"producer-closed-while-allocating-memory","errorCode":null,"errorMessage":"Producer closed while allocating memory","messagePattern":"Producer closed while allocating memory","errorType":"exception","errorClass":"KafkaException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java","lineNumber":151,"sourceCode":"     * @throws IllegalArgumentException if size is larger than the total memory controlled by the pool (and hence we would block\n     *         forever)\n     */\n    public ByteBuffer allocate(int size, long maxTimeToBlockMs) throws InterruptedException {\n        if (allocationMode != AllocationMode.FULL)\n            throw new IllegalStateException(\"allocate() is not supported in \" + allocationMode\n                + \" allocation mode; use allocateChunks()\");\n        if (size > this.totalMemory)\n            throw new IllegalArgumentException(\"Attempt to allocate \" + size\n                                               + \" bytes, but there is a hard limit of \"\n                                               + this.totalMemory\n                                               + \" on memory allocations.\");\n\n        ByteBuffer buffer = null;\n        this.lock.lock();\n\n        if (this.closed) {\n            this.lock.unlock();\n            throw new KafkaException(\"Producer closed while allocating memory\");\n        }\n\n        try {\n            // check if we have a free buffer of the right size pooled\n            if (size == poolableSize && !this.free.isEmpty())\n                return this.free.pollFirst();\n\n            // now check if the request is immediately satisfiable with the\n            // memory on hand or if we need to block\n            int freeListSize = freeSize() * this.poolableSize;\n            if (this.nonPooledAvailableMemory + freeListSize >= size) {\n                // we have enough unallocated or pooled memory to immediately\n                // satisfy the request, but need to allocate the buffer\n                freeUp(size);\n                this.nonPooledAvailableMemory -= size;\n            } else {\n                // we are out of memory and will have to block\n                int accumulated = 0;","sourceCodeStart":133,"sourceCodeEnd":169,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java#L133-L169","documentation":"Thrown by BufferPool.allocate when the producer's BufferPool has already been closed by the time a thread acquires the lock to allocate. Closing the producer marks the pool closed; any subsequent or in-flight allocation attempt that reaches the allocate path aborts with a KafkaException rather than handing out memory from a defunct pool.","triggerScenarios":"One thread calls KafkaProducer.close() while another thread is concurrently calling producer.send(...) whose internal allocate() runs after close set the closed flag. Also when a send is attempted after the producer was already closed elsewhere (e.g. shutdown hook, container teardown, or a finally block that closed the producer before a still-running worker finished).","commonSituations":"Application shutdown races where the producer bean/resource is closed before all worker threads have drained; try-with-resources or @PreDestroy closing the producer while background senders are still active; integration tests that close the producer between iterations but reuse worker threads; reconnect/restart logic that closes a failed producer while callbacks are still in flight.","solutions":["Order shutdown: stop accepting new sends first (signal workers to drain), await in-flight sends/callbacks, then close the producer. Use producer.close(Duration) with a timeout to flush.","Guard application code so sends are not issued after close: track a 'closed' flag in your producer wrapper and reject new sends with a clear domain exception before they hit the buffer pool.","Use a single owner for the producer's lifecycle (e.g. a managed singleton) and ensure no thread can reach producer.send after the owner has begun closing it."],"exampleFix":"// before - shutdown hook closes producer while workers still sending\nRuntime.getRuntime().addShutdownHook(new Thread(producer::close));\n\n// after - drain workers first, then close with a flush timeout\nRuntime.getRuntime().addShutdownHook(new Thread(() -> {\n    workerPool.shutdown();\n    workerPool.awaitTermination(30, TimeUnit.SECONDS);\n    producer.close(Duration.ofSeconds(10));\n}));","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Producer was closed before/at allocation. send() surfaces this as KafkaException.\ntry {\n    producer.send(record);\n} catch (org.apache.kafka.common.KafkaException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"Producer closed\")) {\n        // producer is shutting down; stop submitting, route remaining records elsewhere\n    } else throw e;\n}","preventionTips":["Close the producer only after all futures returned by send() have completed (callback or .get()).","Use a single owner thread for producer lifecycle; never share close() with send() threads unsynchronized.","Track an AtomicBoolean 'closed' in your wrapper and check it before submitting, as a fast-fail hint.","In shutdown hooks, flush() then close() with a timeout, and stop enqueuing new records first."],"tags":["kafka","java","producer","lifecycle","concurrency","shutdown"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}