{"id":"c7bc457763aafddb","repo":"apache/kafka","slug":"producer-closed-while-send-in-progress","errorCode":null,"errorMessage":"Producer closed while send in progress","messagePattern":"Producer closed while send in progress","errorType":"exception","errorClass":"KafkaException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/clients/producer/internals/ChunkedRecordAccumulator.java","lineNumber":322,"sourceCode":"    /**\n     * Try to append to a ProducerBatch, with mid-batch chunk extension support.\n     * <p>\n     * If the open batch is within its batch-size limit but its chunked stream lacks chunk\n     * capacity, returns {@link RecordAppendResult#needsExtension(int)} without\n     * attempting the append; the caller allocates chunks outside the deque lock, attaches\n     * them, and retries. Otherwise defers to the parent, which appends or returns\n     * {@link RecordAppendResult#NEEDS_NEW_BATCH}.\n     *\n     * @return a {@link RecordAppendResult#needsExtension(int) needsBufferExtension} result when the open batch is\n     * within its batch-size limit but its chunks lack capacity (the append is not attempted); otherwise the\n     * parent implementation's outcome: an {@code appended} result ({@link RecordAppendResult#appended()}) or\n     * {@link RecordAppendResult#NEEDS_NEW_BATCH}.\n     */\n    @Override\n    protected RecordAppendResult tryAppend(long timestamp, byte[] key, byte[] value, Header[] headers,\n                                           Callback callback, Deque<ProducerBatch> deque, long nowMs) {\n        if (closed)\n            throw new KafkaException(\"Producer closed while send in progress\");\n        ProducerBatch last = deque.peekLast();\n        // Split batches in an incremental deque are plain ProducerBatch (heap-backed, grow-on-demand)\n        // and never need chunk extension, so the check only applies to chunked batches.\n        if (last instanceof ChunkedProducerBatch) {\n            int extensionBytes = ((ChunkedProducerBatch) last).extensionBytesNeeded(timestamp, key, value, headers);\n            if (extensionBytes > 0)\n                return RecordAppendResult.needsExtension(extensionBytes);\n        }\n        return super.tryAppend(timestamp, key, value, headers, callback, deque, nowMs);\n    }\n\n    @Override\n    protected ProducerBatch createProducerBatch(TopicPartition tp, MemoryRecordsBuilder recordsBuilder, long nowMs) {\n        return new ChunkedProducerBatch(tp, recordsBuilder, nowMs);\n    }\n\n    /**\n     * Build a {@link MemoryRecordsBuilder} backed by the chunked stream.","sourceCodeStart":304,"sourceCodeEnd":340,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/clients/producer/internals/ChunkedRecordAccumulator.java#L304-L340","documentation":"Thrown by ChunkedRecordAccumulator.tryAppend when the accumulator has been closed (the producer is shutting down) at the moment a send thread attempts to append to an existing batch. It is the incremental-strategy analogue of the BufferPool close checks: the producer's chunked accumulator refuses to route a record into a batch once close has been called, surfacing a KafkaException back through producer.send.","triggerScenarios":"Calling producer.send(...) on a thread after another thread has closed the producer (or set the accumulator's closed flag via close). The send reaches ChunkedRecordAccumulator.append, which calls tryAppend on the current partition's deque; tryAppend checks this.closed and throws before touching any batch.","commonSituations":"Shutdown races where worker threads keep calling send while a coordinator/container/@PreDestroy hook closes the producer; failed-producer recreation logic that closes the old producer before all in-flight sends have completed; background scheduler tasks that outlive the producer bean in a Spring/CDI context; tests reusing a producer instance across methods and closing it early.","solutions":["Drive shutdown through a single owner: set a 'closing' flag, stop submitting new sends, await in-flight sends/callbacks, then producer.close(Duration).","Wrap producer.send in a component that tracks open/closed state and rejects sends with a clear domain exception once close has begun, so callers never see the raw KafkaException.","Ensure background tasks (schedulers, callbacks) are cancelled before the producer is closed, e.g. via executor.shutdown() + awaitTermination before producer.close."],"exampleFix":"// before - scheduler outlives the producer bean\nscheduler.submit(() -> producer.send(rec));\nproducer.close(); // scheduler may still call send -> tryAppend throws\n\n// after - cancel scheduler, then close\nscheduler.shutdown();\nscheduler.awaitTermination(30, TimeUnit.SECONDS);\nproducer.close(Duration.ofSeconds(10));","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// tryAppend throws KafkaException when the accumulator is closed mid-send.\ntry {\n    producer.send(record);\n} catch (org.apache.kafka.common.KafkaException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"Producer closed\")) {\n        // The chunked accumulator saw close() during the append critical section.\n        // Treat as terminal for this producer; re-enqueue elsewhere.\n    } else throw e;\n}","preventionTips":["Stop submitting records before close(); use a shutdown flag shared with all sender threads.","await callbacks (.get() on the future) before close() so no append is mid-flight.","For the incremental/chunked accumulator specifically, the append loop can release and re-acquire chunks; a close during that loop is fatal to the record.","Pair every producer with one owning lifecycle object that serializes 'stop producing' with 'close'."],"tags":["kafka","java","producer","lifecycle","concurrency","shutdown","incremental"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}