{"id":"2a9185e3e01f878c","repo":"apache/kafka","slug":"producer-closed-while-send-in-progress-2a9185","errorCode":null,"errorMessage":"Producer closed while send in progress","messagePattern":"Producer closed while send in progress","errorType":"exception","errorClass":"KafkaException","httpStatus":null,"severity":"critical","filePath":"clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java","lineNumber":473,"sourceCode":"\n    /**\n     * Try to append to a ProducerBatch.\n     * <p>\n     * If it is full (or absent), we return {@link RecordAppendResult#NEEDS_NEW_BATCH} and a new batch is created.\n     * We also close the batch for record appends to free up resources like compression buffers. The batch will be\n     * fully closed (ie. the record batch headers will be written and memory records built) in one of the following\n     * cases (whichever comes first): right before send, if it is expired, or when the producer is closed.\n     *\n     * @return one of two outcomes: an {@code appended} result ({@link RecordAppendResult#appended()}) when the\n     * record was appended to the open batch, or {@link RecordAppendResult#NEEDS_NEW_BATCH} when there is\n     * no open batch that can take it (full or absent). The incremental strategy overrides this and may\n     * additionally return a {@link RecordAppendResult#needsExtension(int) needsBufferExtension} result\n     * when the open batch is within its batch-size limit but its chunks lack capacity for the record.\n     */\n    protected RecordAppendResult tryAppend(long timestamp, byte[] key, byte[] value, Header[] headers,\n                                           Callback callback, Deque<ProducerBatch> deque, long nowMs) {\n        if (closed)\n            throw new KafkaException(\"Producer closed while send in progress\");\n        ProducerBatch last = deque.peekLast();\n        if (last != null) {\n            int initialBytes = last.estimatedSizeInBytes();\n            FutureRecordMetadata future = last.tryAppend(timestamp, key, value, headers, callback, nowMs);\n            if (future == null) {\n                last.closeForRecordAppends();\n            } else {\n                int appendedBytes = last.estimatedSizeInBytes() - initialBytes;\n                return RecordAppendResult.appended(future, deque.size() > 1 || last.isFull(), false, appendedBytes);\n            }\n        }\n        return RecordAppendResult.NEEDS_NEW_BATCH;\n    }\n\n    private boolean isMuted(TopicPartition tp) {\n        return muted.contains(tp);\n    }\n","sourceCodeStart":455,"sourceCodeEnd":491,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java#L455-L491","documentation":"Thrown by RecordAccumulator.tryAppend when the accumulator's `closed` flag is true at the moment a record is being appended. tryAppend runs on the user thread inside KafkaProducer.send(); a true flag means close() has already been called (or is in progress), so any further append is rejected to avoid orphaned records that can never be drained by the Sender thread.","triggerScenarios":"Calling KafkaProducer.send() after producer.close() has begun; or close() racing with a concurrent send(). The check at the top of tryAppend fails immediately and propagates out of send() to the caller.","commonSituations":"Producer reused after shutdown in a long-running service; shutdown hook closing the producer while worker threads still enqueue; Spring bean destroyed but background jobs still reference it; a producer field set to null on error and a later send() on a stale reference whose close() was triggered.","solutions":["Stop all send()-calling code before invoking close(); gate sends with an AtomicBoolean 'running' flag flipped in shutdown.","Use a single owner for the producer lifecycle (one creator, one closer) and never share it across independently-stopped components.","Catch this KafkaException at the call site and treat the record as not-sent, then route to a fallback or DLQ; do not retry on the same closed instance.","After close(), create a fresh KafkaProducer if production must continue rather than reusing the closed one."],"exampleFix":"// before\nRuntime.getRuntime().addShutdownHook(new Thread(producer::close));\n// worker threads keep calling producer.send() during shutdown\n\n// after\nprivate final AtomicBoolean open = new AtomicBoolean(true);\n// in worker:\nif (!open.get()) throw new IllegalStateException(\"producer closing\");\nproducer.send(rec);\n// in shutdown hook:\nopen.set(false);\nproducer.close(Duration.ofSeconds(10));","handlingStrategy":"try-catch","validationCode":"// Coordinate close vs send via an AtomicBoolean to eliminate the race at the call site.\nprivate final java.util.concurrent.atomic.AtomicBoolean closed = new java.util.concurrent.atomic.AtomicBoolean();\n...\nif (closed.get()) return; // drop the send; producer is shutting down\nproducer.send(record, callback);","typeGuard":null,"tryCatchPattern":"try {\n    producer.send(record, callback);\n} catch (org.apache.kafka.common.KafkaException ke) {\n    if (ke.getMessage() != null && ke.getMessage().contains(\"Producer closed\")) {\n        // close() raced this send; record was not accepted, safe to drop or hand off\n    }\n}","preventionTips":["Guarantee a happens-before edge between close() and the last send(): close after all senders have stopped.","Use a shutdown flag checked by the sender thread before each send(); close() only after the sender exits.","Never share a producer across threads where one thread may close while another is still sending."],"tags":["producer","lifecycle","concurrency","send","java"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}