{"id":"5bf28dc9a829276d","repo":"apache/kafka","slug":"attempt-to-allocate-totalsize-bytes-numchunks","errorCode":null,"errorMessage":"Attempt to allocate {totalSize} bytes ({numChunks} chunks of {chunkSize} = {memoryRequired} bytes), but the hard limit on memory allocations is {totalMemory}.","messagePattern":"Attempt to allocate (.+?) bytes \\((.+?) chunks of (.+?) = (.+?) bytes\\), but the hard limit on memory allocations is (.+?)\\.","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java","lineNumber":399,"sourceCode":"                result.add(allocateByteBuffer(chunkSize));\n            error = false;\n            return result;\n        } finally {\n            if (error) {\n                // The pooled buffers we already drained are also lost on this path; mirror\n                // safeAllocateByteBuffer's behaviour (the bytes return, the ByteBuffer\n                // instances become garbage).\n                releaseReservedBytes(memoryRequired);\n            }\n        }\n    }\n\n    /**\n     * Throw if the request memory rounded up to whole chunks would exceed the pool.\n     */\n    private void throwIfChunksNeededExceedsPool(int totalSize, int numChunks, int chunkSize, long memoryRequired) {\n        if (memoryRequired > totalMemory())\n            throw new IllegalArgumentException(\"Attempt to allocate \" + totalSize + \" bytes (\"\n                + numChunks + \" chunks of \" + chunkSize + \" = \" + memoryRequired + \" bytes), but the \"\n                + \"hard limit on memory allocations is \" + totalMemory() + \".\");\n    }\n\n    // Protected for testing\n    protected void recordWaitTime(long timeNs) {\n        this.waitTime.record(timeNs, time.milliseconds());\n    }\n\n    /**\n     * Record that a record send was dropped because the buffer pool was exhausted. Shared by the\n     * full strategy (allocate) and the incremental strategy ({@link ChunkedRecordAccumulator}),\n     * so both update the same buffer-exhausted metrics.\n     */\n    void recordBufferExhausted() {\n        this.metrics.sensor(\"buffer-exhausted-records\").record();\n    }\n","sourceCodeStart":381,"sourceCodeEnd":417,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java#L381-L417","documentation":"Thrown by BufferPool.allocateChunks via throwIfChunksNeededExceedsPool when the request, rounded up to a whole number of poolable-sized chunks, would exceed the pool's total memory. Unlike allocate()'s byte-exact check, the incremental strategy works in fixed-size chunks, so the request is rounded up (ceil(totalSize / chunkSize) chunks) before the limit is compared; this rounded total can be larger than totalSize, so an in-range byte request can still exceed the cap.","triggerScenarios":"Producing with the incremental buffer.memory strategy (ChunkedRecordAccumulator) a record or extension whose required chunk-rounded memory exceeds buffer.memory. For example, buffer.memory set just below a multiple of the 16KB chunk size so that the rounding pushes the allocation past the cap; or very large records whose first-append estimate rounds up to more chunks than the pool can ever provide.","commonSituations":"Lowering buffer.memory to a value that is not cleanly larger than the chunk-rounded largest record; using large batch.size with the incremental strategy while leaving buffer.memory tight; setting buffer.memory below the default 32MB in memory-constrained containers while payloads are large; mismatches between batch.size, chunk size (16KB), and buffer.memory after a config refactor.","solutions":["Raise buffer.memory so it exceeds the largest possible chunk-rounded allocation; rule of thumb: buffer.memory > batch.size + several chunks of headroom.","Verify the chunk rounding: ensure (ceil(largest_record / 16384) * 16384) is comfortably under buffer.memory for your worst-case record.","Reduce record size (compress, split, or externalize large payloads) so the chunk-rounded requirement stays well under buffer.memory.","If the incremental strategy is not required, set batch.size below the CHUNK_SIZE threshold so the producer uses the full strategy (which checks the byte-exact request instead of the rounded chunk count)."],"exampleFix":"// before - buffer.memory just under a chunk-rounded large record\nprops.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024);\nprops.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 60 * 1024L); // < 4*16384 for a 64KB request\n\n// after - leave headroom for chunk rounding\nprops.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024);\nprops.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 32 * 1024 * 1024L);","handlingStrategy":"validation","validationCode":"// For the incremental buffer.memory strategy: chunk-rounded request must fit the pool.\nlong bufferMemory = (long) producerProps.get(\"buffer.memory\");\nint batchSize = (int) producerProps.get(\"batch.size\"); // poolable/chunk size\nint est = org.apache.kafka.common.record.AbstractRecords.estimateSizeInBytesUpperBound(\n    org.apache.kafka.common.record.RecordBatch.CURRENT_MAGIC_VALUE,\n    org.apache.kafka.common.record.CompressionType.NONE, key, value, headers);\nint numChunks = (est + batchSize - 1) / batchSize;\nlong rounded = (long) numChunks * batchSize;\nif (rounded > bufferMemory) {\n    throw new IllegalArgumentException(\"record chunks \" + rounded + \"B exceed buffer.memory=\" + bufferMemory);\n}","typeGuard":null,"tryCatchPattern":"try {\n    producer.send(record);\n} catch (org.apache.kafka.common.KafkaException e) {\n    Throwable c = e.getCause();\n    if (c instanceof IllegalArgumentException && c.getMessage().contains(\"hard limit\")) {\n        // Incremental strategy can't fit this record even with chunk rounding; DLQ it.\n    } else throw e;\n}","preventionTips":["With the incremental strategy, ensure batch.size divides cleanly into a small multiple of buffer.memory.","Round-trip the chunk math (ceil(size/chunkSize)*chunkSize) when validating max message size.","Keep batch.size moderate so chunk rounding doesn't inflate a borderline record past buffer.memory.","Reject oversized messages upstream; the incremental strategy does not raise the hard pool limit."],"tags":["kafka","java","producer","memory","configuration","buffer-pool","incremental"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}