{"id":"dc4efac7d9cb868d","repo":"apache/kafka","slug":"attempt-to-allocate-size-bytes-but-there-is-a-h","errorCode":null,"errorMessage":"Attempt to allocate {size} bytes, but there is a hard limit of {totalMemory} on memory allocations.","messagePattern":"Attempt to allocate (.+?) bytes, but there is a hard limit of (.+?) on memory allocations\\.","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java","lineNumber":141,"sourceCode":"    }\n\n    /**\n     * Allocate a buffer of the given size. This method blocks if there is not enough memory and the buffer pool\n     * is configured with blocking mode.\n     *\n     * @param size The buffer size to allocate in bytes\n     * @param maxTimeToBlockMs The maximum time in milliseconds to block for buffer memory to be available\n     * @return The buffer\n     * @throws InterruptedException If the thread is interrupted while blocked\n     * @throws IllegalArgumentException if size is larger than the total memory controlled by the pool (and hence we would block\n     *         forever)\n     */\n    public ByteBuffer allocate(int size, long maxTimeToBlockMs) throws InterruptedException {\n        if (allocationMode != AllocationMode.FULL)\n            throw new IllegalStateException(\"allocate() is not supported in \" + allocationMode\n                + \" allocation mode; use allocateChunks()\");\n        if (size > this.totalMemory)\n            throw new IllegalArgumentException(\"Attempt to allocate \" + size\n                                               + \" bytes, but there is a hard limit of \"\n                                               + this.totalMemory\n                                               + \" on memory allocations.\");\n\n        ByteBuffer buffer = null;\n        this.lock.lock();\n\n        if (this.closed) {\n            this.lock.unlock();\n            throw new KafkaException(\"Producer closed while allocating memory\");\n        }\n\n        try {\n            // check if we have a free buffer of the right size pooled\n            if (size == poolableSize && !this.free.isEmpty())\n                return this.free.pollFirst();\n\n            // now check if the request is immediately satisfiable with the","sourceCodeStart":123,"sourceCodeEnd":159,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java#L123-L159","documentation":"Thrown by BufferPool.allocate when the requested buffer size exceeds the total memory managed by the pool (the producer's buffer.memory). The pool would never be able to satisfy the request even after every other buffer is deallocated, so it refuses immediately rather than blocking forever. This is an IllegalArgumentException surfaced as part of the send path.","triggerScenarios":"Producing a single record whose serialized size (including overhead) is larger than buffer.memory; calling producer.send with a value larger than the configured buffer.memory. The producer estimates the batch size needed and asks the pool for that many bytes; if it exceeds the hard cap, allocate throws.","commonSituations":"buffer.memory set too low (default 32MB was reduced) while messages are large; sending large payloads (images, PDFs, JSON documents) without proportionally raising buffer.memory; max.message.size raised but buffer.memory left at default or lowered; payloads that occasionally spike in size past the configured buffer.","solutions":["Raise buffer.memory (producer config) so it comfortably exceeds your largest single serialized record plus overhead, e.g. buffer.memory > max.request.size + batch.size headroom.","Reduce the size of the records you publish: compress (compression.type=lz4/gzip/snappy/zstd), split large messages, or move bulky payloads to a blob store and send a reference.","Add a pre-send size check on the serialized key+value and route oversized messages to a side channel or reject them with a clear domain error rather than letting the producer throw."],"exampleFix":"// before\nprops.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 4 * 1024 * 1024L); // 4MB\nprops.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 8 * 1024 * 1024L); // 8MB records\n\n// after - buffer.memory must exceed the largest possible record\nprops.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 16 * 1024 * 1024L);\nprops.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 8 * 1024 * 1024L);","handlingStrategy":"validation","validationCode":"// Record (serialized) must fit within buffer.memory (the pool hard limit).\nlong bufferMemory = (long) producerProps.get(\"buffer.memory\"); // default 33554432\nint est = org.apache.kafka.common.record.AbstractRecords.estimateSizeInBytesUpperBound(\n    org.apache.kafka.common.record.RecordBatch.CURRENT_MAGIC_VALUE,\n    org.apache.kafka.common.record.CompressionType.NONE, key, value, headers);\nif (est > bufferMemory) {\n    throw new IllegalArgumentException(\"record ~\" + est + \"B exceeds buffer.memory=\" + bufferMemory);\n}","typeGuard":null,"tryCatchPattern":"// BufferPool.allocate throws IllegalArgumentException synchronously inside send().\ntry {\n    producer.send(record);\n} catch (org.apache.kafka.common.KafkaException e) {\n    if (e.getCause() instanceof IllegalArgumentException\n            || e.getMessage().contains(\"hard limit\")) {\n        // record is unsendable at this buffer.memory; route to DLQ, do NOT retry unchanged\n    } else throw e;\n}","preventionTips":["Size buffer.memory >= the largest possible serialized record (plus batch overhead).","Cap outbound payload size upstream; reject oversized messages before they reach the producer.","Remember compression may shrink on wire but the buffer holds uncompressed bytes during accumulation.","Keep an eye on key + headers + value: estimateSizeInBytesUpperBound covers all three."],"tags":["kafka","java","producer","memory","configuration","buffer-pool"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}