apache/kafka · error · IllegalArgumentException

Attempt to allocate {totalSize} bytes ({numChunks} chunks of

Error message

Attempt to allocate {totalSize} bytes ({numChunks} chunks of {chunkSize} = {memoryRequired} bytes), but the hard limit on memory allocations is {totalMemory}.

What it means

Thrown by BufferPool.allocateChunks via throwIfChunksNeededExceedsPool when the request, rounded up to a whole number of poolable-sized chunks, would exceed the pool's total memory. Unlike allocate()'s byte-exact check, the incremental strategy works in fixed-size chunks, so the request is rounded up (ceil(totalSize / chunkSize) chunks) before the limit is compared; this rounded total can be larger than totalSize, so an in-range byte request can still exceed the cap.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java:399

                result.add(allocateByteBuffer(chunkSize));
            error = false;
            return result;
        } finally {
            if (error) {
                // The pooled buffers we already drained are also lost on this path; mirror
                // safeAllocateByteBuffer's behaviour (the bytes return, the ByteBuffer
                // instances become garbage).
                releaseReservedBytes(memoryRequired);
            }
        }
    }

    /**
     * Throw if the request memory rounded up to whole chunks would exceed the pool.
     */
    private void throwIfChunksNeededExceedsPool(int totalSize, int numChunks, int chunkSize, long memoryRequired) {
        if (memoryRequired > totalMemory())
            throw new IllegalArgumentException("Attempt to allocate " + totalSize + " bytes ("
                + numChunks + " chunks of " + chunkSize + " = " + memoryRequired + " bytes), but the "
                + "hard limit on memory allocations is " + totalMemory() + ".");
    }

    // Protected for testing
    protected void recordWaitTime(long timeNs) {
        this.waitTime.record(timeNs, time.milliseconds());
    }

    /**
     * Record that a record send was dropped because the buffer pool was exhausted. Shared by the
     * full strategy (allocate) and the incremental strategy ({@link ChunkedRecordAccumulator}),
     * so both update the same buffer-exhausted metrics.
     */
    void recordBufferExhausted() {
        this.metrics.sensor("buffer-exhausted-records").record();
    }

View on GitHub (pinned to c31c9215e1)

Solutions

Raise buffer.memory so it exceeds the largest possible chunk-rounded allocation; rule of thumb: buffer.memory > batch.size + several chunks of headroom.
Verify the chunk rounding: ensure (ceil(largest_record / 16384) * 16384) is comfortably under buffer.memory for your worst-case record.
Reduce record size (compress, split, or externalize large payloads) so the chunk-rounded requirement stays well under buffer.memory.
If the incremental strategy is not required, set batch.size below the CHUNK_SIZE threshold so the producer uses the full strategy (which checks the byte-exact request instead of the rounded chunk count).

Example fix

// before - buffer.memory just under a chunk-rounded large record
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024);
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 60 * 1024L); // < 4*16384 for a 64KB request

// after - leave headroom for chunk rounding
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024);
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 32 * 1024 * 1024L);

Defensive patterns

Strategy: validation

Validate before calling

// For the incremental buffer.memory strategy: chunk-rounded request must fit the pool.
long bufferMemory = (long) producerProps.get("buffer.memory");
int batchSize = (int) producerProps.get("batch.size"); // poolable/chunk size
int est = org.apache.kafka.common.record.AbstractRecords.estimateSizeInBytesUpperBound(
    org.apache.kafka.common.record.RecordBatch.CURRENT_MAGIC_VALUE,
    org.apache.kafka.common.record.CompressionType.NONE, key, value, headers);
int numChunks = (est + batchSize - 1) / batchSize;
long rounded = (long) numChunks * batchSize;
if (rounded > bufferMemory) {
    throw new IllegalArgumentException("record chunks " + rounded + "B exceed buffer.memory=" + bufferMemory);
}

Try / catch

try {
    producer.send(record);
} catch (org.apache.kafka.common.KafkaException e) {
    Throwable c = e.getCause();
    if (c instanceof IllegalArgumentException && c.getMessage().contains("hard limit")) {
        // Incremental strategy can't fit this record even with chunk rounding; DLQ it.
    } else throw e;
}

Prevention

With the incremental strategy, ensure batch.size divides cleanly into a small multiple of buffer.memory.
Round-trip the chunk math (ceil(size/chunkSize)*chunkSize) when validating max message size.
Keep batch.size moderate so chunk rounding doesn't inflate a borderline record past buffer.memory.
Reject oversized messages upstream; the incremental strategy does not raise the hard pool limit.

When it happens

Trigger: Producing with the incremental buffer.memory strategy (ChunkedRecordAccumulator) a record or extension whose required chunk-rounded memory exceeds buffer.memory. For example, buffer.memory set just below a multiple of the 16KB chunk size so that the rounding pushes the allocation past the cap; or very large records whose first-append estimate rounds up to more chunks than the pool can ever provide.

Common situations: Lowering buffer.memory to a value that is not cleanly larger than the chunk-rounded largest record; using large batch.size with the incremental strategy while leaving buffer.memory tight; setting buffer.memory below the default 32MB in memory-constrained containers while payloads are large; mismatches between batch.size, chunk size (16KB), and buffer.memory after a config refactor.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/5bf28dc9a829276d.json. Report an issue: GitHub.