apache/kafka · error · IllegalArgumentException

Attempt to allocate {size} bytes, but there is a hard limit

Error message

Attempt to allocate {size} bytes, but there is a hard limit of {totalMemory} on memory allocations.

What it means

Thrown by BufferPool.allocate when the requested buffer size exceeds the total memory managed by the pool (the producer's buffer.memory). The pool would never be able to satisfy the request even after every other buffer is deallocated, so it refuses immediately rather than blocking forever. This is an IllegalArgumentException surfaced as part of the send path.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java:141

    }

    /**
     * Allocate a buffer of the given size. This method blocks if there is not enough memory and the buffer pool
     * is configured with blocking mode.
     *
     * @param size The buffer size to allocate in bytes
     * @param maxTimeToBlockMs The maximum time in milliseconds to block for buffer memory to be available
     * @return The buffer
     * @throws InterruptedException If the thread is interrupted while blocked
     * @throws IllegalArgumentException if size is larger than the total memory controlled by the pool (and hence we would block
     *         forever)
     */
    public ByteBuffer allocate(int size, long maxTimeToBlockMs) throws InterruptedException {
        if (allocationMode != AllocationMode.FULL)
            throw new IllegalStateException("allocate() is not supported in " + allocationMode
                + " allocation mode; use allocateChunks()");
        if (size > this.totalMemory)
            throw new IllegalArgumentException("Attempt to allocate " + size
                                               + " bytes, but there is a hard limit of "
                                               + this.totalMemory
                                               + " on memory allocations.");

        ByteBuffer buffer = null;
        this.lock.lock();

        if (this.closed) {
            this.lock.unlock();
            throw new KafkaException("Producer closed while allocating memory");
        }

        try {
            // check if we have a free buffer of the right size pooled
            if (size == poolableSize && !this.free.isEmpty())
                return this.free.pollFirst();

            // now check if the request is immediately satisfiable with the

View on GitHub (pinned to c31c9215e1)

Solutions

Raise buffer.memory (producer config) so it comfortably exceeds your largest single serialized record plus overhead, e.g. buffer.memory > max.request.size + batch.size headroom.
Reduce the size of the records you publish: compress (compression.type=lz4/gzip/snappy/zstd), split large messages, or move bulky payloads to a blob store and send a reference.
Add a pre-send size check on the serialized key+value and route oversized messages to a side channel or reject them with a clear domain error rather than letting the producer throw.

Example fix

// before
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 4 * 1024 * 1024L); // 4MB
props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 8 * 1024 * 1024L); // 8MB records

// after - buffer.memory must exceed the largest possible record
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 16 * 1024 * 1024L);
props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 8 * 1024 * 1024L);

Defensive patterns

Strategy: validation

Validate before calling

// Record (serialized) must fit within buffer.memory (the pool hard limit).
long bufferMemory = (long) producerProps.get("buffer.memory"); // default 33554432
int est = org.apache.kafka.common.record.AbstractRecords.estimateSizeInBytesUpperBound(
    org.apache.kafka.common.record.RecordBatch.CURRENT_MAGIC_VALUE,
    org.apache.kafka.common.record.CompressionType.NONE, key, value, headers);
if (est > bufferMemory) {
    throw new IllegalArgumentException("record ~" + est + "B exceeds buffer.memory=" + bufferMemory);
}

Try / catch

// BufferPool.allocate throws IllegalArgumentException synchronously inside send().
try {
    producer.send(record);
} catch (org.apache.kafka.common.KafkaException e) {
    if (e.getCause() instanceof IllegalArgumentException
            || e.getMessage().contains("hard limit")) {
        // record is unsendable at this buffer.memory; route to DLQ, do NOT retry unchanged
    } else throw e;
}

Prevention

Size buffer.memory >= the largest possible serialized record (plus batch overhead).
Cap outbound payload size upstream; reject oversized messages before they reach the producer.
Remember compression may shrink on wire but the buffer holds uncompressed bytes during accumulation.
Keep an eye on key + headers + value: estimateSizeInBytesUpperBound covers all three.

When it happens

Trigger: Producing a single record whose serialized size (including overhead) is larger than buffer.memory; calling producer.send with a value larger than the configured buffer.memory. The producer estimates the batch size needed and asks the pool for that many bytes; if it exceeds the hard cap, allocate throws.

Common situations: buffer.memory set too low (default 32MB was reduced) while messages are large; sending large payloads (images, PDFs, JSON documents) without proportionally raising buffer.memory; max.message.size raised but buffer.memory left at default or lowered; payloads that occasionally spike in size past the configured buffer.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/dc4efac7d9cb868d.json. Report an issue: GitHub.