apache/kafka · error · IllegalArgumentException

requested size {sizeBytes} is larger than maxSingleAllocatio

Error message

requested size {sizeBytes} is larger than maxSingleAllocationSize {maxSingleAllocationSize}

What it means

Thrown by SimpleMemoryPool.tryAllocate(int sizeBytes) when the requested size exceeds maxSingleAllocationSize, the per-allocation cap set at pool construction. The cap exists so one oversized request cannot drain the entire pool and starve other users (e.g. one large producer batch cannot consume the whole send buffer). The check runs after the size>0 guard and before any CAS on availableMemory.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/memory/SimpleMemoryPool.java:59

    protected volatile Sensor oomTimeSensor;

    public SimpleMemoryPool(long sizeInBytes, int maxSingleAllocationBytes, boolean strict, Sensor oomPeriodSensor) {
        if (sizeInBytes <= 0 || maxSingleAllocationBytes <= 0 || maxSingleAllocationBytes > sizeInBytes)
            throw new IllegalArgumentException("must provide a positive size and max single allocation size smaller than size."
                + "provided " + sizeInBytes + " and " + maxSingleAllocationBytes + " respectively");
        this.sizeBytes = sizeInBytes;
        this.strict = strict;
        this.availableMemory = new AtomicLong(sizeInBytes);
        this.maxSingleAllocationSize = maxSingleAllocationBytes;
        this.oomTimeSensor = oomPeriodSensor;
    }

    @Override
    public ByteBuffer tryAllocate(int sizeBytes) {
        if (sizeBytes < 1)
            throw new IllegalArgumentException("requested size " + sizeBytes + "<=0");
        if (sizeBytes > maxSingleAllocationSize)
            throw new IllegalArgumentException("requested size " + sizeBytes + " is larger than maxSingleAllocationSize " + maxSingleAllocationSize);

        long available;
        boolean success = false;
        //in strict mode we will only allocate memory if we have at least the size required.
        //in non-strict mode we will allocate memory if we have _any_ memory available (so available memory
        //can dip into the negative and max allocated memory would be sizeBytes + maxSingleAllocationSize)
        long threshold = strict ? sizeBytes : 1;
        while ((available = availableMemory.get()) >= threshold) {
            success = availableMemory.compareAndSet(available, available - sizeBytes);
            if (success)
                break;
        }

        if (success) {
            maybeRecordEndOfDrySpell();
        } else {
            if (oomTimeSensor != null) {
                startOfNoMemPeriod.compareAndSet(0, System.nanoTime());

View on GitHub (pinned to c31c9215e1)

Solutions

1. Compare the message's `requested size` to the `maxSingleAllocationSize` in the exception and decide whether the request or the cap is wrong.
2. To allow larger batches: raise `buffer.memory` proportionally so the derived single-allocation cap grows; on the broker, check `socket.request.max.bytes` and the matching send-buffer pool sizing.
3. To shrink the request: lower `batch.size`, enable or switch compression, split the oversized record, or cap message size at your application boundary.
4. If using Kafka Streams / Connect, verify the corresponding `buffer.memory` and `batch.size` settings on the internal producer are consistent.
5. Add a producer interceptor or pre-send size check to reject oversized records with a clear application error before they reach the pool.

Example fix

// before — batch.size larger than the pool allows
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 1024 * 1024);        // 1 MiB batch
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 512 * 1024);       // 512 KiB pool
// -> tryAllocate throws 'requested size 1048576 is larger than maxSingleAllocationSize ...'

// after — raise buffer.memory so the per-allocation cap covers batch.size
props.put(ProducerConfig.BATCH_SIZE_CONFIG,     1024 * 1024);     // 1 MiB batch
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 32L * 1024 * 1024);// 32 MiB pool

// Respect the pool's per-allocation ceiling.
int sizeBytes = ...;
int max = poolMaxSingleAllocationBytes; // capture at pool construction
if (sizeBytes > max) {
    // split, compress, or reject the request before touching the pool.
    return handleOversized(sizeBytes, max);
}
ByteBuffer buf = pool.tryAllocate(sizeBytes);

// A request fits the pool only if 0 < size <= maxSingleAllocationSize.
static boolean fitsPool(int sizeBytes, int maxSingleAllocationSize) {
    return sizeBytes > 0 && sizeBytes <= maxSingleAllocationSize;
}

try {
    buf = pool.tryAllocate(sizeBytes);
} catch (IllegalArgumentException e) {
    if (sizeBytes > maxSingleAllocationBytes) {
        // graceful degradation: route the oversized payload to a side channel.
        buf = fallbackAllocator.apply(sizeBytes);
    } else {
        throw e;
    }
}

When it happens

Trigger: A producer or broker caller requests a buffer larger than the configured max single allocation. Most commonly: producer BATCH_SIZE_CONFIG set higher than the pool's per-allocation limit, a compression/serde producing an oversized batch, or a request whose encoded size exceeds socket.request.max.bytes (which typically feeds maxSingleAllocationSize).

Common situations: Raising `batch.size` without also raising `buffer.memory` (the per-allocation cap is derived from the pool size); sending very large messages with `max.request.size` left at default; a Connect task emitting oversized records; a schema change or Avro/Protobuf evolution that silently grows payload size past the cap; bumping `socket.request.max.bytes` on the broker without updating the matching send-buffer pool wiring.

Related errors

• must provide a positive size and max single allocation size
• requested size {sizeBytes}<=0
• provided null buffer
• Block checksum mismatch
• mark not supported

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/457fc74dfb6c0335.json. Report an issue: GitHub.