apache/kafka · error · IllegalArgumentException

position+length should not be greater than buffer.limit(), p

Error message

position+length should not be greater than buffer.limit(), position: " + position + ", length: " + length + ", buffer.limit(): " + buffer.limit()

What it means

Thrown by MemoryRecords.writeTo(channel, position, length) when the requested [position, position+length) byte range exceeds buffer.limit(). It guards the underlying transfer so callers cannot read past the end of the backing ByteBuffer. The message prints all three values (position, length, buffer.limit()) so the arithmetic gap is visible.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/MemoryRecords.java:72

    private final Iterable<MutableRecordBatch> batches = this::batchIterator;

    private int validBytes = -1;

    // Construct a writable memory records
    private MemoryRecords(ByteBuffer buffer) {
        Objects.requireNonNull(buffer, "buffer should not be null");
        this.buffer = buffer;
    }

    @Override
    public int sizeInBytes() {
        return buffer.limit();
    }

    @Override
    public int writeTo(TransferableChannel channel, int position, int length) throws IOException {
        if (((long) position) + length > buffer.limit())
            throw new IllegalArgumentException("position+length should not be greater than buffer.limit(), position: "
                    + position + ", length: " + length + ", buffer.limit(): " + buffer.limit());

        return Utils.tryWriteTo(channel, position, length, buffer);
    }

    /**
     * Write all records to the given channel (including partial records).
     * @param channel The channel to write to
     * @return The number of bytes written
     * @throws IOException For any IO errors writing to the channel
     */
    public int writeFullyTo(GatheringByteChannel channel) throws IOException {
        buffer.mark();
        int written = 0;
        while (written < sizeInBytes())
            written += channel.write(buffer);
        buffer.reset();
        return written;

View on GitHub (pinned to c31c9215e1)

Solutions

1. Use buffer.limit() (or MemoryRecords.sizeInBytes()/validBytes()) as the authoritative bound when computing length.
2. For partial trailing-batch trims, pass validBytes() rather than sizeInBytes().
3. Recompute position and length from the current buffer state immediately before the call rather than caching.
4. If summing batch sizes, guard against integer overflow and clamp to (limit - position).

Example fix

// before
int len = buffer.capacity();
records.writeTo(channel, position, len);
// after
int len = Math.min(requestedLen, buffer.limit() - position);
records.writeTo(channel, position, len);

// MemoryRecords.writeTo requires position+length <= buffer.limit().
int limit = records.sizeInBytes();
if (position < 0 || length < 0 || (long) position + length > limit) {
    throw new IllegalArgumentException(
        "Refusing writeTo: position=" + position + ", length=" + length + ", limit=" + limit);
}

// Confirm the requested slice is within the records buffer before writing to a channel.
static boolean withinBufferBounds(MemoryRecords records, int position, int length) {
    int limit = records.sizeInBytes();
    return position >= 0 && length >= 0 && (long) position + length <= limit;
}

try {
    int written = records.writeTo(channel, position, length);
} catch (IllegalArgumentException e) {
    if (e.getMessage().startsWith("position+length should not be greater than buffer.limit()")) {
        // Clamp length to remaining bytes and retry once.
        length = records.sizeInBytes() - position;
        written = records.writeTo(channel, position, length);
    } else {
        throw e;
    }
}

When it happens

Trigger: Calling MemoryRecords.writeTo(...) with a (position,length) pair whose sum exceeds the sizeInBytes()/buffer.limit(). Typically reached during fetch/send paths, replica transfer, or any code that slices a MemoryRecords buffer and computes length from a stale or off-by-one size (e.g. using buffer.capacity() instead of limit(), or summing partial batch sizes).

Common situations: Off-by-one length after trimming trailing partial batch (length should be validBytes not sizeInBytes), buffer compaction that moved limit but caller still holds old size, position computed from a different buffer's index, or integer overflow when casting accumulated offsets.

Related errors

• Incorrect declared batch size, premature EOF reached
• Incorrect declared batch size, records still remaining in fi
• Failed to decompress record stream
• Error checking for remaining bytes after reading batch
• Failed to close record stream

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/cde212d06456a353.json. Report an issue: GitHub.