apache/kafka · critical · IllegalArgumentException

Append of size {} bytes is too large for segment with curren

Error message

Append of size {} bytes is too large for segment with current file position at {}

What it means

Thrown by FileRecords.append when records.sizeInBytes() + current size would overflow Integer.MAX_VALUE. Because size is an int, appending beyond 2 GiB is impossible; this guard rejects the append before writeFullyTo would silently wrap the counter. It complements the constructor's 2 GiB segment-size guard.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:195

            throw new IllegalArgumentException("Invalid size: " + size + " in read from " + this);

        int end = this.start + position + size;
        // Handle integer overflow or if end is beyond the end of the file
        if (end < 0 || end > start + currentSizeInBytes)
            end = this.start + currentSizeInBytes;
        return end - (this.start + position);
    }

    /**
     * Append a set of records to the file. This method is not thread-safe and must be
     * protected with a lock.
     *
     * @param records The records to append
     * @return the number of bytes written to the underlying file
     */
    public int append(MemoryRecords records) throws IOException {
        if (records.sizeInBytes() > Integer.MAX_VALUE - size.get())
            throw new IllegalArgumentException("Append of size " + records.sizeInBytes() +
                    " bytes is too large for segment with current file position at " + size.get());

        int written = records.writeFullyTo(channel);
        size.getAndAdd(written);
        return written;
    }

    /**
     * Commit all written data to the physical disk
     */
    public void flush() throws IOException {
        channel.force(true);
    }

    /**
     * Close this record set
     */
    public void close() throws IOException {

View on GitHub (pinned to c31c9215e1)

Solutions

Lower segment.bytes to the default 1 GiB (1073741824) so segments roll well before the int ceiling.
Verify LogSegment.shouldRoll is being honored on every append and that roll() actually swaps to a new segment.
Cap message.max.bytes / max.message.bytes to a sane value so a single batch cannot push the segment past the limit.
If a segment is already oversized on disk, stop the broker, remove/quarantine the segment, and let ISR replicas rebuild it.

Example fix

// before: segment.bytes raised too high, roll never triggers
props.put(LogConfig.SegmentBytesProp, String.valueOf(Integer.MAX_VALUE));

// after: keep segment.bytes within sane bounds so appends roll in time
props.put(LogConfig.SegmentBytesProp, String.valueOf(1 << 30));

Defensive patterns

Strategy: validation

Validate before calling

int appendBytes = append.sizeInBytes();
int currentBytes = fileRecords.sizeInBytes();
if ((long) appendBytes + (long) currentBytes > Integer.MAX_VALUE) {
    throw new IllegalStateException(
        "Append of " + appendBytes + " bytes would overflow segment at " + currentBytes);
}
fileRecords.append(append);

Prevention

Roll segments (LogSegment.roll) before they approach the 2GiB Integer ceiling.
Cap producer max.request.size and batch.size so a single append can never blow the budget.
Do the overflow check in long arithmetic to avoid silent integer overflow in the guard itself.

When it happens

Trigger: Calling FileRecords.append(memoryRecords) on a segment whose size.get() + records.sizeInBytes() exceeds Integer.MAX_VALUE. Reachable from Log.append or LogSegment.append during normal produce handling once a segment has grown close to 2 GiB.

Common situations: segment.bytes misconfigured to a value near or above 2 GiB, or the roll logic failed to trigger (segment.ms extremely large, broken Log.roll path), so appends keep stacking on one segment. Also possible if a single MemoryRecords batch is unreasonably large (message.max.bytes blown out).

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/ba38cc66579d52e5.json. Report an issue: GitHub.