apache/kafka · critical · KafkaException

The size of segment {} ({}) is larger than the maximum allow

Error message

The size of segment {} ({}) is larger than the maximum allowed segment size of {}

What it means

Thrown by the FileRecords constructor when the underlying FileChannel reports a size greater than Integer.MAX_VALUE (~2 GiB). FileRecords tracks segment size in a 32-bit AtomicInteger and addresses bytes with int offsets, so a segment larger than 2 GiB cannot be represented. This guard fires once at open time, before any mutable state is initialized.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:69

    private volatile File file;

    /**
     * The {@code FileRecords.open} methods should be used instead of this constructor whenever possible.
     * The constructor is visible for tests.
     */
    FileRecords(
        File file,
        FileChannel channel,
        int end
    ) throws IOException {
        this.file = file;
        this.channel = channel;
        this.start = 0;
        this.end = end;
        this.isSlice = false;

        if (channel.size() > Integer.MAX_VALUE) {
            throw new KafkaException(
                "The size of segment " + file + " (" + channel.size() +
                ") is larger than the maximum allowed segment size of " + Integer.MAX_VALUE
            );
        }

        int limit = Math.min((int) channel.size(), end);
        this.size = new AtomicInteger(limit - start);

        // update the file position to the end of the file
        channel.position(limit);

        batches = batchesFrom(start);
    }

    /**
     * Constructor for creating a slice.
     *
     * This overloaded constructor avoids having to declare a checked IO exception.

View on GitHub (pinned to c31c9215e1)

Solutions

Inspect the on-disk segment file size (ls -l on the log dir); if it truly exceeds 2 GiB it is not a valid Kafka segment — restore from a known-good replica or split/remove the offending file.
Check broker config: ensure segment.bytes is at or below the default 1 GiB and segment.ms triggers rolls before segments grow oversized.
If you intentionally preallocate, keep initFileSize within int range and pass preallocate=true so the constructor uses end=0 rather than the channel size.
Verify the file is not corrupt/truncated by comparing against the active broker's replication metadata before reopening.

Example fix

// before: preallocate grows the channel beyond Integer.MAX_VALUE
FileRecords.open(segmentFile, true, false, Integer.MAX_VALUE + 1L, true);

// after: keep preallocation within int range and let Kafka roll segments
FileRecords.open(segmentFile, true, false, 1 << 30 /* 1 GiB */, true);

Defensive patterns

Strategy: validation

Validate before calling

long fileSize = Files.size(file.toPath());
if (fileSize > Integer.MAX_VALUE) {
    throw new IllegalStateException(
        "Refusing to open segment " + file + ": " + fileSize + " bytes exceeds 2GiB limit");
}
FileRecords records = FileRecords.open(file);

Prevention

Keep broker config segment.bytes well under 2GiB (default 1GiB is safe).
Never manually grow or concatenate Kafka log segment files.
Monitor log directory sizes; a single segment > 2GiB indicates misconfigured segment.bytes or a stalled roll.
If you must open arbitrary files with FileRecords.open, gate it on a size check first.

When it happens

Trigger: Calling FileRecords.open(file, ...) (or the package-private constructor) on a file whose FileChannel.size() exceeds Integer.MAX_VALUE. Reachable from any code path that opens a log segment file: LogSegment recovery, Log.roll, snapshot loading, or tools that reopen an existing on-disk segment.

Common situations: A log segment file was grown past 2 GiB out-of-band (manual dd, a preallocated sparse file, a copied segment from a misconfigured broker with segment.bytes far too large), or segment.bytes/segment.index.bytes were set to nonsensical values. Also seen when an external process concatenates segments into one file before Kafka reopens it.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/260c3d24ce90b803.json. Report an issue: GitHub.