{"id":"260c3d24ce90b803","repo":"apache/kafka","slug":"the-size-of-segment-is-larger-than-the-max","errorCode":null,"errorMessage":"The size of segment {} ({}) is larger than the maximum allowed segment size of {}","messagePattern":"The size of segment (.+?) \\((.+?)\\) is larger than the maximum allowed segment size of (.+?)","errorType":"exception","errorClass":"KafkaException","httpStatus":null,"severity":"critical","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java","lineNumber":69,"sourceCode":"    private volatile File file;\n\n    /**\n     * The {@code FileRecords.open} methods should be used instead of this constructor whenever possible.\n     * The constructor is visible for tests.\n     */\n    FileRecords(\n        File file,\n        FileChannel channel,\n        int end\n    ) throws IOException {\n        this.file = file;\n        this.channel = channel;\n        this.start = 0;\n        this.end = end;\n        this.isSlice = false;\n\n        if (channel.size() > Integer.MAX_VALUE) {\n            throw new KafkaException(\n                \"The size of segment \" + file + \" (\" + channel.size() +\n                \") is larger than the maximum allowed segment size of \" + Integer.MAX_VALUE\n            );\n        }\n\n        int limit = Math.min((int) channel.size(), end);\n        this.size = new AtomicInteger(limit - start);\n\n        // update the file position to the end of the file\n        channel.position(limit);\n\n        batches = batchesFrom(start);\n    }\n\n    /**\n     * Constructor for creating a slice.\n     *\n     * This overloaded constructor avoids having to declare a checked IO exception.","sourceCodeStart":51,"sourceCodeEnd":87,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java#L51-L87","documentation":"Thrown by the FileRecords constructor when the underlying FileChannel reports a size greater than Integer.MAX_VALUE (~2 GiB). FileRecords tracks segment size in a 32-bit AtomicInteger and addresses bytes with int offsets, so a segment larger than 2 GiB cannot be represented. This guard fires once at open time, before any mutable state is initialized.","triggerScenarios":"Calling FileRecords.open(file, ...) (or the package-private constructor) on a file whose FileChannel.size() exceeds Integer.MAX_VALUE. Reachable from any code path that opens a log segment file: LogSegment recovery, Log.roll, snapshot loading, or tools that reopen an existing on-disk segment.","commonSituations":"A log segment file was grown past 2 GiB out-of-band (manual dd, a preallocated sparse file, a copied segment from a misconfigured broker with segment.bytes far too large), or segment.bytes/segment.index.bytes were set to nonsensical values. Also seen when an external process concatenates segments into one file before Kafka reopens it.","solutions":["Inspect the on-disk segment file size (ls -l on the log dir); if it truly exceeds 2 GiB it is not a valid Kafka segment — restore from a known-good replica or split/remove the offending file.","Check broker config: ensure segment.bytes is at or below the default 1 GiB and segment.ms triggers rolls before segments grow oversized.","If you intentionally preallocate, keep initFileSize within int range and pass preallocate=true so the constructor uses end=0 rather than the channel size.","Verify the file is not corrupt/truncated by comparing against the active broker's replication metadata before reopening."],"exampleFix":"// before: preallocate grows the channel beyond Integer.MAX_VALUE\nFileRecords.open(segmentFile, true, false, Integer.MAX_VALUE + 1L, true);\n\n// after: keep preallocation within int range and let Kafka roll segments\nFileRecords.open(segmentFile, true, false, 1 << 30 /* 1 GiB */, true);","handlingStrategy":"validation","validationCode":"long fileSize = Files.size(file.toPath());\nif (fileSize > Integer.MAX_VALUE) {\n    throw new IllegalStateException(\n        \"Refusing to open segment \" + file + \": \" + fileSize + \" bytes exceeds 2GiB limit\");\n}\nFileRecords records = FileRecords.open(file);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep broker config segment.bytes well under 2GiB (default 1GiB is safe).","Never manually grow or concatenate Kafka log segment files.","Monitor log directory sizes; a single segment > 2GiB indicates misconfigured segment.bytes or a stalled roll.","If you must open arbitrary files with FileRecords.open, gate it on a size check first."],"tags":["file-records","segment","config","storage","size-limit"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}