{"id":"ba38cc66579d52e5","repo":"apache/kafka","slug":"append-of-size-bytes-is-too-large-for-segment-w","errorCode":null,"errorMessage":"Append of size {} bytes is too large for segment with current file position at {}","messagePattern":"Append of size (.+?) bytes is too large for segment with current file position at (.+?)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"critical","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java","lineNumber":195,"sourceCode":"            throw new IllegalArgumentException(\"Invalid size: \" + size + \" in read from \" + this);\n\n        int end = this.start + position + size;\n        // Handle integer overflow or if end is beyond the end of the file\n        if (end < 0 || end > start + currentSizeInBytes)\n            end = this.start + currentSizeInBytes;\n        return end - (this.start + position);\n    }\n\n    /**\n     * Append a set of records to the file. This method is not thread-safe and must be\n     * protected with a lock.\n     *\n     * @param records The records to append\n     * @return the number of bytes written to the underlying file\n     */\n    public int append(MemoryRecords records) throws IOException {\n        if (records.sizeInBytes() > Integer.MAX_VALUE - size.get())\n            throw new IllegalArgumentException(\"Append of size \" + records.sizeInBytes() +\n                    \" bytes is too large for segment with current file position at \" + size.get());\n\n        int written = records.writeFullyTo(channel);\n        size.getAndAdd(written);\n        return written;\n    }\n\n    /**\n     * Commit all written data to the physical disk\n     */\n    public void flush() throws IOException {\n        channel.force(true);\n    }\n\n    /**\n     * Close this record set\n     */\n    public void close() throws IOException {","sourceCodeStart":177,"sourceCodeEnd":213,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java#L177-L213","documentation":"Thrown by FileRecords.append when records.sizeInBytes() + current size would overflow Integer.MAX_VALUE. Because size is an int, appending beyond 2 GiB is impossible; this guard rejects the append before writeFullyTo would silently wrap the counter. It complements the constructor's 2 GiB segment-size guard.","triggerScenarios":"Calling FileRecords.append(memoryRecords) on a segment whose size.get() + records.sizeInBytes() exceeds Integer.MAX_VALUE. Reachable from Log.append or LogSegment.append during normal produce handling once a segment has grown close to 2 GiB.","commonSituations":"segment.bytes misconfigured to a value near or above 2 GiB, or the roll logic failed to trigger (segment.ms extremely large, broken Log.roll path), so appends keep stacking on one segment. Also possible if a single MemoryRecords batch is unreasonably large (message.max.bytes blown out).","solutions":["Lower segment.bytes to the default 1 GiB (1073741824) so segments roll well before the int ceiling.","Verify LogSegment.shouldRoll is being honored on every append and that roll() actually swaps to a new segment.","Cap message.max.bytes / max.message.bytes to a sane value so a single batch cannot push the segment past the limit.","If a segment is already oversized on disk, stop the broker, remove/quarantine the segment, and let ISR replicas rebuild it."],"exampleFix":"// before: segment.bytes raised too high, roll never triggers\nprops.put(LogConfig.SegmentBytesProp, String.valueOf(Integer.MAX_VALUE));\n\n// after: keep segment.bytes within sane bounds so appends roll in time\nprops.put(LogConfig.SegmentBytesProp, String.valueOf(1 << 30));","handlingStrategy":"validation","validationCode":"int appendBytes = append.sizeInBytes();\nint currentBytes = fileRecords.sizeInBytes();\nif ((long) appendBytes + (long) currentBytes > Integer.MAX_VALUE) {\n    throw new IllegalStateException(\n        \"Append of \" + appendBytes + \" bytes would overflow segment at \" + currentBytes);\n}\nfileRecords.append(append);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Roll segments (LogSegment.roll) before they approach the 2GiB Integer ceiling.","Cap producer max.request.size and batch.size so a single append can never blow the budget.","Do the overflow check in long arithmetic to avoid silent integer overflow in the guard itself."],"tags":["file-records","append","segment","config","size-limit"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}