apache/kafka · error · IllegalArgumentException

Slice from position {} exceeds end position of {}

Error message

Slice from position {} exceeds end position of {}

What it means

Thrown by FileRecords.availableBytes when the requested slice start position exceeds the segment's current sizeInBytes. Because size is read from a AtomicInteger that can change under concurrent appends, the check uses a cached snapshot; a position past the end has no bytes to read. This protects against reads of data that was never written.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:175

     * @param position The start position to begin the read from
     * @param size The number of bytes after the start position to include
     * @return A unaligned slice of records on this message set limited based on the given position and size
     */
    public UnalignedFileRecords sliceUnaligned(int position, int size) {
        int availableBytes = availableBytes(position, size);
        return new UnalignedFileRecords(channel, this.start + position, availableBytes);
    }

    private int availableBytes(int position, int size) {
        // Cache current size in case concurrent write changes it
        int currentSizeInBytes = sizeInBytes();

        if (position < 0)
            throw new IllegalArgumentException("Invalid position: " + position + " in read from " + this);
        // position should always be relative to the start of the file hence compare with file size
        // to verify if the position is within the file.
        if (position > currentSizeInBytes)
            throw new IllegalArgumentException("Slice from position " + position + " exceeds end position of " + this);
        if (size < 0)
            throw new IllegalArgumentException("Invalid size: " + size + " in read from " + this);

        int end = this.start + position + size;
        // Handle integer overflow or if end is beyond the end of the file
        if (end < 0 || end > start + currentSizeInBytes)
            end = this.start + currentSizeInBytes;
        return end - (this.start + position);
    }

    /**
     * Append a set of records to the file. This method is not thread-safe and must be
     * protected with a lock.
     *
     * @param records The records to append
     * @return the number of bytes written to the underlying file
     */
    public int append(MemoryRecords records) throws IOException {

View on GitHub (pinned to c31c9215e1)

Solutions

1. Re-check the segment's actual sizeInBytes (or FileChannel.size()) at the call site and recompute the slice position against it.
2. If racing with truncation, serialize reads against the log lock so position is validated against a stable size snapshot.
3. Validate the offset-index file (.index) matches the .log file size; rebuild indexes with kafka-dump-log or the broker's index recovery if they diverge.
4. For raft snapshot slicing, ensure the snapshot size has been flushed before issuing sliceUnaligned.

Example fix

// before: position taken from a stale index entry
int pos = index.lookup(targetOffset).position;
return records.slice(pos, fetchSize);

// after: bound position against live size
int pos = Math.min(index.lookup(targetOffset).position, records.sizeInBytes());
return records.slice(pos, fetchSize);

int currentSize = records.sizeInBytes();
if (position > currentSize) {
    throw new IllegalArgumentException(
        "slice position " + position + " is past end " + currentSize);
}
FileRecords slice = records.slice(position, size);

When it happens

Trigger: Calling slice(position, size) or sliceUnaligned(position, size) where position > sizeInBytes(). Common during fetch handling when the highWatermark or lastStableOffset advances slower than the fetch offset the client requested, or during log truncation racing with an in-flight read.

Common situations: Consumer/replica fetch asks for bytes beyond the segment end after a truncation or before the leader has appended them. Also seen after manual segment manipulation (deleting bytes from a file without updating metadata) or when a Log offset-index points past the real file size.

Related errors

• Failed to read record batch at position {} from {}
• Invalid position: {} in read from {}
• Invalid size: {} in read from {}
• Attempt to truncate log segment {} to {} bytes failed, size
• Size of FileRecords %s has been truncated during write: old

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/3610ff4f0086c17b.json. Report an issue: GitHub.