apache/kafka · critical · KafkaException

Size of FileRecords %s has been truncated during write: old

Error message

Size of FileRecords %s has been truncated during write: old size %d, new size %d

What it means

Thrown by FileRecords.writeTo when the channel's current size (min'd with end, minus start) is smaller than the cached sizeInBytes at the start of the call. That means the underlying file shrank between the size snapshot and the transferFrom, so the bytes the caller expects to send no longer exist. This is a defensive check against concurrent truncation during a network send.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:295

     */
    public int truncateTo(int targetSize) throws IOException {
        int originalSize = sizeInBytes();
        if (targetSize > originalSize || targetSize < 0)
            throw new KafkaException("Attempt to truncate log segment " + file + " to " + targetSize + " bytes failed, " +
                    " size of this log segment is " + originalSize + " bytes.");
        if (targetSize < (int) channel.size()) {
            channel.truncate(targetSize);
            size.set(targetSize);
        }
        return originalSize - targetSize;
    }

    @Override
    public int writeTo(TransferableChannel destChannel, int offset, int length) throws IOException {
        long newSize = Math.min(channel.size(), end) - start;
        int oldSize = sizeInBytes();
        if (newSize < oldSize)
            throw new KafkaException(String.format(
                    "Size of FileRecords %s has been truncated during write: old size %d, new size %d",
                    file.getAbsolutePath(), oldSize, newSize));

        long position = start + offset;
        int count = Math.min(length, oldSize - offset);
        // safe to cast to int since `count` is an int
        return (int) destChannel.transferFrom(channel, position, count);
    }

    /**
     * Search forward for the file position of the message batch whose last offset that is greater
     * than or equal to the target offset. If no such batch is found, return null.
     *
     * @param targetOffset The offset to search for.
     * @param startingPosition The starting position in the file to begin searching from.
     * @return the batch's base offset, its physical position, and its size (including log overhead)
     */
    public LogOffsetPosition searchForOffsetFromPosition(long targetOffset, int startingPosition) {

View on GitHub (pinned to c31c9215e1)

Solutions

1. Serialize fetch/send paths against the log lock so truncation cannot race with writeTo.
2. On a leader epoch change, cancel in-flight fetch sessions before truncating so writeTo is not mid-transfer.
3. Investigate disk/filesystem health if the channel size changed without an explicit truncate (possible ENOSPC or fs bug).
4. If using tiered storage, ensure the local segment is not evicted under an active fetch transfer.

Example fix

// before: writeTo races with concurrent truncateTo on another thread
executors.submit(() -> segment.writeTo(socketChannel, offset, length));
// ... elsewhere: segment.truncateTo(target);

// after: guard sends with the log lock so truncation cannot interleave
logLock.lock();
try {
    segment.writeTo(socketChannel, offset, length);
} finally {
    logLock.unlock();
}

try {
    fileRecords.writeTo(destChannel, offset, length);
} catch (KafkaException e) {
    // The backing file shrank under us (concurrent truncate or external process).
    // Reopen the segment or abort the replication/fetch; do not silently retry the same transfer.
    log.error("FileRecords {} truncated during write (was {}, now smaller)",
              fileRecords.file().getAbsolutePath(), sizeBefore, e);
    throw e;
}

When it happens

Trigger: Calling FileRecords.writeTo(destChannel, offset, length) — used by fetch response transfer to a SocketChannel / TransferableChannel — while another thread truncates the segment (Log.truncate, recovery, leader epoch change). newSize computed from channel.size() comes back smaller than oldSize from sizeInBytes().

Common situations: A leader demotion or partition reassignment triggers truncation concurrently with an in-flight produce/fetch response that is streaming the segment to a replica or client. Also seen if an operator runs offline truncation while the broker is serving fetches, or after a disk error caused the file to shrink.

Related errors

• Failed to read record batch at position {} from {}
• Slice from position {} exceeds end position of {}
• NetworkClient is no longer active, state is {state}
• Client was shutdown before response was read
• KafkaConsumer is not safe for multi-threaded access. current

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/3586f7bd651be573.json. Report an issue: GitHub.