apache/kafka · error · KafkaException

Failed to read record batch at position {} from {}

Error message

Failed to read record batch at position {} from {}

What it means

Thrown by FileChannelRecordBatch.writeTo() when Utils.readFully() fails with IOException while copying the batch's bytes from the underlying FileChannel into a target ByteBuffer. This path is used when materializing a batch for transmission (e.g. fetch response assembly, replication) without going through the full in-memory load. It wraps the IOException as KafkaException because the failure happened during a channel read that the caller has no direct handle to.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/FileLogInputStream.java:186

        public void ensureValid() {
            loadFullBatch().ensureValid();
        }

        @Override
        public int sizeInBytes() {
            return LOG_OVERHEAD + batchSize;
        }

        @Override
        public void writeTo(ByteBuffer buffer) {
            FileChannel channel = fileRecords.channel();
            try {
                int limit = buffer.limit();
                buffer.limit(buffer.position() + sizeInBytes());
                Utils.readFully(channel, buffer, position);
                buffer.limit(limit);
            } catch (IOException e) {
                throw new KafkaException("Failed to read record batch at position " + position + " from " + fileRecords, e);
            }
        }

        protected abstract RecordBatch toMemoryRecordBatch(ByteBuffer buffer);

        protected abstract int headerSize();

        protected RecordBatch loadFullBatch() {
            if (fullBatch == null) {
                batchHeader = null;
                fullBatch = loadBatchWithSize(sizeInBytes(), "full record batch");
            }
            return fullBatch;
        }

        protected RecordBatch loadBatchHeader() {
            if (fullBatch != null)
                return fullBatch;

View on GitHub (pinned to c31c9215e1)

Solutions

1. Check that the FileRecords/segment is still open and not deleted; do not hold FileChannelRecordBatch references after the source segment may have been rolled/deleted by retention.
2. Inspect the broker log for concurrent segment deletion / log rolling around the time of the failure and correlate with retention/log.roll.ms settings.
3. Verify disk health (dmesg, SMART, FS errors) if the channel read fails with a low-level I/O error.
4. Re-fetch the batch from an in-sync replica or trigger a leader election to recover a clean copy for the consumer.

Example fix

// before: holding a batch past its segment lifetime
FileChannelRecordBatch b = input.nextBatch();
// ... segment rolled/deleted by retention ...
b.writeTo(out); // IOException
// after: copy bytes out while the segment is still open
b.writeTo(out);
// then drop the reference; never retain across retention operations

try {
  batch.writeTo(buffer);
} catch (org.apache.kafka.common.KafkaException e) {
  // writeTo() does Utils.readFully(channel, ...) — IOException means the FileChannel read failed.
  java.io.IOException cause = (java.io.IOException) e.getCause();
  log.warn("Failed to read batch at position {} from {}: {}", batch.position(), batch, cause.getMessage());
}

When it happens

Trigger: Calling batch.writeTo(buffer) on a FileChannelRecordBatch (obtained from FileLogInputStream.nextBatch) where Utils.readFully(channel, buffer, position) raises IOException. The read covers sizeInBytes() bytes (LOG_OVERHEAD + batchSize) starting at the batch's recorded position. Common in fetch request handling, replication fetchers, and mirror makers.

Common situations: The underlying FileChannel/FileRecords was closed before the batch was written out, the segment file was deleted (log retention/roll) while a reference to a batch was still held, a disk/FS error during the read, or the batch's recorded position/size no longer points at valid data (e.g. segment was truncated underneath the holder). Also seen when a stale batch reference is retained past its segment's lifetime.

Related errors

• Failed to load record batch at position {} from {}
• Failed to decompress record stream
• Error checking for remaining bytes after reading batch
• Found record size %d smaller than minimum record overhead (%
• Slice from position {} exceeds end position of {}

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/5dc69a16e1ae9a03.json. Report an issue: GitHub.