apache/kafka · error · CorruptRecordException

Found record size %d smaller than minimum record overhead (%

Error message

Found record size %d smaller than minimum record overhead (%d) in file %s.

What it means

Thrown by FileLogInputStream.nextBatch() when the size field read from a log header is smaller than LegacyRecord.RECORD_OVERHEAD_V0 (the smallest valid record overhead across all magic versions). This is the earliest sanity check on the on-disk header: a size that small cannot contain any valid record, so the header is treated as corrupt. It is a CorruptRecordException (subclass of InvalidRecordException) so the broker/consumer can apply corrupt-record handling such as skipping or quarantining.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/FileLogInputStream.java:78

        this.end = end;
    }

    @Override
    public FileChannelRecordBatch nextBatch() throws IOException {
        FileChannel channel = fileRecords.channel();
        if (position >= end - HEADER_SIZE_UP_TO_MAGIC)
            return null;

        logHeaderBuffer.rewind();
        Utils.readFullyOrFail(channel, logHeaderBuffer, position, "log header");

        logHeaderBuffer.rewind();
        long offset = logHeaderBuffer.getLong(OFFSET_OFFSET);
        int size = logHeaderBuffer.getInt(SIZE_OFFSET);

        // V0 has the smallest overhead, stricter checking is done later
        if (size < LegacyRecord.RECORD_OVERHEAD_V0)
            throw new CorruptRecordException(String.format("Found record size %d smaller than minimum record " +
                            "overhead (%d) in file %s.", size, LegacyRecord.RECORD_OVERHEAD_V0, fileRecords.file()));

        if (position > end - LOG_OVERHEAD - size)
            return null;

        byte magic = logHeaderBuffer.get(MAGIC_OFFSET);
        final FileChannelRecordBatch batch;

        if (magic < RecordBatch.MAGIC_VALUE_V2)
            batch = new LegacyFileChannelRecordBatch(offset, magic, fileRecords, position, size);
        else
            batch = new DefaultFileChannelRecordBatch(offset, magic, fileRecords, position, size);

        position += batch.sizeInBytes();
        return batch;
    }

    /**

View on GitHub (pinned to c31c9215e1)

Solutions

Confirm the active segment's recovery point / last stable offset; the header may simply be in the preallocated tail and should not be scanned (ensure nextBatch is bounded by the segment end, not file.length).
Run kafka-dump-log.sh --index-sanity-check true on the segment to locate the first corrupt header offset.
If genuinely corrupt, truncate the segment to the last valid offset (recovery-point or from the .index/.timeindex) and let the broker re-recover.
Verify that any custom FileRecords reader respects the configured end bound passed to FileLogInputStream rather than reading to EOF.

Example fix

// before: scanning the whole file including preallocated tail
FileLogInputStream in = new FileLogInputStream(fileRecords, 0, Integer.MAX_VALUE);
// after: bound iteration to the segment's known valid end
FileLogInputStream in = new FileLogInputStream(fileRecords, 0, fileRecords.sizeInBytes());

Defensive patterns

Strategy: try-catch

Try / catch

try {
  FileChannelRecordBatch b = inputStream.nextBatch();
} catch (org.apache.kafka.common.errors.CorruptRecordException e) {
  // Header declared a size < LegacyRecord.RECORD_OVERHEAD_V0 — header bytes are garbage.
  // Skip forward by LOG_OVERHEAD and continue, or halt iteration on the segment.
  log.warn("Corrupt log header: {}", e.getMessage());
}

Prevention

CorruptRecordException here means the size field in the log header is implausible — the segment header itself is damaged, retrying the same position will not help.
When implementing recovery, advance the read position past the bad header (by LOG_OVERHEAD) and retry nextBatch() rather than re-reading the same offset.
Validate segments offline with kafka-dump-log before pointing consumers at them.
Prevent at the source: ensure brokers flush and fsync on append and avoid hard kills during log rolling.
For v0/v1 (legacy magic) records the minimum overhead is RECORD_OVERHEAD_V0; for v2 batches this path is not hit — migrating to v2 (magic >= 2) avoids this legacy header check.

When it happens

Trigger: FileLogInputStream.nextBatch() reads the 12-byte-ish header from the FileChannel at the current position, extracts the size int at SIZE_OFFSET, and if size < LegacyRecord.RECORD_OVERHEAD_V0 throws this. Produced whenever any code iterates a FileRecords-backed segment via nextBatch (log scanning, recovery, dump-log, consumer reading from file).

Common situations: A log segment whose tail contains leftover/garbage bytes (e.g. preallocation zeros interpreted as a header after truncation), a torn write leaving a partial header at the active segment end, manual tampering with .log files, an OS-level truncation that left non-record-aligned bytes, or a recovery tool scanning into the preallocated region past the last valid batch.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/6323456d112eb4b6.json. Report an issue: GitHub.