{"id":"6323456d112eb4b6","repo":"apache/kafka","slug":"found-record-size-d-smaller-than-minimum-record-o","errorCode":null,"errorMessage":"Found record size %d smaller than minimum record overhead (%d) in file %s.","messagePattern":"Found record size (.+?) smaller than minimum record overhead \\((.+?)\\) in file (.+?)\\.","errorType":"exception","errorClass":"CorruptRecordException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/FileLogInputStream.java","lineNumber":78,"sourceCode":"        this.end = end;\n    }\n\n    @Override\n    public FileChannelRecordBatch nextBatch() throws IOException {\n        FileChannel channel = fileRecords.channel();\n        if (position >= end - HEADER_SIZE_UP_TO_MAGIC)\n            return null;\n\n        logHeaderBuffer.rewind();\n        Utils.readFullyOrFail(channel, logHeaderBuffer, position, \"log header\");\n\n        logHeaderBuffer.rewind();\n        long offset = logHeaderBuffer.getLong(OFFSET_OFFSET);\n        int size = logHeaderBuffer.getInt(SIZE_OFFSET);\n\n        // V0 has the smallest overhead, stricter checking is done later\n        if (size < LegacyRecord.RECORD_OVERHEAD_V0)\n            throw new CorruptRecordException(String.format(\"Found record size %d smaller than minimum record \" +\n                            \"overhead (%d) in file %s.\", size, LegacyRecord.RECORD_OVERHEAD_V0, fileRecords.file()));\n\n        if (position > end - LOG_OVERHEAD - size)\n            return null;\n\n        byte magic = logHeaderBuffer.get(MAGIC_OFFSET);\n        final FileChannelRecordBatch batch;\n\n        if (magic < RecordBatch.MAGIC_VALUE_V2)\n            batch = new LegacyFileChannelRecordBatch(offset, magic, fileRecords, position, size);\n        else\n            batch = new DefaultFileChannelRecordBatch(offset, magic, fileRecords, position, size);\n\n        position += batch.sizeInBytes();\n        return batch;\n    }\n\n    /**","sourceCodeStart":60,"sourceCodeEnd":96,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/FileLogInputStream.java#L60-L96","documentation":"Thrown by FileLogInputStream.nextBatch() when the size field read from a log header is smaller than LegacyRecord.RECORD_OVERHEAD_V0 (the smallest valid record overhead across all magic versions). This is the earliest sanity check on the on-disk header: a size that small cannot contain any valid record, so the header is treated as corrupt. It is a CorruptRecordException (subclass of InvalidRecordException) so the broker/consumer can apply corrupt-record handling such as skipping or quarantining.","triggerScenarios":"FileLogInputStream.nextBatch() reads the 12-byte-ish header from the FileChannel at the current position, extracts the size int at SIZE_OFFSET, and if size < LegacyRecord.RECORD_OVERHEAD_V0 throws this. Produced whenever any code iterates a FileRecords-backed segment via nextBatch (log scanning, recovery, dump-log, consumer reading from file).","commonSituations":"A log segment whose tail contains leftover/garbage bytes (e.g. preallocation zeros interpreted as a header after truncation), a torn write leaving a partial header at the active segment end, manual tampering with .log files, an OS-level truncation that left non-record-aligned bytes, or a recovery tool scanning into the preallocated region past the last valid batch.","solutions":["Confirm the active segment's recovery point / last stable offset; the header may simply be in the preallocated tail and should not be scanned (ensure nextBatch is bounded by the segment end, not file.length).","Run kafka-dump-log.sh --index-sanity-check true on the segment to locate the first corrupt header offset.","If genuinely corrupt, truncate the segment to the last valid offset (recovery-point or from the .index/.timeindex) and let the broker re-recover.","Verify that any custom FileRecords reader respects the configured end bound passed to FileLogInputStream rather than reading to EOF."],"exampleFix":"// before: scanning the whole file including preallocated tail\nFileLogInputStream in = new FileLogInputStream(fileRecords, 0, Integer.MAX_VALUE);\n// after: bound iteration to the segment's known valid end\nFileLogInputStream in = new FileLogInputStream(fileRecords, 0, fileRecords.sizeInBytes());","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  FileChannelRecordBatch b = inputStream.nextBatch();\n} catch (org.apache.kafka.common.errors.CorruptRecordException e) {\n  // Header declared a size < LegacyRecord.RECORD_OVERHEAD_V0 — header bytes are garbage.\n  // Skip forward by LOG_OVERHEAD and continue, or halt iteration on the segment.\n  log.warn(\"Corrupt log header: {}\", e.getMessage());\n}","preventionTips":["CorruptRecordException here means the size field in the log header is implausible — the segment header itself is damaged, retrying the same position will not help.","When implementing recovery, advance the read position past the bad header (by LOG_OVERHEAD) and retry nextBatch() rather than re-reading the same offset.","Validate segments offline with kafka-dump-log before pointing consumers at them.","Prevent at the source: ensure brokers flush and fsync on append and avoid hard kills during log rolling.","For v0/v1 (legacy magic) records the minimum overhead is RECORD_OVERHEAD_V0; for v2 batches this path is not hit — migrating to v2 (magic >= 2) avoids this legacy header check."],"tags":["kafka","records","file-records","corruption","log-segment","broker"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}