{"id":"427b0c06804db8ee","repo":"apache/kafka","slug":"record-batch-is-corrupt-the-size-is-smaller-th","errorCode":null,"errorMessage":"Record batch is corrupt (the size {} is smaller than the minimum allowed overhead {})","messagePattern":"Record batch is corrupt \\(the size (.+?) is smaller than the minimum allowed overhead (.+?)\\)","errorType":"exception","errorClass":"CorruptRecordException","httpStatus":null,"severity":"critical","filePath":"clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecordBatch.java","lineNumber":153,"sourceCode":"    private static final int CONTROL_FLAG_MASK = 0x20;\n    private static final byte DELETE_HORIZON_FLAG_MASK = 0x40;\n    private static final byte TIMESTAMP_TYPE_MASK = 0x08;\n\n    private final ByteBuffer buffer;\n\n    DefaultRecordBatch(ByteBuffer buffer) {\n        this.buffer = buffer;\n    }\n\n    @Override\n    public byte magic() {\n        return buffer.get(MAGIC_OFFSET);\n    }\n\n    @Override\n    public void ensureValid() {\n        if (sizeInBytes() < RECORD_BATCH_OVERHEAD)\n            throw new CorruptRecordException(\"Record batch is corrupt (the size \" + sizeInBytes() +\n                    \" is smaller than the minimum allowed overhead \" + RECORD_BATCH_OVERHEAD + \")\");\n\n        if (!isValid())\n            throw new CorruptRecordException(\"Record is corrupt (stored crc = \" + checksum()\n                    + \", computed crc = \" + computeChecksum() + \")\");\n    }\n\n    /**\n     * Gets the base timestamp of the batch which is used to calculate the record timestamps from the deltas.\n     *\n     * @return The base timestamp\n     */\n    public long baseTimestamp() {\n        return buffer.getLong(BASE_TIMESTAMP_OFFSET);\n    }\n\n    @Override\n    public long maxTimestamp() {","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecordBatch.java#L135-L171","documentation":"Thrown by DefaultRecordBatch.ensureValid when the batch's total sizeInBytes is smaller than RECORD_BATCH_OVERHEAD (the fixed header bytes before any records: base offset, length, leader epoch, magic, CRC, producer id/epoch, sequence, timestamps, etc.). A batch that short cannot contain a valid header, so it is treated as structural corruption. CorruptRecordException.","triggerScenarios":"Produced when a buffer sliced shorter than the batch overhead is presented to a DefaultRecordBatch and ensureValid() is called (broker append path, consumer validation, log recovery). Often follows an undersized ByteBuffer allocation, a partial read off disk/network, or a torn write leaving a sub-overhead remnant at a segment tail.","commonSituations":"Truncated log segment after an unclean broker shutdown; undersized buffer in custom code that reads exactly fetch.minBytes / a miscomputed length; inter-broker or client fetch of a segment whose last batch was only partially flushed; OS page-cache vs disk inconsistency after a crash.","solutions":["Run kafka-dump-log --files <segment-log> to confirm which batch/offset is short and whether the segment tail is truncated.","If the segment is torn at the tail, delete the truncated segment (or use kafka-storage/kafka-clean tooling) so recovery skips the remnant; the broker will truncate to the last complete batch.","Ensure client/broker code reads whole batches via MemoryRecords rather than slicing at computed offsets; never allocate a batch buffer smaller than RECORD_BATCH_OVERHEAD.","Verify broker log.flush / log.segment configuration is not producing partial flushes; check disk health (df, dmesg) for I/O errors."],"exampleFix":"// before: slicing a buffer to a length smaller than the batch header\nDefaultRecordBatch b = new DefaultRecordBatch(buf.slice(0, partialLen));\nb.ensureValid();\n\n// after: only treat complete batches as batches\nif (readable >= DefaultRecordBatch.RECORD_BATCH_OVERHEAD) {\n    DefaultRecordBatch b = new DefaultRecordBatch(buf);\n    b.ensureValid();\n}","handlingStrategy":"try-catch","validationCode":"// Thrown from DefaultRecordBatch.ensureValid() while READING a buffer; the user does not\n// supply 'size' as an argument. The only pre-call defense is to reject undersized buffers\n// before handing them to the batch reader:\nvoid checkBatch(ByteBuffer b) {\n    if (b == null || b.remaining() < DefaultRecordBatch.RECORD_BATCH_OVERHEAD) {\n        throw new CorruptRecordException(\"buffer too small to be a batch: \" + (b == null ? -1 : b.remaining()));\n    }\n}","typeGuard":"// Narrow 'raw bytes' to a validated batch wrapper before use:\nstatic final class VerifiedBatch {\n    private final DefaultRecordBatch batch;\n    private VerifiedBatch(DefaultRecordBatch b) { this.batch = b; }\n    static VerifiedBatch of(ByteBuffer buf) {\n        if (buf.remaining() < DefaultRecordBatch.RECORD_BATCH_OVERHEAD)\n            throw new CorruptRecordException(\"undersized\");\n        return new VerifiedBatch(new DefaultRecordBatch(buf));\n    }\n    DefaultRecordBatch get() { return batch; }\n}","tryCatchPattern":"// On consume, isolate the corrupt batch and continue:\ntry {\n    batch.ensureValid();\n    process(batch);\n} catch (CorruptRecordException e) { // size < RECORD_BATCH_OVERHEAD\n    log.warn(\"Truncated/corrupt batch at {} offset {}, skipping\", partition, offset, e);\n    consumer.seek(partition, offset + 1);\n}","preventionTips":["This almost always indicates disk/network truncation — check broker logs for segment corruption or under-replicated partitions.","If you read raw files (kafka-storage tool, log inspector), always size-check against RECORD_BATCH_OVERHEAD before constructing DefaultRecordBatch.","Enable broker-side log.recovery.enable on corrupted segments so bad batches are quarantined, not served to consumers."],"tags":["kafka","record-format","broker","corruption","log-recovery"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}