What causes "Invalid magic found in record: {}" in apache/kafka?

Confirm the segment is a real Kafka .log file and that you are not reading past the last valid batch (check the segment's recovered offset / truncated length).

apache/kafka · critical · CorruptRecordException

Invalid magic found in record: {}

Error message

Invalid magic found in record: {}

What it means

Thrown by ByteBufferLogInputStream.nextBatchSize after reading the magic byte (buffer position + MAGIC_OFFSET) and finding a value that is negative or greater than RecordBatch.CURRENT_MAGIC_VALUE. The magic identifies the record-batch format version; only 0, 1, and 2 are defined. Anything else means the header is garbage, so the stream raises CorruptRecordException rather than guessing a format.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/ByteBufferLogInputStream.java:84

    Integer nextBatchSize() throws CorruptRecordException {
        int remaining = buffer.remaining();
        if (remaining < LOG_OVERHEAD)
            return null;
        int recordSize = buffer.getInt(buffer.position() + SIZE_OFFSET);
        // V0 has the smallest overhead, stricter checking is done later
        if (recordSize < LegacyRecord.RECORD_OVERHEAD_V0)
            throw new CorruptRecordException(String.format("Record size %d is less than the minimum record overhead (%d)",
                    recordSize, LegacyRecord.RECORD_OVERHEAD_V0));
        if (recordSize > maxMessageSize)
            throw new CorruptRecordException(String.format("Record size %d exceeds the largest allowable message size (%d).",
                    recordSize, maxMessageSize));

        if (remaining < HEADER_SIZE_UP_TO_MAGIC)
            return null;

        byte magic = buffer.get(buffer.position() + MAGIC_OFFSET);
        if (magic < 0 || magic > RecordBatch.CURRENT_MAGIC_VALUE)
            throw new CorruptRecordException("Invalid magic found in record: " + magic);

        return recordSize + LOG_OVERHEAD;
    }
}

View on GitHub (pinned to c31c9215e1)

Solutions

Confirm the segment is a real Kafka .log file and that you are not reading past the last valid batch (check the segment's recovered offset / truncated length).
Inspect with kafka-dump-log --files <segment.log> to locate the bad batch.
If genuinely corrupt, restore the segment from another replica or delete it so the broker truncates to the last valid offset.

Defensive patterns

Strategy: try-catch

Validate before calling

// Magic byte is read from a potentially corrupt stream; pre-validation means
// parsing it yourself first:
byte magic = buffer.get(buffer.position() + MAGIC_OFFSET);
if (magic < 0 || magic > RecordBatch.CURRENT_MAGIC_VALUE) {
    // unsupported/corrupt magic; stop reading this batch
}

Try / catch

try {
    batch = stream.nextBatch();
} catch (org.apache.kafka.common.errors.CorruptRecordException e) {
    // unknown magic value; likely cross-version corruption or bit-rot.
    // Skip and continue, or surface a hard failure depending on durability policy.
}

Prevention

Only supported magic values are 0, 1, 2; anything else indicates corruption or a truncated/garbled segment.
Avoid mixing record formats in a single log segment across uncoordinated writes.
When upgrading message.format.version, do a controlled roll rather than interleaving versions.

When it happens

Trigger: nextBatchSize reads magic < 0 || magic > RecordBatch.CURRENT_MAGIC_VALUE (currently 2). Encountered by any code scanning a byte buffer as log batches — fetch handlers, replication, log recovery, kafka-dump-log.

Common situations: Truncated or partially written segment (magic byte not yet flushed); disk/filesystem corruption bit-flipping the magic; reading a non-Kafka file as a log; tail of a segment that was pre-allocated and not fully populated but read past its valid data.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/c2fc2204be706f70.json. Report an issue: GitHub.