apache/kafka · error · IllegalArgumentException

Invalid magic used in LegacyRecord: {}

Error message

Invalid magic used in LegacyRecord: {}

What it means

Thrown by LegacyRecord.recordOverhead() (and the parallel headerSize() at line 555) when computing the per-record overhead/header size for a magic value that is neither 0 (V0) nor 1 (V1). LegacyRecord only supports the two historical message formats, so any other byte (e.g. 2, the v2 default) is meaningless here and indicates the caller used the wrong record class.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/LegacyRecord.java:547

            Checksums.update(crc, key, size);
        }
        // update for the value
        if (value == null) {
            Checksums.updateInt(crc, -1);
        } else {
            int size = value.remaining();
            Checksums.updateInt(crc, size);
            Checksums.update(crc, value, size);
        }
        return crc.getValue();
    }

    static int recordOverhead(byte magic) {
        if (magic == 0)
            return RECORD_OVERHEAD_V0;
        else if (magic == 1)
            return RECORD_OVERHEAD_V1;
        throw new IllegalArgumentException("Invalid magic used in LegacyRecord: " + magic);
    }

    static int headerSize(byte magic) {
        if (magic == 0)
            return HEADER_SIZE_V0;
        else if (magic == 1)
            return HEADER_SIZE_V1;
        throw new IllegalArgumentException("Invalid magic used in LegacyRecord: " + magic);
    }

    private static int keyOffset(byte magic) {
        if (magic == 0)
            return KEY_OFFSET_V0;
        else if (magic == 1)
            return KEY_OFFSET_V1;
        throw new IllegalArgumentException("Invalid magic used in LegacyRecord: " + magic);
    }

View on GitHub (pinned to c31c9215e1)

Solutions

1. Route magic==2 (and above) through DefaultRecord / DefaultRecordBatch, not LegacyRecord.
2. Validate magic against RecordBatch.MAGIC_VALUE_V0 / MAGIC_VALUE_V1 before calling recordOverhead/headerSize.
3. Ensure any 'default magic' constant used in the call site is the legacy value when LegacyRecord is intended.

Example fix

// before
int overhead = LegacyRecord.recordOverhead(batchMagic); // batchMagic == 2
// after
if (batchMagic < RecordBatch.MAGIC_VALUE_V2) {
    int overhead = LegacyRecord.recordOverhead(batchMagic);
} else {
    int overhead = DefaultRecord.recordOverhead(...);
}

// LegacyRecord only supports magic 0 (v0) and 1 (v1).
if (magic != RecordBatch.MAGIC_VALUE_V0 && magic != RecordBatch.MAGIC_VALUE_V1) {
    throw new IllegalArgumentException(
        "Unsupported legacy magic " + magic + "; expected 0 or 1");
}

// Accept only the two legacy magic values before any LegacyRecord call.
static boolean isLegacyMagic(byte magic) {
    return magic == RecordBatch.MAGIC_VALUE_V0 || magic == RecordBatch.MAGIC_VALUE_V1;
}

try {
    int overhead = LegacyRecord.recordOverhead(magic);
} catch (IllegalArgumentException e) {
    if (e.getMessage().startsWith("Invalid magic used in LegacyRecord")) {
        // Upgrade to the v2 record batch path; legacy format cannot represent this magic.
        magic = RecordBatch.MAGIC_VALUE_V2;
    } else {
        throw e;
    }
}

When it happens

Trigger: Calling recordOverhead(magic) or headerSize(magic) with magic != 0 and magic != 1. Most often hit when a code path assumes the modern default magic value (2, RecordBatch.MAGIC_VALUE_V2) is passed through LegacyRecord instead of the newer DefaultRecord/DefaultRecordBatch APIs.

Common situations: Refactoring a record path from legacy to v2 and forgetting to switch from LegacyRecord to DefaultRecordBatch; defaulting magic to the current value (2) and feeding it into a legacy-only utility; reading a v2 batch through a legacy decoder.

Related errors

• Invalid message timestamp {}
• Timestamp type must be provided to compute attributes for me
• The target time for partition {} is {}. The target time cann
• The timeout cannot be negative.
• Topic partitions collection to assign to cannot be null

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/e9b5b80ef3bc82a2.json. Report an issue: GitHub.