apache/kafka · error · IllegalArgumentException

Invalid magic value {}

Error message

Invalid magic value {}

What it means

Thrown by LegacyRecord.write when the magic argument is neither MAGIC_VALUE_V0 (0) nor MAGIC_VALUE_V1 (1). LegacyRecord only supports message-format versions 0 and 1; magic 2+ belongs to the default RecordBatch implementation. This guard prevents silently writing a v2 payload through the v0/v1 wire format.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/LegacyRecord.java:447

                             long crc,
                             byte attributes,
                             long timestamp,
                             byte[] key,
                             byte[] value) throws IOException {
        write(out, magic, crc, attributes, timestamp, wrapNullable(key), wrapNullable(value));
    }

    // Write a record to the buffer, if the record's compression type is none, then
    // its value payload should be already compressed with the specified type
    private static void write(DataOutputStream out,
                              byte magic,
                              long crc,
                              byte attributes,
                              long timestamp,
                              ByteBuffer key,
                              ByteBuffer value) throws IOException {
        if (magic != RecordBatch.MAGIC_VALUE_V0 && magic != RecordBatch.MAGIC_VALUE_V1)
            throw new IllegalArgumentException("Invalid magic value " + magic);
        if (timestamp < 0 && timestamp != RecordBatch.NO_TIMESTAMP)
            throw new IllegalArgumentException("Invalid message timestamp " + timestamp);

        // write crc
        out.writeInt((int) (crc & 0xffffffffL));
        // write magic value
        out.writeByte(magic);
        // write attributes
        out.writeByte(attributes);

        // maybe write timestamp
        if (magic > RecordBatch.MAGIC_VALUE_V0)
            out.writeLong(timestamp);

        // write the key
        if (key == null) {
            out.writeInt(-1);
        } else {

View on GitHub (pinned to c31c9215e1)

Solutions

1. At the call site, branch on the magic byte: use LegacyRecord only for magic 0/1, and DefaultRecordBatch / DefaultRecord for magic 2+.
2. If you are upgrading a producer to v2, switch the writer class entirely — LegacyRecord is not the v2 path.
3. Add a precondition in your factory method so magic is validated once, at the boundary, rather than deep inside write().
4. Double-check log.message.format.version / message.format.version broker config — if it is 2 or higher, no LegacyRecord code path should be producing records.

Example fix

// before: LegacyRecord used for every magic, throws on v2
LegacyRecord.create(magic, ts, key, value);

// after: route by message-format version
if (magic < RecordBatch.MAGIC_VALUE_V2) {
    LegacyRecord.create(magic, ts, key, value);
} else {
    MemoryRecords.withRecords(magic, compression, new SimpleRecord(ts, key, value));
}

if (magic != RecordBatch.MAGIC_VALUE_V0 && magic != RecordBatch.MAGIC_VALUE_V1) {
    throw new IllegalArgumentException(
        "LegacyRecord magic must be 0 or 1, got " + magic);
}
LegacyRecord.create(magic, timestamp, key, value);

static boolean isLegacyMagic(byte magic) {
    return magic == RecordBatch.MAGIC_VALUE_V0
        || magic == RecordBatch.MAGIC_VALUE_V1;
}

// usage:
if (!isLegacyMagic(magic)) {
    throw new IllegalArgumentException("Not a legacy magic: " + magic);
}

When it happens

Trigger: Calling LegacyRecord.write / LegacyRecord.create with a magic byte other than 0 or 1. Reachable from producer code that constructs legacy records, from converter/upgrade tooling, or from any code path that picks LegacyRecord for a format it does not support.

Common situations: A custom serializer or test helper hard-coded magic=2 but still routes through LegacyRecord; mixing up RecordBatch magic constants with legacy constants; or message.format.version interop code that did not gate on the version before choosing the writer.

Related errors

• Invalid magic value {}
• Version %d is unsupported
• Invalid wrapper magic found in legacy deep record iterator {
• Compressed message magic {} does not match wrapper magic {}
• Found invalid compressed record set with no inner records

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/5df898fbb399e9da.json. Report an issue: GitHub.