apache/kafka · error · InvalidRecordException

Invalid negative header key size {}

Error message

Invalid negative header key size {}

What it means

Thrown on the InputStream partial-read path when the varint decoded for an individual header's key length is negative. Header keys are UTF-8 strings whose length is encoded as a non-negative varint; a negative value means the byte sequence is corrupt and the parser refuses to skip a negative number of bytes.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/DefaultRecord.java:408

                DefaultRecordBatch.incrementSequence(baseSequence, offsetDelta) :
                RecordBatch.NO_SEQUENCE;

            // skip key
            int keySize = ByteUtils.readVarint(input);
            skipBytes(input, keySize);

            // skip value
            int valueSize = ByteUtils.readVarint(input);
            skipBytes(input, valueSize);

            // skip header
            int numHeaders = ByteUtils.readVarint(input);
            if (numHeaders < 0)
                throw new InvalidRecordException("Found invalid number of record headers " + numHeaders);
            for (int i = 0; i < numHeaders; i++) {
                int headerKeySize = ByteUtils.readVarint(input);
                if (headerKeySize < 0)
                    throw new InvalidRecordException("Invalid negative header key size " + headerKeySize);
                skipBytes(input, headerKeySize);

                // headerValueSize
                int headerValueSize = ByteUtils.readVarint(input);
                skipBytes(input, headerValueSize);
            }

            return new PartialDefaultRecord(sizeInBytes, attributes, offset, timestamp, sequence, keySize, valueSize);
        } catch (BufferUnderflowException | IllegalArgumentException e) {
            throw new InvalidRecordException("Found invalid record structure", e);
        }
    }


    /**
     * Skips over and discards exactly {@code bytesToSkip} bytes from the input stream.
     *
     * We require a loop over {@link InputStream#skip(long)} because it is possible for InputStream to skip smaller

View on GitHub (pinned to c31c9215e1)

Solutions

  1. Dump the failing batch with kafka-dump-log --print-data-log and locate the record whose header section is malformed.
  2. Audit the producer's header construction; ensure every header has a non-null key (DefaultRecord.writeTo rejects null keys at line 207) and uses the standard Header/RecordHeader type.
  3. Validate the batch CRC; if it fails, treat the record as corrupt on disk and recover from a replica.
  4. Round-trip the same header set through DefaultRecord.writeTo in a producer-side test to confirm the encoding matches what the consumer expects.
Defensive patterns

Strategy: try-catch

Validate before calling

// headerKeySize is a varint decoded inside the parser. If you decode it
// yourself when walking headers:
int headerKeySize = ByteUtils.readVarint(input);
if (headerKeySize < 0) { /* malformed header key length */ }

Type guard

private static boolean isValidHeaderKeySize(int headerKeySize) {
    return headerKeySize >= 0;
}

Try / catch

try {
    PartialDefaultRecord r = DefaultRecord.readPartiallyFrom(input, baseOffset, baseTimestamp, baseSequence, logAppendTime);
} catch (InvalidRecordException | IOException e) {
    LOG.warn("Negative header key size near offset {}", baseOffset, e);
}

Prevention

When it happens

Trigger: Raised at DefaultRecord.java:407-408 inside the header-skipping loop of readPartiallyFrom when ByteUtils.readVarint(input) at line 406 returns a value < 0. Triggered by a malformed header section in a record being stream-scanned, typically because the bytes that should encode headerKeySize decode as -1 or another negative varint.

Common situations: Producer that wrote headers with a custom or buggy serializer (e.g. emitting a null-marker varint where a key-length was expected), on-disk corruption of the header region of a record, or a third-party client that encodes header key lengths differently from the Apache Kafka spec.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/9c0bbd237c336178.json. Report an issue: GitHub.