apache/kafka · error · InvalidRecordException

Invalid version found for end transaction marker: {}. May in

Error message

Invalid version found for end transaction marker: {}. May indicate data corruption

What it means

Thrown by EndTransactionMarker.deserializeValue() when the version prefix read from the marker value is below EndTxnMarker.LOWEST_SUPPORTED_VERSION. The version field is the first short in the value ByteBuffer; an out-of-range low version means the bytes do not represent a valid end-txn marker value, which the message explicitly flags as a possible sign of data corruption. It is an InvalidRecordException so it is treated as a corrupt-record condition by the broker/consumer.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/record/internal/EndTransactionMarker.java:93

    }

    private static void ensureTransactionMarkerControlType(ControlRecordType type) {
        if (type != ControlRecordType.COMMIT && type != ControlRecordType.ABORT)
            throw new IllegalArgumentException("Invalid control record type for end transaction marker " + type);
    }

    public static EndTransactionMarker deserialize(Record record) {
        ControlRecordType type = ControlRecordType.parse(record.key());
        return deserializeValue(type, record.value());
    }

    // Visible for testing
    static EndTransactionMarker deserializeValue(ControlRecordType type, ByteBuffer value) {
        ensureTransactionMarkerControlType(type);

        short version = value.getShort();
        if (version < EndTxnMarker.LOWEST_SUPPORTED_VERSION)
            throw new InvalidRecordException("Invalid version found for end transaction marker: " + version +
                    ". May indicate data corruption");

        if (version > EndTxnMarker.HIGHEST_SUPPORTED_VERSION) {
            log.debug("Received end transaction marker value version {}. Parsing as version {}", version,
                    EndTxnMarker.HIGHEST_SUPPORTED_VERSION);
            version = EndTxnMarker.HIGHEST_SUPPORTED_VERSION;
        }
        EndTxnMarker marker = new EndTxnMarker(new ByteBufferAccessor(value), version);
        return new EndTransactionMarker(type, marker.coordinatorEpoch());
    }

    public int endTxnMarkerValueSize() {
        return DefaultRecord.sizeInBytes(0, 0L,
                type.controlRecordKeySize(),
                buffer.remaining(),
                Record.EMPTY_HEADERS);
    }

View on GitHub (pinned to c31c9215e1)

Solutions

1. Dump the affected control batch with kafka-dump-log.sh --deep-iteration --print-data-log to inspect the raw marker value bytes and confirm the version short is invalid.
2. Recover a clean copy of the segment from an in-sync replica; for RF=1, truncate past the corrupt marker.
3. If a tool/serializer rewrote the value, fix it to preserve the version-prefixed ByteBuffer produced by EndTransactionMarker.serializeValue.
4. Add broker-side CRC validation logging (log.message.format.version, message.timestamp.type) and watch CorruptRecordException rates to catch the underlying writer bug.

Example fix

// before: writing a control record value without the version prefix
buffer.putShort((short) 0).put(coordinatorEpoch);
// after: use the provided serializer so the version prefix is always correct
EndTransactionMarker m = new EndTransactionMarker(ControlRecordType.COMMIT, epoch);
ByteBuffer value = m.serializeValue();

try {
  EndTransactionMarker m = EndTransactionMarker.deserialize(record);
} catch (org.apache.kafka.common.InvalidRecordException e) {
  // Marker version below LOWEST_SUPPORTED_VERSION: the control-record value is corrupt.
  // Re-fetch the control record or treat the transaction as indeterminate.
  log.warn("Corrupt end-txn marker at offset {}: {}", record.offset(), e.getMessage());
}

When it happens

Trigger: Calling EndTransactionMarker.deserialize(record) where record.value() begins with a short less than EndTxnMarker.LOWEST_SUPPORTED_VERSION. Happens when the value ByteBuffer is too short, misaligned, or contains garbage where the version short is expected. Triggered during transaction marker processing in the transaction coordinator, log validation, or consumer reading control batches.

Common situations: Disk/log corruption that altered the marker value bytes, a segment partially overwritten, a manual or buggy tool that rewrote control records without preserving the version-prefixed value format, a wire-level or serialization bug that truncated the value below 2 bytes, or a client build so old it predates the versioned EndTxnMarker schema. Note: versions above HIGHEST_SUPPORTED_VERSION are tolerated (downgraded with a debug log), so this only fires for too-low or garbage versions.

Related errors

• Invalid control record type for end transaction marker {}
• Incorrect declared batch size, records still remaining in fi
• Found record size %d smaller than minimum record overhead (%
• Record batch for partition {} at offset {} is invalid, cause
• Encountered corrupt message when fetching offset {} for topi

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/9c29a117102b7daa.json. Report an issue: GitHub.