apache/kafka · error · SchemaException

Buffer underflow while parsing consumer protocol's header

Error message

Buffer underflow while parsing consumer protocol's header

What it means

Thrown by ConsumerProtocol.deserializeVersion() when the supplied ByteBuffer has fewer than 2 bytes remaining, so the version-prefix short cannot be read. Kafka's consumer protocol wire payload begins with a 2-byte version header; if the buffer is empty, truncated, or positioned past its data, deserialization cannot even begin. The original BufferUnderflowException is attached as the cause.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerProtocol.java:64

    static {
        // Safety check to ensure that both parts of the consumer protocol remain in sync.
        if (ConsumerProtocolSubscription.LOWEST_SUPPORTED_VERSION
                != ConsumerProtocolAssignment.LOWEST_SUPPORTED_VERSION)
            throw new IllegalStateException("Subscription and Assignment schemas must have the " +
                "same lowest version");

        if (ConsumerProtocolSubscription.HIGHEST_SUPPORTED_VERSION
                != ConsumerProtocolAssignment.HIGHEST_SUPPORTED_VERSION)
            throw new IllegalStateException("Subscription and Assignment schemas must have the " +
                "same highest version");
    }

    public static short deserializeVersion(final ByteBuffer buffer) {
        try {
            return buffer.getShort();
        } catch (BufferUnderflowException e) {
            throw new SchemaException("Buffer underflow while parsing consumer protocol's header", e);
        }
    }

    public static ByteBuffer serializeSubscription(final Subscription subscription) {
        return serializeSubscription(subscription, ConsumerProtocolSubscription.HIGHEST_SUPPORTED_VERSION);
    }

    public static ByteBuffer serializeSubscription(final Subscription subscription, short version) {
        version = checkSubscriptionVersion(version);

        ConsumerProtocolSubscription data = new ConsumerProtocolSubscription();

        List<String> topics = new ArrayList<>(subscription.topics());
        Collections.sort(topics);
        data.setTopics(topics);

        data.setUserData(subscription.userData() != null ? subscription.userData().duplicate() : null);

View on GitHub (pinned to c31c9215e1)

Solutions

Validate buffer.remaining() >= 2 before calling ConsumerProtocol.deserializeVersion.
Ensure the buffer passed in was produced by ConsumerProtocol.serializeSubscription/serializeAssignment without intermediate tampering or position shifts (call buffer.flip() or rewind() after writing).
Verify broker and client versions are compatible and that the assignor in use is the standard ClusterResource/CooperativeStickyAssignor, not a custom one returning bad data.
If the data comes from a snapshot or external store, re-fetch it; do not attempt to repair bytes manually.

Example fix

// before
short version = ConsumerProtocol.deserializeVersion(buf);

// after
if (buf == null || buf.remaining() < Short.BYTES) {
    throw new SchemaException("Cannot deserialize consumer protocol: buffer too small (" + (buf == null ? 0 : buf.remaining()) + " bytes)");
}
short version = ConsumerProtocol.deserializeVersion(buf);

Defensive patterns

Strategy: validation

Validate before calling

// deserializeVersion reads a 2-byte short header; reject buffers that are too small
// before calling ConsumerProtocol.deserializeVersion(buffer).
import java.nio.ByteBuffer;

static void ensureVersionHeaderReadable(ByteBuffer buffer) {
    if (buffer == null)
        throw new IllegalArgumentException("buffer must not be null");
    if (buffer.remaining() < Short.BYTES) {
        throw new IllegalArgumentException(
            "Consumer protocol buffer too short: need >= " + Short.BYTES
            + " bytes for version header, have " + buffer.remaining());
    }
}

Try / catch

try {
    short version = ConsumerProtocol.deserializeVersion(buffer);
} catch (org.apache.kafka.common.protocol.types.SchemaException e) {
    // The wire payload is corrupt or truncated — usually a bug in a custom assignor,
    // an intercepted/modified subscription, or a version mismatch with the broker.
    log.error("Could not read consumer protocol version header", e);
    // do not retry with the same buffer; discard and force a rejoin/rebalance.
}

Prevention

If you implement a custom ConsumerPartitionAssignor, always serialize via ConsumerProtocol.serializeSubscription/serializeAssignment so the version header is written correctly.
Never pass a ByteBuffer whose position you have already advanced past the version header to deserializeVersion — duplicate()/slice() first.
Validate buffer.remaining() >= 2 before reading the version header so the error is caught at the trust boundary with a clear message.
Treat a SchemaException here as data corruption, not a transient failure; do not retry with the same payload.

When it happens

Trigger: Calling ConsumerProtocol.deserializeVersion / deserializeSubscription / deserializeAssignment on a ByteBuffer that is empty, has position==limit, or contains fewer than 2 bytes. This typically happens when the broker returns a zero-length subscription/assignment payload, or when user-provided Subscription.userData is fed into the deserializer by mistake.

Common situations: Custom ConsumerPartitionAssignor implementations that hand off malformed ByteBuffers; broker/client version skew where a newer broker sends an envelope an older client cannot frame; corrupted state in __consumer_offsets or group-state snapshots replayed by an assignor; calling deserialize on a buffer whose position was not rewound after a write.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/f829566a41f4066a.json. Report an issue: GitHub.