apache/kafka · error · SchemaException

Malformed consumer protocol subscription

Error message

Malformed consumer protocol subscription

What it means

Thrown by ConsumerProtocol.deserializeSubscription(ByteBuffer, short) when constructing a ConsumerProtocolSubscription from the buffer raises any RuntimeException. This wraps the underlying parse failure (typically BufferUnderflowException, IllegalArgumentException, or ArrayIndexOutOfBoundsException) into a SchemaException so callers see a single, well-typed error describing a structurally invalid subscription payload.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerProtocol.java:120

        try {
            ConsumerProtocolSubscription data =
                new ConsumerProtocolSubscription(new ByteBufferAccessor(buffer), version);

            List<TopicPartition> ownedPartitions = new ArrayList<>();
            for (ConsumerProtocolSubscription.TopicPartition tp : data.ownedPartitions()) {
                for (Integer partition : tp.partitions()) {
                    ownedPartitions.add(new TopicPartition(tp.topic(), partition));
                }
            }

            return new Subscription(
                data.topics(),
                data.userData() != null ? data.userData().duplicate() : null,
                ownedPartitions,
                data.generationId(),
                data.rackId() == null || data.rackId().isEmpty() ? Optional.empty() : Optional.of(data.rackId()));
        } catch (RuntimeException e) {
            throw new SchemaException("Malformed consumer protocol subscription", e);
        }
    }

    public static Subscription deserializeSubscription(final ByteBuffer buffer) {
        return deserializeSubscription(buffer, deserializeVersion(buffer));
    }

    public static ConsumerProtocolSubscription deserializeConsumerProtocolSubscription(
        final ByteBuffer buffer,
        short version
    ) {
        version = checkSubscriptionVersion(version);

        try {
            return new ConsumerProtocolSubscription(new ByteBufferAccessor(buffer), version);
        } catch (RuntimeException e) {
            throw new SchemaException("Malformed consumer protocol subscription", e);
        }

View on GitHub (pinned to c31c9215e1)

Solutions

1. Inspect the cause (e.getCause()) of the SchemaException to identify the exact field that failed to parse.
2. Ensure all clients in the consumer group run a broker-and-protocol-compatible Kafka client version (subscriptions are forward-compatible but malformed payloads indicate a non-standard producer of the bytes).
3. Remove or fix any custom ConsumerPartitionAssignor that constructs subscription ByteBuffers by hand.
4. If the error appears after an upgrade, downgrade or roll forward all group members together so they share a schema.

// The body parse cannot be meaningfully pre-validated without decoding the schema,
// so the only cheap pre-check is buffer sanity. Always run this before deserialize*.
static void requireReadable(ByteBuffer buffer) {
    if (buffer == null) throw new IllegalArgumentException("buffer == null");
    if (buffer.remaining() < Short.BYTES)
        throw new IllegalArgumentException("subscription buffer too short to contain a version header");
}

try {
    Subscription sub = ConsumerProtocol.deserializeSubscription(buffer, version);
} catch (org.apache.kafka.common.protocol.types.SchemaException e) {
    // Subscription payload is malformed (bad topic string, truncated ownedPartitions, etc.).
    // Most common when a non-consumer-protocol payload was sent on the group path,
    // or when broker and client run incompatible consumer protocol versions.
    log.warn("Rejecting malformed consumer protocol subscription", e);
    // trigger a fresh join/rebalance rather than retrying the same bytes.
}

When it happens

Trigger: A consumer group leader or broker sends a subscription ByteBuffer whose contents do not match the consumer protocol schema for the given version, or whose declared lengths exceed remaining bytes. Surfaced during SyncGroup/JoinGroup response handling on the broker-side assignor or when a client deserializes a peer's subscription.

Common situations: Mixed client versions where a newer client serializes subscription fields an older client cannot decode; custom assignor plugins producing non-conformant payloads; truncated network frames or corrupted stored group metadata being replayed during assignment.

Related errors

• Buffer underflow while parsing consumer protocol's header
• Malformed consumer protocol assignment
• Unsupported subscription version: {}
• Unsupported assignment version: {}
• Array size ${size} cannot be negative

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/0ff99ff55dc829a9.json. Report an issue: GitHub.