{"id":"f829566a41f4066a","repo":"apache/kafka","slug":"buffer-underflow-while-parsing-consumer-protocol-s","errorCode":null,"errorMessage":"Buffer underflow while parsing consumer protocol's header","messagePattern":"Buffer underflow while parsing consumer protocol's header","errorType":"exception","errorClass":"SchemaException","httpStatus":null,"severity":"error","filePath":"clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerProtocol.java","lineNumber":64,"sourceCode":"\n    static {\n        // Safety check to ensure that both parts of the consumer protocol remain in sync.\n        if (ConsumerProtocolSubscription.LOWEST_SUPPORTED_VERSION\n                != ConsumerProtocolAssignment.LOWEST_SUPPORTED_VERSION)\n            throw new IllegalStateException(\"Subscription and Assignment schemas must have the \" +\n                \"same lowest version\");\n\n        if (ConsumerProtocolSubscription.HIGHEST_SUPPORTED_VERSION\n                != ConsumerProtocolAssignment.HIGHEST_SUPPORTED_VERSION)\n            throw new IllegalStateException(\"Subscription and Assignment schemas must have the \" +\n                \"same highest version\");\n    }\n\n    public static short deserializeVersion(final ByteBuffer buffer) {\n        try {\n            return buffer.getShort();\n        } catch (BufferUnderflowException e) {\n            throw new SchemaException(\"Buffer underflow while parsing consumer protocol's header\", e);\n        }\n    }\n\n    public static ByteBuffer serializeSubscription(final Subscription subscription) {\n        return serializeSubscription(subscription, ConsumerProtocolSubscription.HIGHEST_SUPPORTED_VERSION);\n    }\n\n    public static ByteBuffer serializeSubscription(final Subscription subscription, short version) {\n        version = checkSubscriptionVersion(version);\n\n        ConsumerProtocolSubscription data = new ConsumerProtocolSubscription();\n\n        List<String> topics = new ArrayList<>(subscription.topics());\n        Collections.sort(topics);\n        data.setTopics(topics);\n\n        data.setUserData(subscription.userData() != null ? subscription.userData().duplicate() : null);\n","sourceCodeStart":46,"sourceCodeEnd":82,"githubUrl":"https://github.com/apache/kafka/blob/c31c9215e131f8c17e79f8901b48c13ee6aa8e7a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerProtocol.java#L46-L82","documentation":"Thrown by ConsumerProtocol.deserializeVersion() when the supplied ByteBuffer has fewer than 2 bytes remaining, so the version-prefix short cannot be read. Kafka's consumer protocol wire payload begins with a 2-byte version header; if the buffer is empty, truncated, or positioned past its data, deserialization cannot even begin. The original BufferUnderflowException is attached as the cause.","triggerScenarios":"Calling ConsumerProtocol.deserializeVersion / deserializeSubscription / deserializeAssignment on a ByteBuffer that is empty, has position==limit, or contains fewer than 2 bytes. This typically happens when the broker returns a zero-length subscription/assignment payload, or when user-provided Subscription.userData is fed into the deserializer by mistake.","commonSituations":"Custom ConsumerPartitionAssignor implementations that hand off malformed ByteBuffers; broker/client version skew where a newer broker sends an envelope an older client cannot frame; corrupted state in __consumer_offsets or group-state snapshots replayed by an assignor; calling deserialize on a buffer whose position was not rewound after a write.","solutions":["Validate buffer.remaining() >= 2 before calling ConsumerProtocol.deserializeVersion.","Ensure the buffer passed in was produced by ConsumerProtocol.serializeSubscription/serializeAssignment without intermediate tampering or position shifts (call buffer.flip() or rewind() after writing).","Verify broker and client versions are compatible and that the assignor in use is the standard ClusterResource/CooperativeStickyAssignor, not a custom one returning bad data.","If the data comes from a snapshot or external store, re-fetch it; do not attempt to repair bytes manually."],"exampleFix":"// before\nshort version = ConsumerProtocol.deserializeVersion(buf);\n\n// after\nif (buf == null || buf.remaining() < Short.BYTES) {\n    throw new SchemaException(\"Cannot deserialize consumer protocol: buffer too small (\" + (buf == null ? 0 : buf.remaining()) + \" bytes)\");\n}\nshort version = ConsumerProtocol.deserializeVersion(buf);","handlingStrategy":"validation","validationCode":"// deserializeVersion reads a 2-byte short header; reject buffers that are too small\n// before calling ConsumerProtocol.deserializeVersion(buffer).\nimport java.nio.ByteBuffer;\n\nstatic void ensureVersionHeaderReadable(ByteBuffer buffer) {\n    if (buffer == null)\n        throw new IllegalArgumentException(\"buffer must not be null\");\n    if (buffer.remaining() < Short.BYTES) {\n        throw new IllegalArgumentException(\n            \"Consumer protocol buffer too short: need >= \" + Short.BYTES\n            + \" bytes for version header, have \" + buffer.remaining());\n    }\n}","typeGuard":null,"tryCatchPattern":"try {\n    short version = ConsumerProtocol.deserializeVersion(buffer);\n} catch (org.apache.kafka.common.protocol.types.SchemaException e) {\n    // The wire payload is corrupt or truncated — usually a bug in a custom assignor,\n    // an intercepted/modified subscription, or a version mismatch with the broker.\n    log.error(\"Could not read consumer protocol version header\", e);\n    // do not retry with the same buffer; discard and force a rejoin/rebalance.\n}","preventionTips":["If you implement a custom ConsumerPartitionAssignor, always serialize via ConsumerProtocol.serializeSubscription/serializeAssignment so the version header is written correctly.","Never pass a ByteBuffer whose position you have already advanced past the version header to deserializeVersion — duplicate()/slice() first.","Validate buffer.remaining() >= 2 before reading the version header so the error is caught at the trust boundary with a clear message.","Treat a SchemaException here as data corruption, not a transient failure; do not retry with the same payload."],"tags":["consumer","protocol","serialization","schema","bytebuffer"],"analyzedSha":"c31c9215e131f8c17e79f8901b48c13ee6aa8e7a","analyzedAt":"2026-08-03T12:34:05.770Z","schemaVersion":2}