apache/kafka · error · SchemaException

Buffer underflow while parsing response for request with hea

Error message

Buffer underflow while parsing response for request with header {requestHeader}

What it means

Thrown as a SchemaException by NetworkClient.parseResponse when AbstractResponse.parseResponse hits a BufferUnderflowException while deserializing a response ByteBuffer against the request header. It means the bytes available were fewer than the protocol schema requires, so the response is structurally incomplete or misaligned with the request the client sent. The request header is included to identify which exchange produced the malformed bytes.

Source

Thrown at clients/src/main/java/org/apache/kafka/clients/NetworkClient.java:921

     * <p>
     * If bootstrap is disabled or already complete, throw IllegalStateException.
     * If bootstrap is enabled but not yet complete, return an empty {@link LeastLoadedNode}
     * so that the caller can continue polling while DNS resolution finishes.
     */
    private LeastLoadedNode handleEmptyNodeList() {
        if (bootstrapConfiguration == BootstrapConfiguration.DISABLED || metadataUpdater.isBootstrapped()) {
            throw new IllegalStateException("There are no nodes in the Kafka cluster");
        }

        log.debug("No nodes available yet, still in bootstrap phase");
        return new LeastLoadedNode(null, false);
    }

    public static AbstractResponse parseResponse(ByteBuffer responseBuffer, RequestHeader requestHeader) {
        try {
            return AbstractResponse.parseResponse(responseBuffer, requestHeader);
        } catch (BufferUnderflowException e) {
            throw new SchemaException("Buffer underflow while parsing response for request with header " + requestHeader, e);
        } catch (CorrelationIdMismatchException e) {
            if (SaslClientAuthenticator.isReserved(requestHeader.correlationId())
                && !SaslClientAuthenticator.isReserved(e.responseCorrelationId()))
                throw new SchemaException("The response is unrelated to Sasl request since its correlation id is "
                    + e.responseCorrelationId() + " and the reserved range for Sasl request is [ "
                    + SaslClientAuthenticator.MIN_RESERVED_CORRELATION_ID + ","
                    + SaslClientAuthenticator.MAX_RESERVED_CORRELATION_ID + "]");
            else {
                throw e;
            }
        }
    }

    /**
     * Post process disconnection of a node
     *
     * @param responses The list of responses to update
     * @param nodeId Id of the node to be disconnected

View on GitHub (pinned to c31c9215e1)

Solutions

1. Align client and broker versions; pin the client to a Kafka version compatible with the broker.
2. Remove any L4/L7 proxy, sidecar, or SSL terminator between client and broker, or verify it preserves raw TCP framing.
3. Enable client/broker-side debug logging for the affected API key and confirm the request header (api key, version) matches what the broker expects.
4. If reproducible against one broker only, check that broker's logs for OOM, disk, or protocol-level errors.
5. Upgrade both ends to the latest patch release to pick up fixed response encoding.

Example fix

// before: old client against new broker without api versions negotiation
<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-clients</artifactId>
  <version>2.8.0</version>
</dependency>
// after: align with broker version
<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-clients</artifactId>
  <version>3.7.0</version>
</dependency>

// No caller-side input can prevent a truncated wire response, but you can pre-check channel health:
if (!networkClient.isReady(node, now)) {
    throw new org.apache.kafka.common.errors.DisconnectException(
        "Refusing to send to node " + node + " - connection not ready");
}
// Prefer known-good connection before sending; this reduces (not eliminates) underflow triggers.

// Type-guard the response before trusting its bytes.
public static boolean looksComplete(java.nio.ByteBuffer buf, int minBytes) {
    return buf != null && buf.remaining() >= minBytes;
}
// Note: the wire schema is variable-length so a full guard requires the API key + version;
// if you parse manually, validate response size >= expected header size first.

try {
    org.apache.kafka.common.requests.AbstractResponse r =
            networkClient.parseResponse(buf, header);
} catch (org.apache.kafka.common.errors.SchemaException e) {
    if (e.getMessage().contains("Buffer underflow")) {
    // Truncated/partial frame on the wire. Close the connection, refresh metadata,
    // and retry the request with a fresh correlation id. Treat as transient.
    networkClient.disconnect(nodeId);
    return retryWithBackoff();
    }
    throw e;
}

When it happens

Trigger: A response arrives whose size prefix or body does not match the schema expected for the API key/version in RequestHeader. Causes include network-layer truncation, a proxy/load-balancer rewriting or truncating frames, a client/broker API version mismatch, or a corrupted SSL/TLS record. Fires inside parseResponse, which is invoked from NetworkClient.handleCompletedReceives during poll().

Common situations: Incompatible client/broker versions where the broker encodes fields the older client does not parse (or vice-versa); an L7 proxy or sidecar that does not preserve TCP byte stream framing; TLS offload misconfiguration; transient kernel/socket buffer corruption on a flaky link; running an outdated client against a much newer cluster.

Related errors

• Buffer underflow while parsing consumer protocol's header
• Malformed consumer protocol subscription
• Malformed consumer protocol assignment
• Unsupported subscription version: {}
• Unsupported assignment version: {}

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/3715fa17690a4342.json. Report an issue: GitHub.