apache/kafka · error · SchemaException

Error reading bytes of size ${size}, only ${remaining} bytes

Error message

Error reading bytes of size ${size}, only ${remaining} bytes available

What it means

Thrown by bytesRead() when the BYTES field's declared size is positive but larger than buffer.remaining(). This is the truncated-frame guard for the raw-byte schema type and is raised as a SchemaException before slicing the buffer. It protects against underflow and malformed length prefixes during Kafka protocol decoding.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/protocol/types/Type.java:132

        @Override
        public String toString() {
            return typeName();
        }
    }

    public static String stringRead(ByteBuffer buffer, int length) {
        if (length > Short.MAX_VALUE)
            throw new SchemaException("String length " + length + " is larger than the maximum string length.");
        if (length > buffer.remaining())
            throw new SchemaException("Error reading string of length " + length + ", only " + buffer.remaining() + " bytes available");
        String result = Utils.utf8(buffer, length);
        buffer.position(buffer.position() + length);
        return result;
    }

    public static ByteBuffer bytesRead(ByteBuffer buffer, int size) {
        if (size > buffer.remaining())
            throw new SchemaException("Error reading bytes of size " + size + ", only " + buffer.remaining() + " bytes available");

        int limit = buffer.limit();
        int newPosition = buffer.position() + size;
        buffer.limit(newPosition);
        ByteBuffer val = buffer.slice();
        buffer.limit(limit);
        buffer.position(newPosition);
        return val;
    }

    /**
     * The Boolean type represents a boolean value in a byte by using
     * the value of 0 to represent false, and 1 to represent true.
     *
     * If for some reason a value that is not 0 or 1 is read,
     * then any non-zero value will return true.
     */
    public static final DocumentedType BOOLEAN = new DocumentedType() {

View on GitHub (pinned to c31c9215e1)

Solutions

1. Confirm the complete response frame has been read off the socket before invoking the schema decoder.
2. Log size vs buffer.remaining() at the failure point to confirm truncation vs schema drift.
3. Check for an earlier field whose wrong length shifted the position (decode field-by-field in DEBUG).
4. Align client and broker versions for the API key in the failing exchange.

Example fix

// before: decoding before the frame is fully buffered
ByteBuffer val = Type.bytesRead(buffer, size); // throws if size > remaining

// after: ensure full frame presence first
if (buffer.remaining() < size) {
    throw new IllegalStateException("Incomplete frame: need " + size
        + " bytes, have " + buffer.remaining());
}
ByteBuffer val = buffer.slice();

// bytesRead(buffer, size) is public and takes size as a param.
if (size > buffer.remaining()) {
    throw new IllegalArgumentException(
        "Need " + size + " bytes but only " + buffer.remaining() + " available");
}
java.nio.ByteBuffer value = org.apache.kafka.common.protocol.types.Type.bytesRead(buffer, size);

When it happens

Trigger: BYTES.read(buffer) or COMPACT_BYTES.read(buffer) calling bytesRead(buffer, size) where size > buffer.remaining(). Hit during ApiMessage/Struct deserialization of any byte-array field (e.g. record batches, SASL tokens, serialized values) when the source buffer is short.

Common situations: Partial network read passed to the decoder. Buffer position corrupted by an earlier mis-sized field. Client/broker version mismatch changing a field's encoding. A proxy or custom serializer emitting an incorrect length prefix.

Related errors

• Error reading string of length ${length}, only ${remaining}
• Bytes size ${size} cannot be negative
• Buffer underflow while parsing consumer protocol's header
• Malformed consumer protocol subscription
• Malformed consumer protocol assignment

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/a9a89925646b017a.json. Report an issue: GitHub.