apache/kafka · error · SchemaException

Error reading array of size ${size}, only ${remaining} bytes

Error message

Error reading array of size ${size}, only ${remaining} bytes available

What it means

Thrown by CompactArrayOf.read after computing `size = n - 1` when size exceeds buffer.remaining(). Same underflow guard as ArrayOf but applied to varint-length compact arrays introduced with flexible message versions. It blocks allocating an array larger than the available payload and aborts the read.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/protocol/types/CompactArrayOf.java:83

        ByteUtils.writeUnsignedVarint(size + 1, buffer);

        for (Object obj : objs)
            type.write(buffer, obj);
    }

    @Override
    public Object read(ByteBuffer buffer) {
        int n = ByteUtils.readUnsignedVarint(buffer);
        if (n == 0) {
            if (isNullable()) {
                return null;
            } else {
                throw new SchemaException("This array is not nullable.");
            }
        }
        int size = n - 1;
        if (size > buffer.remaining())
            throw new SchemaException("Error reading array of size " + size + ", only " + buffer.remaining() + " bytes available");
        Object[] objs = new Object[size];
        for (int i = 0; i < size; i++)
            objs[i] = type.read(buffer);
        return objs;
    }

    @Override
    public int sizeOf(Object o) {
        if (o == null) {
            return 1;
        }
        Object[] objs = (Object[]) o;
        int size = ByteUtils.sizeOfUnsignedVarint(objs.length + 1);
        for (Object obj : objs) {
            size += type.sizeOf(obj);
        }
        return size;
    }

View on GitHub (pinned to c31c9215e1)

Solutions

Verify the request/response API version on the wire matches the schema (and hence the compact-array encoding) used to decode it.
Ensure the ByteBuffer limit spans the entire serialized struct, not a sub-slice.
For recorded/test payloads, confirm they were produced with the same flexible-version setting; regenerate fixtures from current schemas.
Run kafka-dump-log or hex-inspect the region to rule out on-disk corruption.

Example fix

// before - decoding a flexible body with a non-flexible schema
Struct s = (Struct) LEGACY_SCHEMA.read(buf);

// after - pick schema by API version
Schema schema = apiVersion >= 3 ? FLEXIBLE_SCHEMA : LEGACY_SCHEMA;
Struct s = (Struct) schema.read(buf);

Defensive patterns

Strategy: try-catch

Validate before calling

// The (size-1) length is decoded from the varint inside read, so only a
// minimum-feasible pre-check is possible. Verify the buffer can hold the
// varint length prefix before calling read.
if (buffer == null || buffer.remaining() < 1) {
    throw new IllegalArgumentException("buffer too small for compact-array length varint");
}

Try / catch

try {
    Object[] arr = (Object[]) compactArrayOf.read(buffer);
} catch (org.apache.kafka.common.protocol.types.SchemaException e) {
    if (e.getMessage().contains("only") && e.getMessage().contains("bytes available")) {
        // declared size exceeds remaining bytes -> truncated/corrupt compact array
        failFrame(e);
    } else {
        throw e;
    }
}

Prevention

Verify the framing layer delivered the full declared payload before invoking the compact-array reader.
Remember compact length is encoded as N+1 via varint; off-by-one in custom framing causes this.
On short reads, discard the buffer and reconnect rather than re-reading from a tainted position.
Cross-check that the varint decoder (ByteUtils) and the framing agree on byte boundaries.

When it happens

Trigger: CompactArrayOf.read decodes varint N, sets size = N - 1, and checks `size > buffer.remaining()`. Produced by truncated flexible-version payloads, a wrong API version schema (so the varint is actually part of another field), or a sliced buffer that does not cover the full struct.

Common situations: Reading a v3+ (flexible) request/response body with a v2 (non-flexible) schema or vice versa; ByteBuffer slice mis-sized in a custom network codec; log replay after a partial write or segment corruption.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/d0a4b7c917345940.json. Report an issue: GitHub.