apache/kafka · error · SchemaException

Error reading array of size ${size}, only ${remaining} bytes

Error message

Error reading array of size ${size}, only ${remaining} bytes available

What it means

Thrown by ArrayOf.read when the declared array length exceeds the bytes left in the ByteBuffer. It is a sanity guard preventing an oversized allocation and an underflow when element deserialization would run past the buffer end. The library treats this as unrecoverable wire corruption.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/protocol/types/ArrayOf.java:78

        Object[] objs = (Object[]) o;
        int size = objs.length;
        buffer.putInt(size);

        for (Object obj : objs)
            type.write(buffer, obj);
    }

    @Override
    public Object read(ByteBuffer buffer) {
        int size = buffer.getInt();
        if (size < 0 && isNullable())
            return null;
        else if (size < 0)
            throw new SchemaException("Array size " + size + " cannot be negative");

        if (size > buffer.remaining())
            throw new SchemaException("Error reading array of size " + size + ", only " + buffer.remaining() + " bytes available");
        Object[] objs = new Object[size];
        for (int i = 0; i < size; i++)
            objs[i] = type.read(buffer);
        return objs;
    }

    @Override
    public int sizeOf(Object o) {
        int size = 4;
        if (o == null)
            return size;

        Object[] objs = (Object[]) o;
        for (Object obj : objs)
            size += type.sizeOf(obj);
        return size;
    }

View on GitHub (pinned to c31c9215e1)

Solutions

Confirm the ByteBuffer passed to read covers the complete message body (check limit-position vs. the framed size from the transport).
Ensure you are decoding with the Schema for the exact API key + API version present on the wire.
If reading from a log/record, verify the record offset and segment integrity (e.g. kafka-dump-log) before deserializing.
Add a length check in your framing layer so partial reads are rejected before reaching the schema.

Example fix

// before - reading a possibly partial slice
Struct s = (Struct) schema.read(slice);

// after - validate the slice spans the whole framed message first
if (slice.remaining() < framedSize) {
    throw new IllegalStateException("underfull frame: " + slice.remaining() + " < " + framedSize);
}
Struct s = (Struct) schema.read(slice);

Defensive patterns

Strategy: try-catch

Validate before calling

// Best-effort: ensure the buffer at least carries the 4-byte length prefix.
// The declared size is read internally, so a full pre-check is impossible;
// pair this with the try-catch below.
if (buffer == null || buffer.remaining() < Integer.BYTES) {
    throw new IllegalArgumentException("buffer has no array-length prefix");
}

Try / catch

try {
    Object[] arr = (Object[]) arrayOf.read(buffer);
} catch (org.apache.kafka.common.protocol.types.SchemaException e) {
    if (e.getMessage().contains("only") && e.getMessage().contains("bytes available")) {
        // truncated frame: size claimed more than the wire actually delivered
        handleShortRead(e);
    } else {
        throw e;
    }
}

Prevention

Frame messages with an explicit length prefix at the transport layer so partial reads are detected before deserialization.
Treat a short buffer as a connection-level failure: close and re-establish rather than retrying the same bytes.
When slicing buffers for sub-readers, double-check slice length equals the documented payload size.
Log buffer.remaining() at the framing boundary to catch truncation early in diagnostics.

When it happens

Trigger: ArrayOf.read reads a length N then checks `size > buffer.remaining()`; the next N elements of the element Type cannot fit. Happens when the element Type's sizeOf assumptions differ from what was written, or when the buffer was sliced/truncated before being passed to read.

Common situations: Message body was truncated by a network layer or truncated ByteBuffer slice; reading a payload with the wrong API version's schema (so the length prefix is actually some other field's bytes); corrupted on-disk log being replayed; a custom interceptor that re-slices buffers incorrectly.

Related errors

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/4c7d1462490e7630.json. Report an issue: GitHub.