apache/kafka · error · SchemaException

Error reading field of size ${size}, only ${remaining} bytes

Error message

Error reading field of size ${size}, only ${remaining} bytes available

What it means

Thrown by TaggedFields.read when a tagged field's declared size exceeds the remaining bytes in the ByteBuffer. The check prevents allocating an oversized byte[] and subsequent BufferUnderflow when copying the field payload. It indicates either truncation or a misread size (the size varint actually belonged to different bytes).

Source

Thrown at clients/src/main/java/org/apache/kafka/common/protocol/types/TaggedFields.java:99

    @Override
    public NavigableMap<Integer, Object> read(ByteBuffer buffer) {
        int numTaggedFields = ByteUtils.readUnsignedVarint(buffer);
        if (numTaggedFields == 0) {
            return Collections.emptyNavigableMap();
        }
        NavigableMap<Integer, Object> objects = new TreeMap<>();
        int prevTag = -1;
        for (int i = 0; i < numTaggedFields; i++) {
            int tag = ByteUtils.readUnsignedVarint(buffer);
            if (tag <= prevTag) {
                throw new RuntimeException("Invalid or out-of-order tag " + tag);
            }
            prevTag = tag;
            int size = ByteUtils.readUnsignedVarint(buffer);
            if (size < 0)
                throw new SchemaException("field size " + size + " cannot be negative");
            if (size > buffer.remaining())
                throw new SchemaException("Error reading field of size " + size + ", only " + buffer.remaining() + " bytes available");

            Field field = fields.get(tag);
            if (field == null) {
                byte[] bytes = new byte[size];
                buffer.get(bytes);
                objects.put(tag, new RawTaggedField(tag, bytes));
            } else {
                objects.put(tag, field.type.read(buffer));
            }
        }
        return objects;
    }

    @SuppressWarnings("unchecked")
    @Override
    public int sizeOf(Object o) {
        int size = 0;
        NavigableMap<Integer, Object> objects = (NavigableMap<Integer, Object>) o;

View on GitHub (pinned to c31c9215e1)

Solutions

1. Verify the buffer limit spans the full framed message before invoking read (compare to the size advertised by the transport).
2. Confirm the API version on the wire matches the schema so tagged-field offsets are correct.
3. Inspect the reported size vs remaining; if the size is implausibly large, suspect a misread varint from buffer misalignment and trace the preceding reads.
4. Validate log/record integrity (kafka-dump-log) and regenerate test payloads from current schemas.

Example fix

// before - passing an unbounded slice
ByteBuffer slice = buf.duplicate();
return taggedFields.read(slice);

// after - bound the slice to the declared frame length first
ByteBuffer slice = (ByteBuffer) buf.slice().limit(frameLength);
return taggedFields.read(slice);

// Like the other short-read cases, the declared size is internal to read().
// A pre-check can only confirm the buffer is non-empty before the call:
if (buffer == null || buffer.remaining() < 1) {
    throw new IllegalArgumentException("buffer too small for tagged-field payload");
}

try {
    NavigableMap<Integer, Object> tf = taggedFields.read(buffer);
} catch (org.apache.kafka.common.protocol.types.SchemaException e) {
    if (e.getMessage().contains("only") && e.getMessage().contains("bytes available")) {
        // tagged-field payload was truncated relative to its declared size
        handleShortRead(e);
    } else {
        throw e;
    }
}

When it happens

Trigger: TaggedFields.read reads the size varint then checks `size > buffer.remaining()`. Produced by a payload that ends before the declared tagged-field payload, a frame that was sliced too short, or a buffer position advanced incorrectly so the size is misread.

Common situations: Network layer truncated the frame; ByteBuffer slice limit was set short of the full message; reading a flexible-version body with a non-flexible schema so the trailing tagged-fields bytes are interpreted with the wrong offsets; on-disk log corruption; test fixture with a hand-coded wrong size varint.

Related errors

• Error reading array of size ${size}, only ${remaining} bytes
• Invalid or out-of-order tag ${tag}
• field size ${size} cannot be negative
• Error reading array of size ${size}, only ${remaining} bytes
• This array is not nullable.

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/81f9df61391c514a.json. Report an issue: GitHub.