apache/flink · error · KryoException

Buffer too small: capacity: {}, required: {}

Error message

Buffer too small: capacity: {}, required: {}

What it means

NoFetchingInput.require(required) throws this KryoException when a single read needs more bytes than the buffer's total capacity. The adapter is constructed with capacity 8, so any Kryo primitive read requiring more than 8 contiguous bytes (e.g. a 9+ byte variable-length long, or a length-prefixed chunk larger than the buffer) cannot be satisfied.

Source

Thrown at flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/NoFetchingInput.java:73

     * then it will load exactly the difference between required and currently available number of
     * bytes. Thus, it will only load the data which is required and never prefetch data.
     *
     * @param required the number of bytes being available in the buffer
     * @return The number of bytes remaining in the buffer, which will be at least <code>required
     *     </code> bytes.
     * @throws KryoException
     */
    @Override
    protected int require(int required) throws KryoException {
        // The main change between this and Kryo 5 Input.require is this will never read more bytes
        // than required.
        // There are also formatting changes to be compliant with the Flink project styling rules.
        int remaining = limit - position;
        if (remaining >= required) {
            return remaining;
        }
        if (required > capacity) {
            throw new KryoException(
                    "Buffer too small: capacity: " + capacity + ", required: " + required);
        }

        int count;
        // Try to fill the buffer.
        if (remaining > 0) {
            // Logical change 1 (from Kryo Input.require): "capacity - limit" -> "required - limit"
            count = fill(buffer, limit, required - limit);
            if (count == -1) {
                throw new KryoBufferUnderflowException("Buffer underflow.");
            }
            remaining += count;
            if (remaining >= required) {
                limit += count;
                return remaining;
            }
        }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Verify the data was written and read with the same serializer/Kryo configuration (registration ids, class-name vs registration mode)
2. Check for stream corruption: truncated checkpoints, shared mutable buffers, or offset bugs in the surrounding operator
3. Report upstream if the payload legitimately exceeds the fixed capacity - NoFetchingInput may need a larger buffer for such codecs

try {
    T value = kryoDeserializer.deserialize(input);
} catch (KryoException e) {
    if (e.getMessage().startsWith("Buffer too small")) {
        // required bytes exceeded NoFetchingInput capacity -> stream format mismatch or corruption
        throw new IllegalStateException("Serialized stream incompatible with NoFetchingInput: " + e.getMessage(), e);
    }
    throw e;
}

When it happens

Trigger: Deserializing through KryoDeserializer with NoFetchingInput when the data stream contains a varlong of 9 bytes (negative longs encode to 9-10 bytes in Kryo) or any read whose 'required' size exceeds 8; typically indicates a corrupted or mismatched stream rather than a legitimately oversized primitive.

Common situations: Stream corruption from partial writes or offset misalignment; data serialized by a different Kryo version/config (different varint encoding) than the reader; reading Kryo bytes not produced through the matching Flink serializer.

Related errors

• Buffer underflow.
• Failed to serialize value '{value}'
• Cannot register null class or serializer.
• Could not instantiate record
• Could not copy object by serializing/deserializing it.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/2ba9796acc1e8b37. Report an issue: GitHub.