apache/flink · error · IllegalArgumentException

The request entry sent to the buffer was of size [%s], when

Error message

The request entry sent to the buffer was of size [%s], when the maxRecordSizeInBytes was set to [%s].

What it means

Thrown as an IllegalArgumentException by AsyncSinkWriter.addEntryToBuffer when a request entry's computed size (from getSizeInBytes) exceeds the configured maxRecordSizeInBytes limit. maxRecordSizeInBytes represents the maximum payload size the destination system accepts per record. This check prevents oversized records from entering the buffer where they would always fail on submission and could never be retried.

Source

Thrown at flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java:416

        ListIterator<RequestEntryT> iterator =
                failedRequestEntries.listIterator(failedRequestEntries.size());
        while (iterator.hasPrevious()) {
            addEntryToBuffer(iterator.previous(), true);
        }
        nonBlockingFlush();
    }

    private void addEntryToBuffer(RequestEntryT entry, boolean insertAtHead) {
        addEntryToBuffer(new RequestEntryWrapper<>(entry, getSizeInBytes(entry)), insertAtHead);
    }

    private void addEntryToBuffer(RequestEntryWrapper<RequestEntryT> entry, boolean insertAtHead) {
        if (bufferedRequestEntries.isEmpty() && !existsActiveTimerCallback) {
            registerCallback();
        }

        if (entry.getSize() > maxRecordSizeInBytes) {
            throw new IllegalArgumentException(
                    String.format(
                            "The request entry sent to the buffer was of size [%s], when the maxRecordSizeInBytes was set to [%s].",
                            entry.getSize(), maxRecordSizeInBytes));
        }

        bufferedRequestEntries.add(entry, insertAtHead);
    }

    /**
     * In flight requests will be retried if the sink is still healthy. But if in-flight requests
     * fail after a checkpoint has been triggered and Flink needs to recover from the checkpoint,
     * the (failed) in-flight requests are gone and cannot be retried. Hence, there cannot be any
     * outstanding in-flight requests when a commit is initialized.
     *
     * <p>To this end, all in-flight requests need to completed before proceeding with the commit.
     */
    @Override
    public void flush(boolean flush) throws InterruptedException {

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Pre-filter or split oversized records before they reach the sink (e.g., chunk large payloads into multiple smaller entries).
2. Increase maxRecordSizeInBytes in the sink builder if the destination supports larger records: builder.setMaxRecordSizeInBytes(largerValue).
3. Verify getSizeInBytes() in your AsyncSinkWriter returns the correct byte size — debug-log it for the failing record.
4. If the destination has a hard limit (e.g., Kinesis 1MB), implement a pre-processing step to split or compress records exceeding it.

Example fix

// before — record exceeds max size
builder.setMaxRecordSizeInBytes(1024 * 1024); // 1MB default
// record is 2MB -> throws
// after — split large records or increase limit
builder.setMaxRecordSizeInBytes(4 * 1024 * 1024); // 4MB if destination supports it
// Or split in ElementConverter:
@Override
public String apply(String input, Context ctx) {
    if (input.getBytes().length > MAX) { return split(input); }
    return input;
}

// Validate record size before adding to buffer
long recordSize = getSizeInBytes(requestEntry);
if (recordSize > maxRecordSizeInBytes) {
    // split, compress, or drop the record
    throw new IllegalArgumentException("Record size " + recordSize + " exceeds max " + maxRecordSizeInBytes);
}

try {
    writer.invokeInternal(record);
} catch (IllegalArgumentException e) {
    if (e.getMessage().contains("maxRecordSizeInBytes")) {
        // split or compress the record, or route to a dead-letter sink
    }
}

When it happens

Trigger: An input element converts to a request entry whose getSizeInBytes() return value exceeds the maxRecordSizeInBytes configured via the sink builder's setMaxRecordSizeInBytes().

Common situations: A large record (e.g., a big JSON document or binary blob) exceeds the destination's per-record limit (e.g., AWS Kinesis 1MB, AWS Firehose limits); maxRecordSizeInBytes was left at a default that is too small for the data; getSizeInBytes returns an incorrect (inflated) size due to a bug in the ElementConverter.

Related errors

• Cannot create empty classifier chain.
• {e.getMessage()}
• Cannot access jar file{t.getMessage() == null ? "." : ": " +
• The jarFile and entryPointClassName can not be null at the s
• Not allowed configuration change(s) were detected:\n - {erro

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/952f47c9a7f20726. Report an issue: GitHub.