apache/flink · error · IllegalArgumentException

Buffer size must be greater than length of delimiter.

Error message

Buffer size must be greater than length of delimiter.

What it means

In initBuffers(), DelimitedInputFormat normalizes bufferSize (defaulting to 1 MB if <= 0) and then requires the buffer to be strictly larger than the delimiter length so the delimiter-scanning loop can always fully observe the delimiter within a single buffer window. If bufferSize <= delimiter.length, scanning would miss delimiters, so the format rejects the configuration.

Source

Thrown at flink-core/src/main/java/org/apache/flink/api/common/io/DelimitedInputFormat.java:515

        if (this.splitStart != 0) {
            this.stream.seek(offset);
            readLine();
            // if the first partial record already pushes the stream over
            // the limit of our split, then no record starts within this split
            if (this.overLimit) {
                this.end = true;
            }
        } else {
            fillBuffer(0);
        }
        initializeSplit(split, null);
    }

    private void initBuffers() {
        this.bufferSize = this.bufferSize <= 0 ? DEFAULT_READ_BUFFER_SIZE : this.bufferSize;

        if (this.bufferSize <= this.delimiter.length) {
            throw new IllegalArgumentException(
                    "Buffer size must be greater than length of delimiter.");
        }

        if (this.readBuffer == null || this.readBuffer.length != this.bufferSize) {
            this.readBuffer = new byte[this.bufferSize];
        }
        if (this.wrapBuffer == null || this.wrapBuffer.length < 256) {
            this.wrapBuffer = new byte[256];
        }

        this.readPos = 0;
        this.limit = 0;
        this.overLimit = false;
        this.end = false;
    }

    /**
     * Checks whether the current split is at its end.

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Increase bufferSize so it is strictly greater than delimiter.length (e.g. format.setBufferSize(Math.max(bufferBytes, delimiter.length + 1))).
2. Shorten the delimiter if a long one is not required.
3. Leave both at defaults (1 MB buffer, single-byte newline).

Example fix

// before
format.setDelimiter("\r\n\n".getBytes(StandardCharsets.UTF_8)); // 3 bytes
format.setBufferSize(2); // throws: 2 <= 3

// after
byte[] delim = "\r\n\n".getBytes(StandardCharsets.UTF_8);
format.setDelimiter(delim);
format.setBufferSize(Math.max(4 * 1024, delim.length + 1));

byte[] delim = configuredDelimiter;
int minBuffer = Math.max(bufferSize, delim.length + 1);
format.setBufferSize(minBuffer);
format.setDelimiter(delim);

When it happens

Trigger: Setting a multi-byte delimiter (e.g. '\r\n' or a long custom byte sequence) larger than or equal to the configured read buffer; setting bufferSize to a small value (e.g. 2) while using a 2-byte delimiter.

Common situations: Custom multi-byte record separators combined with aggressive buffer downsizing; copying a delimiter string that is longer than expected; using a very small buffer for testing.

Related errors

• Delimiter must not be null
• Buffer size must be at least 2.
• Line length limit must be at least 1.
• Number of line samples must not be negative.
• The block size parameter must be set and larger than 0.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/472e3d4674ac2109. Report an issue: GitHub.