apache/flink · error · IOException

The record length exceeded the maximum record length (${line

Error message

The record length exceeded the maximum record length (${lineLengthLimit}).

What it means

While reading, DelimitedInputFormat accumulates bytes for the current record into a wrap buffer when a record spans the end of the read buffer. If the accumulated record length exceeds lineLengthLimit, it throws IOException — this is a safety valve to stop runaway reads caused by a wrong/missing delimiter (the whole file would otherwise be treated as one record).

Source

Thrown at flink-core/src/main/java/org/apache/flink/api/common/io/DelimitedInputFormat.java:663

                        this.wrapBuffer = nb;
                    }
                    if (count >= 0) {
                        System.arraycopy(
                                this.readBuffer, 0, this.wrapBuffer, countInWrapBuffer, count);
                    }
                    setResult(this.wrapBuffer, 0, countInWrapBuffer + count);
                    return true;
                } else {
                    setResult(this.readBuffer, startPos, count);
                    return true;
                }
            } else {
                // we reached the end of the readBuffer
                count = this.limit - startPos;

                // check against the maximum record length
                if (((long) countInWrapBuffer) + count > this.lineLengthLimit) {
                    throw new IOException(
                            "The record length exceeded the maximum record length ("
                                    + this.lineLengthLimit
                                    + ").");
                }

                // Compute number of bytes to move to wrapBuffer
                // Chars of partially read delimiter must remain in the readBuffer. We might need to
                // go back.
                int bytesToMove = count - delimPos;
                // ensure wrapBuffer is large enough
                if (this.wrapBuffer.length - countInWrapBuffer < bytesToMove) {
                    // reallocate
                    byte[] tmp =
                            new byte
                                    [Math.max(
                                            this.wrapBuffer.length * 2,
                                            countInWrapBuffer + bytesToMove)];
                    System.arraycopy(this.wrapBuffer, 0, tmp, 0, countInWrapBuffer);

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Verify and correct the delimiter to match the actual file format (use a hex viewer / od to confirm).
2. Increase lineLengthLimit via setLineLengthLimit to accommodate the largest legitimate record.
3. Decompress the file first (or let the format decompress via the appropriate InflaterInputStreamFactory / file extension).
4. If the file is binary, use a dedicated InputFormat rather than the delimited text format.

Example fix

// before: wrong delimiter -> whole file read as one record -> throws
format.setDelimiter(";");
format.setLineLengthLimit(1024);
// file uses newlines

// after
format.setDelimiter("\n");
format.setLineLengthLimit(10 * 1024 * 1024);

// Validate the delimiter actually appears in a sample of the file before reading.
// (defensive; the real guard is catching IOException at runtime)
format.setLineLengthLimit(largestExpectedRecordBytes);

try {
    while (!format.reachedEnd()) {
        OT rec = format.nextRecord(null);
        // process
    }
} catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("maximum record length")) {
        // delimiter likely wrong; raise a clearer error or switch delimiter
        throw new RuntimeException("Record exceeded lineLengthLimit; verify delimiter and limit", e);
    }
    throw e;
}

When it happens

Trigger: Reading a file where the configured delimiter never appears within lineLengthLimit bytes (wrong delimiter, binary file read as text, or a genuinely oversized record).

Common situations: Wrong delimiter for the file format (e.g. '\n' set but file uses '\r\n' or a custom separator); reading a binary/compressed-without-decompression file as delimited text; lineLengthLimit left at default on files with legitimately long records.

Related errors

• Delimiter must not be null
• Buffer size must be greater than length of delimiter.
• Line length limit must be at least 1.
• Buffer size must be at least 2.
• Number of line samples must not be negative.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/fa7feb82a6f3b20b. Report an issue: GitHub.