{"record":{"id":"472e3d4674ac2109","repo":"apache/flink","slug":"buffer-size-must-be-greater-than-length-of-delimit","errorCode":null,"errorMessage":"Buffer size must be greater than length of delimiter.","messagePattern":"Buffer size must be greater than length of delimiter\\.","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"flink-core/src/main/java/org/apache/flink/api/common/io/DelimitedInputFormat.java","lineNumber":515,"sourceCode":"        if (this.splitStart != 0) {\n            this.stream.seek(offset);\n            readLine();\n            // if the first partial record already pushes the stream over\n            // the limit of our split, then no record starts within this split\n            if (this.overLimit) {\n                this.end = true;\n            }\n        } else {\n            fillBuffer(0);\n        }\n        initializeSplit(split, null);\n    }\n\n    private void initBuffers() {\n        this.bufferSize = this.bufferSize <= 0 ? DEFAULT_READ_BUFFER_SIZE : this.bufferSize;\n\n        if (this.bufferSize <= this.delimiter.length) {\n            throw new IllegalArgumentException(\n                    \"Buffer size must be greater than length of delimiter.\");\n        }\n\n        if (this.readBuffer == null || this.readBuffer.length != this.bufferSize) {\n            this.readBuffer = new byte[this.bufferSize];\n        }\n        if (this.wrapBuffer == null || this.wrapBuffer.length < 256) {\n            this.wrapBuffer = new byte[256];\n        }\n\n        this.readPos = 0;\n        this.limit = 0;\n        this.overLimit = false;\n        this.end = false;\n    }\n\n    /**\n     * Checks whether the current split is at its end.","sourceCodeStart":497,"sourceCodeEnd":533,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-core/src/main/java/org/apache/flink/api/common/io/DelimitedInputFormat.java#L497-L533","documentation":"In initBuffers(), DelimitedInputFormat normalizes bufferSize (defaulting to 1 MB if <= 0) and then requires the buffer to be strictly larger than the delimiter length so the delimiter-scanning loop can always fully observe the delimiter within a single buffer window. If bufferSize <= delimiter.length, scanning would miss delimiters, so the format rejects the configuration.","triggerScenarios":"Setting a multi-byte delimiter (e.g. '\\r\\n' or a long custom byte sequence) larger than or equal to the configured read buffer; setting bufferSize to a small value (e.g. 2) while using a 2-byte delimiter.","commonSituations":"Custom multi-byte record separators combined with aggressive buffer downsizing; copying a delimiter string that is longer than expected; using a very small buffer for testing.","solutions":["Increase bufferSize so it is strictly greater than delimiter.length (e.g. format.setBufferSize(Math.max(bufferBytes, delimiter.length + 1))).","Shorten the delimiter if a long one is not required.","Leave both at defaults (1 MB buffer, single-byte newline)."],"exampleFix":"// before\nformat.setDelimiter(\"\\r\\n\\n\".getBytes(StandardCharsets.UTF_8)); // 3 bytes\nformat.setBufferSize(2); // throws: 2 <= 3\n\n// after\nbyte[] delim = \"\\r\\n\\n\".getBytes(StandardCharsets.UTF_8);\nformat.setDelimiter(delim);\nformat.setBufferSize(Math.max(4 * 1024, delim.length + 1));","handlingStrategy":"validation","validationCode":"byte[] delim = configuredDelimiter;\nint minBuffer = Math.max(bufferSize, delim.length + 1);\nformat.setBufferSize(minBuffer);\nformat.setDelimiter(delim);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always size the buffer strictly larger than the delimiter.","When raising delimiter length, raise bufferSize correspondingly.","Unit-test the format with your exact delimiter before production."],"tags":["input-format","delimited","buffer-size","delimiter","validation","illegal-argument"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}