{"record":{"id":"1e142ba6b64b1539","repo":"apache/hadoop","slug":"too-many-bytes-before-delimiter-bytesconsumed","errorCode":null,"errorMessage":"Too many bytes before delimiter: \" + bytesConsumed","messagePattern":"Too many bytes before delimiter: \" \\+ bytesConsumed","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java","lineNumber":368,"sourceCode":"        // since it is now certain that the split did not split a delimiter we\n        // should not read the next record: clear the flag otherwise duplicate\n        // records could be generated\n        unsetNeedAdditionalRecordAfterSplit();\n      }\n      if (appendLength > 0) {\n        str.append(buffer, startPosn, appendLength);\n        txtLength += appendLength;\n      }\n      if (bufferPosn >= bufferLength) {\n        if (delPosn > 0 && delPosn < recordDelimiterBytes.length) {\n          ambiguousByteCount = delPosn;\n          bytesConsumed -= ambiguousByteCount; //to be consumed in next\n        }\n      }\n    } while (delPosn < recordDelimiterBytes.length \n        && bytesConsumed < maxBytesToConsume);\n    if (bytesConsumed > Integer.MAX_VALUE) {\n      throw new IOException(\"Too many bytes before delimiter: \" + bytesConsumed);\n    }\n    return (int) bytesConsumed; \n  }\n\n  /**\n   * Read from the InputStream into the given Text.\n   * @param str the object to store the given line\n   * @param maxLineLength the maximum number of bytes to store into str.\n   * @return the number of bytes read including the newline\n   * @throws IOException if the underlying stream throws\n   */\n  public int readLine(Text str, int maxLineLength) throws IOException {\n    return readLine(str, maxLineLength, Integer.MAX_VALUE);\n  }\n\n  /**\n   * Read from the InputStream into the given Text.\n   * @param str the object to store the given line","sourceCodeStart":350,"sourceCodeEnd":386,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java#L350-L386","documentation":"The custom-delimiter path readCustomLine() scans for recordDelimiterBytes. Partial delimiter matches at buffer boundaries are deferred via ambiguousByteCount, so a delimiter that never fully materializes keeps the loop consuming; when bytesConsumed exceeds Integer.MAX_VALUE without a complete match it throws IOException(\"Too many bytes before delimiter: N\") — the same 2 GiB guard as the newline path.","triggerScenarios":"LineReader configured with delimiter bytes that never fully occur in the data (\"\\r\\n\" on an LF-only file, a tab or sentinel string that is absent); binary data repeatedly producing partial delimiter-prefix matches; a legitimate single record over 2 GiB.","commonSituations":"textinputformat.record.delimiter (or a hand-built LineReader) set to the wrong byte sequence after a producer changed format; sentinel-delimited exports where the sentinel gained escaping; job upgrades that dropped the delimiter setting.","solutions":["Hexdump a record boundary (od -c) and rebuild the delimiter byte[] to match the file exactly","Fix the delimiter configuration (e.g. textinputformat.record.delimiter) to the actual separator","Choose a shorter delimiter guaranteed to appear per record, or pre-split oversized records","If the data itself is corrupt, restore it from source"],"exampleFix":"// before: guessed delimiter never occurs in the file\nLineReader r = new LineReader(in, \"\\r\\n\".getBytes(StandardCharsets.UTF_8));\n\n// after: delimiter verified against actual data (LF-only producer)\nLineReader r = new LineReader(in, \"\\n\".getBytes(StandardCharsets.UTF_8));","handlingStrategy":"try-catch","validationCode":"// verify the delimiter bytes actually occur in the file before processing\nbyte[] probe = Files.readAllBytes(Path.of(file)); // or stream first MB\nbyte[] d = \"\\u0001\".getBytes(StandardCharsets.UTF_8);\nboolean found = indexOf(probe, d) >= 0;\nif (!found) throw new IOException(\"record delimiter not present in input\");","typeGuard":null,"tryCatchPattern":"try { n = reader.readLine(text, maxLen); } catch (IOException e) { if (e.getMessage() != null && e.getMessage().startsWith(\"Too many bytes before delimiter\")) { throw new IOException(\"Delimiter mismatch: configured bytes never terminate a record\", e); } throw e; }","preventionTips":["Confirm delimiter bytes with od -c on a record boundary before configuring jobs","Keep delimiter configuration with the data producer contract and add a contract test","Beware escaped or doubled sentinels after producer upgrades"],"tags":["hadoop","java","io","text-parsing","record-delimiter"],"backgroundTag":"record-delimiter-not-found","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}