{"record":{"id":"f6fed437e1976a31","repo":"apache/hadoop","slug":"too-many-bytes-before-newline-bytesconsumed","errorCode":null,"errorMessage":"Too many bytes before newline: \" + bytesConsumed","messagePattern":"Too many bytes before newline: \" \\+ bytesConsumed","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java","lineNumber":260,"sourceCode":"        prevCharCR = (buffer[bufferPosn] == CR);\n      }\n      int readLength = bufferPosn - startPosn;\n      if (prevCharCR && newlineLength == 0) {\n        --readLength; //CR at the end of the buffer\n      }\n      bytesConsumed += readLength;\n      int appendLength = readLength - newlineLength;\n      if (appendLength > maxLineLength - txtLength) {\n        appendLength = maxLineLength - txtLength;\n      }\n      if (appendLength > 0) {\n        str.append(buffer, startPosn, appendLength);\n        txtLength += appendLength;\n      }\n    } while (newlineLength == 0 && bytesConsumed < maxBytesToConsume);\n\n    if (bytesConsumed > Integer.MAX_VALUE) {\n      throw new IOException(\"Too many bytes before newline: \" + bytesConsumed);\n    }\n    return (int)bytesConsumed;\n  }\n\n  /**\n   * Read a line terminated by a custom delimiter.\n   */\n  private int readCustomLine(Text str, int maxLineLength, int maxBytesToConsume)\n      throws IOException {\n   /* We're reading data from inputStream, but the head of the stream may be\n    *  already captured in the previous buffer, so we have several cases:\n    * \n    * 1. The buffer tail does not contain any character sequence which\n    *    matches with the head of delimiter. We count it as a \n    *    ambiguous byte count = 0\n    *    \n    * 2. The buffer tail contains a X number of characters,\n    *    that forms a sequence, which matches with the","sourceCodeStart":242,"sourceCodeEnd":278,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java#L242-L278","documentation":"LineReader.readDefaultLine() accumulates consumed bytes into a long until it sees '\\n' or reaches maxBytesToConsume (the public readLine passes Integer.MAX_VALUE). If more than 2^31-1 bytes flow by without a newline byte, the count can no longer be returned as int, so it throws IOException(\"Too many bytes before newline: N\"). The throw is about the byte counter, not line truncation — text was already appended up to maxLineLength.","triggerScenarios":"Reading a file with no '\\n' (0x0A) byte within the first 2 GiB: binary blobs (zip/avro/parquet) fed through LineRecordReader; data whose real record separator is not '\\n'; a legitimate single record larger than Integer.MAX_VALUE bytes.","commonSituations":"Pointing a text InputFormat at non-text data; a record-delimiter mismatch so the actual separator never appears; unsplit single-record exports (huge XML/JSON) processed as text lines.","solutions":["Verify the input is actually newline-delimited text (head -c 4096 file | od -c | grep '\\n')","If records use a different separator, construct LineReader(in, recordDelimiterBytes) instead of relying on '\\n'","Pre-split or convert oversized single-record files before they reach LineReader","For corrupt files, re-generate or restore the input rather than tuning limits"],"exampleFix":"// before: default '\\n' reader on possibly-binary data\nLineReader reader = new LineReader(in);\n\n// after: explicit delimiter matches the actual record separator\nLineReader reader = new LineReader(in, \"\\u0001\".getBytes(StandardCharsets.UTF_8));","handlingStrategy":"try-catch","validationCode":"// cheap pre-check that the input actually contains newline bytes\nbyte[] probe = new byte[8192];\nint n = in.read(probe);\nboolean hasNewline = false;\nfor (int i = 0; i < n; i++) { if (probe[i] == '\\n') { hasNewline = true; break; } }\nif (!hasNewline) throw new IOException(\"input does not look like newline-delimited text\");","typeGuard":null,"tryCatchPattern":"try { n = reader.readLine(text, maxLen); } catch (IOException e) { if (e.getMessage() != null && e.getMessage().startsWith(\"Too many bytes before newline\")) { throw new IOException(\"Record exceeds 2GiB or input is not \\\\n-delimited text\", e); } throw e; }","preventionTips":["Sanity-check input format before streaming (file type, sample od -c)","Set the real record delimiter via LineReader(in, recordDelimiterBytes)","In tests, feed binary garbage to assert your pipeline fails with a clear message"],"tags":["hadoop","java","io","text-parsing","line-reader"],"backgroundTag":"line-length-overflow","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}