apache/hadoop · error · IOException

Too many bytes before newline: " + bytesConsumed

Error message

Too many bytes before newline: " + bytesConsumed

What it means

LineReader.readDefaultLine() accumulates consumed bytes into a long until it sees '\n' or reaches maxBytesToConsume (the public readLine passes Integer.MAX_VALUE). If more than 2^31-1 bytes flow by without a newline byte, the count can no longer be returned as int, so it throws IOException("Too many bytes before newline: N"). The throw is about the byte counter, not line truncation — text was already appended up to maxLineLength.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java:260

        prevCharCR = (buffer[bufferPosn] == CR);
      }
      int readLength = bufferPosn - startPosn;
      if (prevCharCR && newlineLength == 0) {
        --readLength; //CR at the end of the buffer
      }
      bytesConsumed += readLength;
      int appendLength = readLength - newlineLength;
      if (appendLength > maxLineLength - txtLength) {
        appendLength = maxLineLength - txtLength;
      }
      if (appendLength > 0) {
        str.append(buffer, startPosn, appendLength);
        txtLength += appendLength;
      }
    } while (newlineLength == 0 && bytesConsumed < maxBytesToConsume);

    if (bytesConsumed > Integer.MAX_VALUE) {
      throw new IOException("Too many bytes before newline: " + bytesConsumed);
    }
    return (int)bytesConsumed;
  }

  /**
   * Read a line terminated by a custom delimiter.
   */
  private int readCustomLine(Text str, int maxLineLength, int maxBytesToConsume)
      throws IOException {
   /* We're reading data from inputStream, but the head of the stream may be
    *  already captured in the previous buffer, so we have several cases:
    * 
    * 1. The buffer tail does not contain any character sequence which
    *    matches with the head of delimiter. We count it as a 
    *    ambiguous byte count = 0
    *    
    * 2. The buffer tail contains a X number of characters,
    *    that forms a sequence, which matches with the

View on GitHub (pinned to 2add963021)

Solutions

  1. Verify the input is actually newline-delimited text (head -c 4096 file | od -c | grep '\n')
  2. If records use a different separator, construct LineReader(in, recordDelimiterBytes) instead of relying on '\n'
  3. Pre-split or convert oversized single-record files before they reach LineReader
  4. For corrupt files, re-generate or restore the input rather than tuning limits

Example fix

// before: default '\n' reader on possibly-binary data
LineReader reader = new LineReader(in);

// after: explicit delimiter matches the actual record separator
LineReader reader = new LineReader(in, "\u0001".getBytes(StandardCharsets.UTF_8));
Defensive patterns

Strategy: try-catch

Validate before calling

// cheap pre-check that the input actually contains newline bytes
byte[] probe = new byte[8192];
int n = in.read(probe);
boolean hasNewline = false;
for (int i = 0; i < n; i++) { if (probe[i] == '\n') { hasNewline = true; break; } }
if (!hasNewline) throw new IOException("input does not look like newline-delimited text");

Try / catch

try { n = reader.readLine(text, maxLen); } catch (IOException e) { if (e.getMessage() != null && e.getMessage().startsWith("Too many bytes before newline")) { throw new IOException("Record exceeds 2GiB or input is not \\n-delimited text", e); } throw e; }

Prevention

When it happens

Trigger: Reading a file with no '\n' (0x0A) byte within the first 2 GiB: binary blobs (zip/avro/parquet) fed through LineRecordReader; data whose real record separator is not '\n'; a legitimate single record larger than Integer.MAX_VALUE bytes.

Common situations: Pointing a text InputFormat at non-text data; a record-delimiter mismatch so the actual separator never appears; unsplit single-record exports (huge XML/JSON) processed as text lines.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f6fed437e1976a31. Report an issue: GitHub.